Can I read Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection on EtoBox?

Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection by Cao, Yichao; Tang, Qingfei; Yang, Feng; Su, Xiu; You, Shan; Lu, Xiaobo; Xu, Chang is a scholarly article available to read on EtoBox.

What is Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection about?

Human-Object Interaction (HOI) detection is a challenging computer vision task that requires visual models to address the complex interactive relationship between humans and objects and predict HOI triplets. Despite the challenges posed by the numerous interaction combinations, they also offer opportunities for multimodal learning of visual texts. In this paper, we present a systematic and unified framework (RmLR) that enhances HOI detection by incorporating structured text knowledge. Firstly, we qualitatively and quantitatively analyze the loss of interaction information in the two-stage HOI detector and propose a re-mining strategy to generate more comprehensive visual representation.Secondly, we design more fine-grained sentence- and word-level alignment and knowledge transfer strategies to effectively address the many-to-many matching problem between multiple interactions and multiple texts.These strategies alleviate the matching confusion problem that arises when multiple interactions occur simultaneously, thereby improving the effectiveness of the alignment process. Finally, HOI reasoning by visual features augmented with textual knowledge substantially improves the understan

Author

Cao, Yichao; Tang, Qingfei; Yang, Feng; Su, Xiu; You, Shan; Lu, Xiaobo; Xu, Chang

Published

2023

Language

EN