Can I read BIAS: A Body-based Interpretable Active Speaker Approach on EtoBox?

BIAS: A Body-based Interpretable Active Speaker Approach by Roxo, Tiago; Costa, Joana C.; Inácio, Pedro R. M.; Proença, Hugo is a scholarly article available to read on EtoBox.

What is BIAS: A Body-based Interpretable Active Speaker Approach about?

State-of-the-art Active Speaker Detection (ASD) approaches heavily rely on audio and facial features to perform, which is not a sustainable approach in wild scenarios. Although these methods achieve good results in the standard AVA-ActiveSpeaker set, a recent wilder ASD dataset (WASD) showed the limitations of such models and raised the need for new approaches. As such, we propose BIAS, a model that, for the first time, combines audio, face, and body information, to accurately predict active speakers in varying/challenging conditions. Additionally, we design BIAS to provide interpretability by proposing a novel use for Squeeze-and-Excitation blocks, namely in attention heatmaps creation and feature importance assessment. For a full interpretability setup, we annotate an ASD-related actions dataset (ASD-Text) to finetune a ViT-GPT2 for text scene description to complement BIAS interpretability. The results show that BIAS is state-of-the-art in challenging conditions where body-based features are of utmost importance (Columbia, open-settings, and WASD), and yields competitive results in AVA-ActiveSpeaker, where face is more influential than body for ASD. BIAS interpretability also sh

Author

Roxo, Tiago; Costa, Joana C.; Inácio, Pedro R. M.; Proença, Hugo

Published

2024

Language

EN