Can I read Rethinking Vision Transformer and Masked Autoencoder in Multimodal Face Anti-Spoofing on EtoBox?

Rethinking Vision Transformer and Masked Autoencoder in Multimodal Face Anti-Spoofing by Yu, Zitong; Cai, Rizhao; Cui, Yawen; Liu, Xin; Hu, Yongjian; Kot, Alex is a scholarly article available to read on EtoBox.

What is Rethinking Vision Transformer and Masked Autoencoder in Multimodal Face Anti-Spoofing about?

Recently, vision transformer (ViT) based multimodal learning methods have been proposed to improve the robustness of face anti-spoofing (FAS) systems. However, there are still no works to explore the fundamental natures (\textit{e.g.}, modality-aware inputs, suitable multimodal pre-training, and efficient finetuning) in vanilla ViT for multimodal FAS. In this paper, we investigate three key factors (i.e., inputs, pre-training, and finetuning) in ViT for multimodal FAS with RGB, Infrared (IR), and Depth. First, in terms of the ViT inputs, we find that leveraging local feature descriptors benefits the ViT on IR modality but not RGB or Depth modalities. Second, in observation of the inefficiency on direct finetuning the whole or partial ViT, we design an adaptive multimodal adapter (AMA), which can efficiently aggregate local multimodal features while freezing majority of ViT parameters. Finally, in consideration of the task (FAS vs. generic object classification) and modality (multimodal vs. unimodal) gaps, ImageNet pre-trained models might be sub-optimal for the multimodal FAS task. To bridge these gaps, we propose the modality-asymmetric masked autoencoder (M$^{2}$A$^{2}$E) for mul

Author

Yu, Zitong; Cai, Rizhao; Cui, Yawen; Liu, Xin; Hu, Yongjian; Kot, Alex

Published

2023

Language

EN