Can I read Cross-Modal Learning with 3D Deformable Attention for Action Recognition on EtoBox?

Cross-Modal Learning with 3D Deformable Attention for Action Recognition by Kim, Sangwon; Ahn, Dasom; Ko, Byoung Chul is a scholarly article available to read on EtoBox.

What is Cross-Modal Learning with 3D Deformable Attention for Action Recognition about?

An important challenge in vision-based action recognition is the embedding of spatiotemporal features with two or more heterogeneous modalities into a single feature. In this study, we propose a new 3D deformable transformer for action recognition with adaptive spatiotemporal receptive fields and a cross-modal learning scheme. The 3D deformable transformer consists of three attention modules: 3D deformability, local joint stride, and temporal stride attention. The two cross-modal tokens are input into the 3D deformable attention module to create a cross-attention token with a reflected spatiotemporal correlation. Local joint stride attention is applied to spatially combine attention and pose tokens. Temporal stride attention temporally reduces the number of input tokens in the attention module and supports temporal expression learning without the simultaneous use of all tokens. The deformable transformer iterates L-times and combines the last cross-modal token for classification. The proposed 3D deformable transformer was tested on the NTU60, NTU120, FineGYM, and PennAction datasets, and showed results better than or similar to pre-trained state-of-the-art methods even without a pr

Author

Kim, Sangwon; Ahn, Dasom; Ko, Byoung Chul

Published

2022

Language

EN