Can I read PoseTalk: Text-and-Audio-based Pose Control and Motion Refinement for One-Shot Talking Head Generation on EtoBox?

PoseTalk: Text-and-Audio-based Pose Control and Motion Refinement for One-Shot Talking Head Generation by Ling, Jun; Wang, Yiwen; Xue, Han; Xie, Rong; Song, Li is a scholarly article available to read on EtoBox.

What is PoseTalk: Text-and-Audio-based Pose Control and Motion Refinement for One-Shot Talking Head Generation about?

While previous audio-driven talking head generation (THG) methods generate head poses from driving audio, the generated poses or lips cannot match the audio well or are not editable. In this study, we propose \textbf{PoseTalk}, a THG system that can freely generate lip-synchronized talking head videos with free head poses conditioned on text prompts and audio. The core insight of our method is using head pose to connect visual, linguistic, and audio signals. First, we propose to generate poses from both audio and text prompts, where the audio offers short-term variations and rhythm correspondence of the head movements and the text prompts describe the long-term semantics of head motions. To achieve this goal, we devise a Pose Latent Diffusion (PLD) model to generate motion latent from text prompts and audio cues in a pose latent space. Second, we observe a loss-imbalance problem: the loss for the lip region contributes less than 4\% of the total reconstruction loss caused by both pose and lip, making optimization lean towards head movements rather than lip shapes. To address this issue, we propose a refinement-based learning strategy to synthesize natural talking videos using two c

Author

Ling, Jun; Wang, Yiwen; Xue, Han; Xie, Rong; Song, Li

Published

2024

Language

EN