Can I read Universal Facial Encoding of Codec Avatars from VR Headsets on EtoBox?

Universal Facial Encoding of Codec Avatars from VR Headsets by Bai, Shaojie; Wang, Te-Li; Li, Chenghui; Venkatesh, Akshay; Simon, Tomas; Cao, Chen; Schwartz, Gabriel; Wrench, Ryan; Saragih, Jason; Sheikh, Yaser; Wei, Shih-En is a scholarly article available to read on EtoBox.

What is Universal Facial Encoding of Codec Avatars from VR Headsets about?

Faithful real-time facial animation is essential for avatar-mediated telepresence in Virtual Reality (VR). To emulate authentic communication, avatar animation needs to be efficient and accurate: able to capture both extreme and subtle expressions within a few milliseconds to sustain the rhythm of natural conversations. The oblique and incomplete views of the face, variability in the donning of headsets, and illumination variation due to the environment are some of the unique challenges in generalization to unseen faces. In this paper, we present a method that can animate a photorealistic avatar in realtime from head-mounted cameras (HMCs) on a consumer VR headset. We present a self-supervised learning approach, based on a cross-view reconstruction objective, that enables generalization to unseen users. We present a lightweight expression calibration mechanism that increases accuracy with minimal additional cost to run-time efficiency. We present an improved parameterization for precise ground-truth generation that provides robustness to environmental variation. The resulting system produces accurate facial animation for unseen users wearing VR headsets in realtime. We compare our

Author

Bai, Shaojie; Wang, Te-Li; Li, Chenghui; Venkatesh, Akshay; Simon, Tomas; Cao, Chen; Schwartz, Gabriel; Wrench, Ryan; Saragih, Jason; Sheikh, Yaser; Wei, Shih-En

Published

2024

Language

EN