Can I read GazeGen: Gaze-Driven User Interaction for Visual Content Generation on EtoBox?

GazeGen: Gaze-Driven User Interaction for Visual Content Generation by Hsieh, He-Yen; Li, Ziyun; Zhang, Sai Qian; Ting, Wei-Te Mark; Chang, Kao-Den; De Salvo, Barbara; Liu, Chiao; Kung, H. T. is a scholarly article available to read on EtoBox.

What is GazeGen: Gaze-Driven User Interaction for Visual Content Generation about?

We present GazeGen, a user interaction system that generates visual content (images and videos) for locations indicated by the user's eye gaze. GazeGen allows intuitive manipulation of visual content by targeting regions of interest with gaze. Using advanced techniques in object detection and generative AI, GazeGen performs gaze-controlled image adding/deleting, repositioning, and surface style changes of image objects, and converts static images into videos. Central to GazeGen is the DFT Gaze (Distilled and Fine-Tuned Gaze) agent, an ultra-lightweight model with only 281K parameters, performing accurate real-time gaze predictions tailored to individual users' eyes on small edge devices. GazeGen is the first system to combine visual content generation with real-time gaze estimation, made possible exclusively by DFT Gaze. This real-time gaze estimation enables various visual content generation tasks, all controlled by the user's gaze. The input for DFT Gaze is the user's eye images, while the inputs for visual content generation are the user's view and the predicted gaze point from DFT Gaze. To achieve efficient gaze predictions, we derive the small model from a large model (10x lar

Author

Hsieh, He-Yen; Li, Ziyun; Zhang, Sai Qian; Ting, Wei-Te Mark; Chang, Kao-Den; De Salvo, Barbara; Liu, Chiao; Kung, H. T.

Published

2024

Language

EN