Can I read S2DM: Sector-Shaped Diffusion Models for Video Generation on EtoBox?

S2DM: Sector-Shaped Diffusion Models for Video Generation by Lang, Haoran; Ge, Yuxuan; Tian, Zheng is a scholarly article available to read on EtoBox.

What is S2DM: Sector-Shaped Diffusion Models for Video Generation about?

Diffusion models have achieved great success in image generation. However, when leveraging this idea for video generation, we face significant challenges in maintaining the consistency and continuity across video frames. This is mainly caused by the lack of an effective framework to align frames of videos with desired temporal features while preserving consistent semantic and stochastic features. In this work, we propose a novel Sector-Shaped Diffusion Model (S2DM) whose sector-shaped diffusion region is formed by a set of ray-shaped reverse diffusion processes starting at the same noise point. S2DM can generate a group of intrinsically related data sharing the same semantic and stochastic features while varying on temporal features with appropriate guided conditions. We apply S2DM to video generation tasks, and explore the use of optical flow as temporal conditions. Our experimental results show that S2DM outperforms many existing methods in the task of video generation without any temporal-feature modelling modules. For text-to-video generation tasks where temporal conditions are not explicitly given, we propose a two-stage generation strategy which can decouple the generation of

Author

Lang, Haoran; Ge, Yuxuan; Tian, Zheng

Published

2024

Language

EN