Can I read SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction on EtoBox?

SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction by Chen, Xinyuan; Wang, Yaohui; Zhang, Lingjun; Zhuang, Shaobin; Ma, Xin; Yu, Jiashuo; Wang, Yali; Lin, Dahua; Qiao, Yu; Liu, Ziwei is a scholarly article available to read on EtoBox.

What is SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction about?

Recently video generation has achieved substantial progress with realistic results. Nevertheless, existing AI-generated videos are usually very short clips ("shot-level") depicting a single scene. To deliver a coherent long video ("story-level"), it is desirable to have creative transition and prediction effects across different clips. This paper presents a short-to-long video diffusion model, SEINE, that focuses on generative transition and prediction. The goal is to generate high-quality long videos with smooth and creative transitions between scenes and varying lengths of shot-level videos. Specifically, we propose a random-mask video diffusion model to automatically generate transitions based on textual descriptions. By providing the images of different scenes as inputs, combined with text-based control, our model generates transition videos that ensure coherence and visual quality. Furthermore, the model can be readily extended to various tasks such as image-to-video animation and autoregressive video prediction. To conduct a comprehensive evaluation of this new generative task, we propose three assessing criteria for smooth and creative transition: temporal consistency, seman

Author

Chen, Xinyuan; Wang, Yaohui; Zhang, Lingjun; Zhuang, Shaobin; Ma, Xin; Yu, Jiashuo; Wang, Yali; Lin, Dahua; Qiao, Yu; Liu, Ziwei

Published

2023

Language

EN