Can I read Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation on EtoBox?

Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation by Li, Yunxin; Shi, Haoyuan; Hu, Baotian; Wang, Longyue; Zhu, Jiashun; Xu, Jinyi; Zhao, Zhen; Zhang, Min is a scholarly article available to read on EtoBox.

What is Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation about?

Traditional animation generation methods depend on training generative models with human-labelled data, entailing a sophisticated multi-stage pipeline that demands substantial human effort and incurs high training costs. Due to limited prompting plans, these methods typically produce brief, information-poor, and context-incoherent animations. To overcome these limitations and automate the animation process, we pioneer the introduction of large multimodal models (LMMs) as the core processor to build an autonomous animation-making agent, named Anim-Director. This agent mainly harnesses the advanced understanding and reasoning capabilities of LMMs and generative AI tools to create animated videos from concise narratives or simple instructions. Specifically, it operates in three main stages: Firstly, the Anim-Director generates a coherent storyline from user inputs, followed by a detailed director's script that encompasses settings of character profiles and interior/exterior descriptions, and context-coherent scene descriptions that include appearing characters, interiors or exteriors, and scene events. Secondly, we employ LMMs with the image generation tool to produce visual images of

Author

Li, Yunxin; Shi, Haoyuan; Hu, Baotian; Wang, Longyue; Zhu, Jiashun; Xu, Jinyi; Zhao, Zhen; Zhang, Min

Published

2024

Language

EN