Can I read RefFusion: Reference Adapted Diffusion Models for 3D Scene Inpainting on EtoBox?

RefFusion: Reference Adapted Diffusion Models for 3D Scene Inpainting by Mirzaei, Ashkan; De Lutio, Riccardo; Kim, Seung Wook; Acuna, David; Kelly, Jonathan; Fidler, Sanja; Gilitschenski, Igor; Gojcic, Zan is a scholarly article available to read on EtoBox.

What is RefFusion: Reference Adapted Diffusion Models for 3D Scene Inpainting about?

Neural reconstruction approaches are rapidly emerging as the preferred representation for 3D scenes, but their limited editability is still posing a challenge. In this work, we propose an approach for 3D scene inpainting -- the task of coherently replacing parts of the reconstructed scene with desired content. Scene inpainting is an inherently ill-posed task as there exist many solutions that plausibly replace the missing content. A good inpainting method should therefore not only enable high-quality synthesis but also a high degree of control. Based on this observation, we focus on enabling explicit control over the inpainted content and leverage a reference image as an efficient means to achieve this goal. Specifically, we introduce RefFusion, a novel 3D inpainting method based on a multi-scale personalization of an image inpainting diffusion model to the given reference view. The personalization effectively adapts the prior distribution to the target scene, resulting in a lower variance of score distillation objective and hence significantly sharper details. Our framework achieves state-of-the-art results for object removal while maintaining high controllability. We further demo

Author

Mirzaei, Ashkan; De Lutio, Riccardo; Kim, Seung Wook; Acuna, David; Kelly, Jonathan; Fidler, Sanja; Gilitschenski, Igor; Gojcic, Zan

Published

2024

Language

EN