Can I read SpectralCLIP: Preventing Artifacts in Text-Guided Style Transfer from a Spectral Perspective on EtoBox?

SpectralCLIP: Preventing Artifacts in Text-Guided Style Transfer from a Spectral Perspective by Xu, Zipeng; Xing, Songlong; Sangineto, Enver; Sebe, Nicu is a scholarly article available to read on EtoBox.

What is SpectralCLIP: Preventing Artifacts in Text-Guided Style Transfer from a Spectral Perspective about?

Owing to the power of vision-language foundation models, e.g., CLIP, the area of image synthesis has seen recent important advances. Particularly, for style transfer, CLIP enables transferring more general and abstract styles without collecting the style images in advance, as the style can be efficiently described with natural language, and the result is optimized by minimizing the CLIP similarity between the text description and the stylized image. However, directly using CLIP to guide style transfer leads to undesirable artifacts (mainly written words and unrelated visual entities) spread over the image. In this paper, we propose SpectralCLIP, which is based on a spectral representation of the CLIP embedding sequence, where most of the common artifacts occupy specific frequencies. By masking the band including these frequencies, we can condition the generation process to adhere to the target style properties (e.g., color, texture, paint stroke, etc.) while excluding the generation of larger-scale structures corresponding to the artifacts. Experimental results show that SpectralCLIP prevents the generation of artifacts effectively in quantitative and qualitative terms, without imp

Author

Xu, Zipeng; Xing, Songlong; Sangineto, Enver; Sebe, Nicu

Published

2023

Language

EN