Can I read Harnessing Diffusion Models for Visual Perception with Meta Prompts on EtoBox?

Harnessing Diffusion Models for Visual Perception with Meta Prompts by Wan, Qiang; Huang, Zilong; Kang, Bingyi; Feng, Jiashi; Zhang, Li is a scholarly article available to read on EtoBox.

What is Harnessing Diffusion Models for Visual Perception with Meta Prompts about?

The issue of generative pretraining for vision models has persisted as a long-standing conundrum. At present, the text-to-image (T2I) diffusion model demonstrates remarkable proficiency in generating high-definition images matching textual inputs, a feat made possible through its pre-training on large-scale image-text pairs. This leads to a natural inquiry: can diffusion models be utilized to tackle visual perception tasks? In this paper, we propose a simple yet effective scheme to harness a diffusion model for visual perception tasks. Our key insight is to introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception. The effect of meta prompts are two-fold. First, as a direct replacement of the text embeddings in the T2I models, it can activate task-relevant features during feature extraction. Second, it will be used to re-arrange the extracted features to ensures that the model focuses on the most pertinent features for the task on hand. Additionally, we design a recurrent refinement training strategy that fully leverages the property of diffusion models, thereby yielding stronger visual features. Extensive experiment

Author

Wan, Qiang; Huang, Zilong; Kang, Bingyi; Feng, Jiashi; Zhang, Li

Published

2023

Language

EN