Can I read Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizers on EtoBox?

Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizers by Williams, Joshua Nathaniel; Schwarzschild, Avi; Kolter, J. Zico is a scholarly article available to read on EtoBox.

What is Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizers about?

Recovering natural language prompts for image generation models, solely based on the generated images is a difficult discrete optimization problem. In this work, we present the first head-to-head comparison of recent discrete optimization techniques for the problem of prompt inversion. We evaluate Greedy Coordinate Gradients (GCG), PEZ , Random Search, AutoDAN and BLIP2's image captioner across various evaluation metrics related to the quality of inverted prompts and the quality of the images generated by the inverted prompts. We find that focusing on the CLIP similarity between the inverted prompts and the ground truth image acts as a poor proxy for the similarity between ground truth image and the image generated by the inverted prompts. While the discrete optimizers effectively minimize their objectives, simply using responses from a well-trained captioner often leads to generated images that more closely resemble those produced by the original prompts.

Author

Williams, Joshua Nathaniel; Schwarzschild, Avi; Kolter, J. Zico

Published

2024

Language

EN