PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation

Yingchaojie Feng, Xingbo Wang, Kam Kwai Wong, Sijia Wang, Yuhong Lu, Minfeng Zhu, Baicheng Wang, Wei Chen

DOI: 10.1109/TVCG.2023.3327168

Room: 106

2023-10-25T00:45:00ZGMT-0600Change your timezone on the schedule page
2023-10-25T00:45:00Z
Exemplar figure, described by caption below
The user interface of PromptMagician consists of four views. The Model Input View (A) configures the prompts and model parameters for image creation. The Image Browser View (B) visualizes the generated and retrieved images and the recommended prompt keywords. The Image Evaluation View (C) helps evaluate and filter images based on multiple criteria. The Local Exploration View (D) helps users explore and validate the prompt keywords and guidance scales for images of interest.
Fast forward
Full Video
Keywords

Prompt engineering, text-to-image generation, image visualization

Abstract

Generative text-to-image models have gained great popularity among the public for their powerful capability to generate high-quality images based on natural language prompts. However, developing effective prompts for desired images can be challenging due to the complexity and ambiguity of natural language. This research proposes PromptMagician, a visual analysis system that helps users explore the image results and refine the input prompts. The backbone of our system is a prompt recommendation model that takes user prompts as input, retrieves similar prompt-image pairs from DiffusionDB, and identifies special (important and relevant) prompt keywords. To facilitate interactive prompt refinement, PromptMagician introduces a multi-level visualization for the cross-modal embedding of the retrieved images and recommended keywords, and supports users in specifying multiple criteria for personalized exploration. Two usage scenarios, a user study, and expert interviews demonstrate the effectiveness and usability of our system, suggesting it facilitates prompt engineering and improves the creativity support of the generative text-to-image model.