REVIEW 7 cited by
Self-correcting LLM-controlled Diffusion Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Text-to-image generation has witnessed significant progress with the advent of diffusion models. Despite the ability to generate photorealistic images, current text-to-image diffusion models still often struggle to accurately interpret and follow complex input text prompts. In contrast to existing models that aim to generate images only with their best effort, we introduce Self-correcting LLM-controlled Diffusion (SLD). SLD is a framework that generates an image from the input prompt, assesses its alignment with the prompt, and performs self-corrections on the inaccuracies in the generated image. Steered by an LLM controller, SLD turns text-to-image generation into an iterative closed-loop process, ensuring correctness in the resulting image. SLD is not only training-free but can also be seamlessly integrated with diffusion models behind API access, such as DALL-E 3, to further boost the performance of state-of-the-art diffusion models. Experimental results show that our approach can rectify a majority of incorrect generations, particularly in generative numeracy, attribute binding, and spatial relationships. Furthermore, by simply adjusting the instructions to the LLM, SLD can perform image editing tasks, bridging the gap between text-to-image generation and image editing pipelines. We will make our code available for future research and applications.
Forward citations
Cited by 7 Pith papers
-
Audit & Repair: An Agentic Framework for Consistent Story Visualization in Text-to-Image Diffusion Models
A multi-agent system that audits story images with a vision-language model and repairs inconsistencies with targeted diffusion edits improves multi-panel consistency over existing story visualization methods.
-
EditInspector: A Benchmark for Evaluation of Text-Guided Image Edits
A new human-labeled benchmark shows leading vision-language models are unreliable at judging image edits, and the authors' methods improve artifact detection and difference captioning.
-
A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data
A metric-oriented survey that classifies intrinsic quality and trustworthiness metrics for LLM-generated data across six modalities and documents systematic evaluation gaps in the current literature.
-
GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesis
A generate-plan-edit wrapper, using GPT-4o to plan atomic edits and a diffusion editor to execute them, improves compositional text-to-image fidelity across many T2I models.
-
A Data-Centric Approach for Safe and Secure Large Language Models against Threatening and Toxic Content
A BART-based post-generation corrector lowers toxicity and jailbreaking scores, but the reported gains are partly in-sample because thresholds are optimized on the evaluation data.
-
LLM-guided Instance-level Image Manipulation with Diffusion U-Net Cross-Attention Maps
An LLM plus open-vocabulary detector locates instances in a generated image, and cross-attention and feature guidance reposition one instance without masks or retraining.
-
DynamicAvatars: Accurate Dynamic Facial Avatars Reconstruction and Precise Editing with Diffusion Models
DynamicAvatars reconstructs dynamic 3D head avatars from video and enables prompt-based editing via dual Gaussian tracking, semantic masks, and LLM-guided diffusion editing.
Discussion (0). Continue with ORCID to comment.