REVIEW 20 cited by
GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Recent breakthroughs in diffusion models, multimodal pretraining, and efficient finetuning have led to an explosion of text-to-image generative models. Given human evaluation is expensive and difficult to scale, automated methods are critical for evaluating the increasingly large number of new models. However, most current automated evaluation metrics like FID or CLIPScore only offer a holistic measure of image quality or image-text alignment, and are unsuited for fine-grained or instance-level analysis. In this paper, we introduce GenEval, an object-focused framework to evaluate compositional image properties such as object co-occurrence, position, count, and color. We show that current object detection models can be leveraged to evaluate text-to-image models on a variety of generation tasks with strong human agreement, and that other discriminative vision models can be linked to this pipeline to further verify properties like object color. We then evaluate several open-source text-to-image models and analyze their relative generative capabilities on our benchmark. We find that recent models demonstrate significant improvement on these tasks, though they are still lacking in complex capabilities such as spatial relations and attribute binding. Finally, we demonstrate how GenEval might be used to help discover existing failure modes, in order to inform development of the next generation of text-to-image models. Our code to run the GenEval framework is publicly available at https://github.com/djghosh13/geneval.
Forward citations
Cited by 20 Pith papers
-
Do You See What You Draw? A Semantic Closed-Loop Framework for Holistic Evaluation of Unified Multimodal Models
By asking a unified multimodal model to caption an image, regenerate it, and answer questions about the regenerated image, SGU consistently exposes accuracy drops that separate benchmarks do not reveal.
-
AR-RAG: Autoregressive Retrieval Augmentation for Image Generation
Autoregressive patch-level retrieval augmentation improves text-to-image generation on GenEval, DPG-Bench, and Midjourney-30K, with a training-free decoding variant and a fine-tuned variant.
-
SABRE: Scalable and Automated Benchmarking of VLMs under Stress
SABRE is a modular pipeline for automatically building and refreshing VLM stress tests, and its SABRE-Prior instantiation finds that six frontier VLMs score 17.8 to 31.3 percent when visual evidence conflicts with wor...
-
Simile Understanding in Text-to-Image Models: An Evaluation Framework
A YOLO-based evaluation framework shows that text-to-image models consistently render the literal vehicle of a simile, a failure that CLIPScore and PickScore miss.
-
RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination
A unified transformer model generates language-and-image planning sequences for embodied tasks, and shows real-robot manipulation without large-scale action pretraining.
-
Improving Sample Diversity in Autoregressive Text-to-Image Generation via Cluster Truncation
p-less cluster decoding, which truncates and samples over K-means clusters of visual tokens rather than individual tokens, yields higher per-prompt sample diversity than default or dynamic-temperature baselines on mos...
-
NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning
NormGuard, a hinge penalty on excess velocity norm during RL post-training of flow models, improves perceptual quality and realism without sacrificing reward.
-
Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generation
A curated GPT-4o synthetic image dataset improves open-source generation models on instruction-following, surreal scenes, and multi-reference synthesis, plus two new benchmarks to measure those skills.
-
Enhancing Reward Models for High-quality Image Generation: Beyond Text-Image Alignment
The paper proposes ICT and HP scores, trained on a newly built high-quality triplet dataset, to replace CLIP-style text-image alignment rewards that allegedly penalize detail-rich images; the new rewards modestly impr...
-
Transition Matching: Scalable and Flexible Generative Modeling
Transition Matching unifies flow matching and continuous autoregressive generation as discrete-time Markov processes, with three variants that improve text-to-image quality and speed.
-
Ambient Diffusion Omni: Training Good Models with Bad Data
Ambient Diffusion Omni trains diffusion models on mixed-quality data by learning when corrupted images can be treated as clean, improving generation quality and diversity.
-
Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas
Chipmunk speeds up diffusion transformer inference by recomputing, at each step, only the top percent of attention and MLP activation columns that change most between steps, caching the rest in column-sparse GPU kernels.
-
RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction
RICO refines image captions by reconstructing them into images with a text-to-image model and asking GPT-4o to fix discrepancies against the original, iteratively, with a DPO-distilled fast variant.
-
UniCMs: A Unified Consistency Model For Efficient Multimodal Generation and Understanding
UniCMs applies consistency distillation to a unified multimodal transformer, treating image mask-diffusion steps and text parallel-decoding steps as one shared denoising trajectory, enabling 2 to 8 step generation and...
-
VIGOR: VIdeo Geometry-Oriented Reward for Temporal Generative Alignment
A VGGT-based pointwise reprojection reward with geometry-aware sampling improves video geometric consistency via SFT/DPO and causal test-time search.
-
Optimizing Few-Step Generation with Adaptive Matching Distillation
Adaptive Matching Distillation uses reward-model scores to reweight teacher and fake-teacher gradients, improving few-step diffusion distillation on image and video benchmarks.
-
Why Settle for Mid: A Probabilistic Viewpoint to Spatial Relationship Alignment in Text-to-image Models
A probabilistic overlap measure for object positions yields a human-aligned spatial relationship metric and a training-free generation guidance method for text-to-image models.
-
Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis
Deep fusion of a frozen LLM with a DiT improves text-image alignment over shallow fusion baselines, and a scaled recipe (FuseDiT) achieves competitive results despite limited data and compute.
-
SkipVAR: Accelerating Visual Autoregressive Modeling via Adaptive Frequency-Aware Skipping
SkipVAR selects, per sample, between step skipping and unconditional branch replacement using handcrafted frequency features and a trained logistic regression, to accelerate visual autoregressive generation.
-
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey
A comprehensive survey that frames multimodal understanding and generation as next token prediction and proposes a five-part taxonomy.
Discussion (0). Continue with ORCID to comment.