Pith. sign in

REVIEW 10 cited by

Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.07546 v2 pith:F4ANDLOL submitted 2024-06-11 cs.CV cs.AIcs.CL

Xingyu Fu , Muyu He , Yujie Lu , William Yang Wang , Dan Roth This is my paper
classification cs.CVcs.AIcs.CL
keywords commonsense-t2icommonsenselightbulbmodelsbenchmarkchallengegenerationlife
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present a novel task and benchmark for evaluating the ability of text-to-image(T2I) generation models to produce images that align with commonsense in real life, which we call Commonsense-T2I. Given two adversarial text prompts containing an identical set of action words with minor differences, such as "a lightbulb without electricity" v.s. "a lightbulb with electricity", we evaluate whether T2I models can conduct visual-commonsense reasoning, e.g. produce images that fit "the lightbulb is unlit" vs. "the lightbulb is lit" correspondingly. Commonsense-T2I presents an adversarial challenge, providing pairwise text prompts along with expected outputs. The dataset is carefully hand-curated by experts and annotated with fine-grained labels, such as commonsense type and likelihood of the expected outputs, to assist analyzing model behavior. We benchmark a variety of state-of-the-art (sota) T2I models and surprisingly find that, there is still a large gap between image synthesis and real life photos--even the DALL-E 3 model could only achieve 48.92% on Commonsense-T2I, and the stable diffusion XL model only achieves 24.92% accuracy. Our experiments show that GPT-enriched prompts cannot solve this challenge, and we include a detailed analysis about possible reasons for such deficiency. We aim for Commonsense-T2I to serve as a high-quality evaluation benchmark for T2I commonsense checking, fostering advancements in real life image generation.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Do Image Editing Models Understand Lighting?

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    New 3DLP benchmark with real-world 1K HDR pairs shows state-of-the-art image editing models vary in physical lighting consistency, with best models close to reality but error-prone in low-light regions.

  2. FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmark

    cs.CV 2025-09 conditional novelty 7.0 of 10

    The authors build a 6M-image, 20M-caption reasoning dataset with generation chain-of-thought and a 7-track VLM-judged benchmark, then rank 19 text-to-image models.

  3. GenSpace: Benchmarking Spatially-Aware Image Generation

    cs.CV 2025-05 conditional novelty 7.0 of 10

    GenSpace benchmarks spatial awareness in image generation with a 3D reconstruction-based evaluator, showing models struggle with allocentric relations and metric measurements.

  4. R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation

    cs.CV 2025-05 conditional novelty 7.0 of 10

    A 3,068-prompt benchmark with per-instance Q&A scoring shows that current text-to-image models, including reasoning-enhanced ones, handle reasoning-driven prompts poorly, with mathematical reasoning near zero.

  5. ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    ToolArtist trains a unified multimodal model to reason, search the web, and generate images as one policy, improving scores on WISE and WorldGenBench-Humanities.

  6. OmniPhys: Knowledge-Graph-Driven Benchmarking and Collective Optimization for Physical Commonsense in Text-to-Image Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A knowledge-graph benchmark (OmniPhys, 1,551 prompts, 14 physics knowledge points) and a batch-feedback prompt optimizer (OmniPrompt) improve measured physical consistency of text-to-image models by 0.01–0.03 Joint Sc...

  7. AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    AIGI-Holmes combines visual expert pretraining, SFT on explanation data, and direct preference optimization to deliver human-verifiable explanations and top detection accuracy on unseen AI generators.

  8. OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    OneIG-Bench introduces a 2,440-prompt, six-dimension benchmark with automated metrics for text-to-image models, covering alignment, text, reasoning, style, and diversity in English and Chinese.

  9. MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    MMIG-Bench is a unified benchmark of 4,850 prompts and 1,750 reference images with a three-level evaluation suite, including the VQA-based Aspect Matching Score that correlates with human ratings.

  10. Align Beyond Prompts: Evaluating World Knowledge Alignment in Text-to-Image Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    ABP evaluates and improves how well text-to-image models render implicit real-world knowledge.

Pith tools