REVIEW 13 cited by
TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Despite thousands of researchers, engineers, and artists actively working on improving text-to-image generation models, systems often fail to produce images that accurately align with the text inputs. We introduce TIFA (Text-to-Image Faithfulness evaluation with question Answering), an automatic evaluation metric that measures the faithfulness of a generated image to its text input via visual question answering (VQA). Specifically, given a text input, we automatically generate several question-answer pairs using a language model. We calculate image faithfulness by checking whether existing VQA models can answer these questions using the generated image. TIFA is a reference-free metric that allows for fine-grained and interpretable evaluations of generated images. TIFA also has better correlations with human judgments than existing metrics. Based on this approach, we introduce TIFA v1.0, a benchmark consisting of 4K diverse text inputs and 25K questions across 12 categories (object, counting, etc.). We present a comprehensive evaluation of existing text-to-image models using TIFA v1.0 and highlight the limitations and challenges of current models. For instance, we find that current text-to-image models, despite doing well on color and material, still struggle in counting, spatial relations, and composing multiple objects. We hope our benchmark will help carefully measure the research progress in text-to-image synthesis and provide valuable insights for further research.
Forward citations
Cited by 13 Pith papers
-
Poplar: A Scalable Pipeline for Human-Centric Image Dataset Synthesis
Poplar-9K is a curated human-centric image dataset generated through a reproducible attribute sampling, diffusion rendering, and vision-language inspection pipeline, with 9,401 accepted pairs from 11,765 candidates.
-
Discovering Divergent Representations between Text-to-Image Models
An evolutionary algorithm discovers visual attributes that appear in one text-to-image model's outputs but not another's, and identifies the prompt concepts that trigger them.
-
Understanding and evaluating computer vision models through the lens of counterfactuals
Counterfactual-based methods for concept attribution in classifiers and for dynamic bias evaluation and mitigation in text-to-image models.
-
Disentanglement in T-space for Faster and Distributed Training of Diffusion Models with Fewer Latent-states
With a carefully selected noise schedule, diffusion models trained with as few as 32 latent states, or composed from single-state models, match 1,000-state training and converge 4-6x faster.
-
DIMCIM: A Quantitative Evaluation Framework for Default-mode Diversity and Generalization in Text-to-Image Generative Models
A new evaluation framework, DIMCIM, measures default-mode diversity and prompted generalization in text-to-image models, finding a scale trade-off and a 0.85 correlation between default diversity and training data diversity.
-
Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning
CoCA redistributes a single final image reward across denoising steps using cosine similarity between intermediate and final latents, improving RL fine-tuning sample efficiency on four human preference rewards.
-
TNG-CLIP:Training-Time Negation Data Generation for Negation Awareness of CLIP
A training-time pipeline that synthesizes diverse negation captions from batch neighbors improves CLIP's negation accuracy on matching and generation benchmarks, and a new NEG-TTOI benchmark measures negation handling...
-
What makes a good metric? Evaluating automatic metrics for text-to-image consistency
None of the four tested text-to-image consistency metrics satisfies all proposed validity criteria, and the VQA-based metrics appear to rely largely on text priors such as yes-bias.
-
Efficient Scaling of Diffusion Transformers for Text-to-Image Generation
Controlled scaling shows a 2.3B self-attention U-ViT matches or slightly outperforms SDXL U-Net and larger cross-attention DiT variants, while long captions and larger datasets improve text-image alignment.
-
A Reconstruction-Based Framework for Caption Evaluation Beyond Reference Captions
Reference-free caption quality is scored by the downstream vision-language accuracy of a caption-conditioned reconstructed image, via a new CTTD benchmark.
-
AIGVE-MACS: Unified Multi-Aspect Commenting and Scoring Model for AI-Generated Video Evaluation
A finetuned vision-language model jointly predicts nine aspect scores and written comments for AI-generated videos, with a new benchmark and claims of state-of-the-art alignment with human judgment.
-
Evaluating Hallucination in Text-to-Image Diffusion Models with Scene-Graph based Question-Answering Agent
A scene-graph plus LLM question-answering pipeline scores text-to-image faithfulness and categorizes hallucination types, but the measured correlation gain over prior metrics is small and not statistically tested.
-
Towards Evaluating Robustness of Prompt Adherence in Text to Image Models
New benchmark results show that Stable Diffusion 3.x and Janus Pro models struggle to place simple geometric shapes in the correct image quadrant, with best F1 scores around 0.41 to 0.5.
Discussion (0). Continue with ORCID to comment.