REVIEW 3 major objections 5 minor 1 cited by
CompAlign: Improving Compositional Text-to-Image Generation with a Complex Benchmark and Fine-Grained Feedback
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Decomposing complex prompts into atomic yes/no questions answered by a multimodal LLM yields an interpretable compositional-accuracy metric, and turning that metric into win/lose preference labels improves open-weight diffusion models on…
desk verdict A genuinely harder compositional T2I benchmark with an interpretable atomic evaluation framework, but the alignment gains rest on an unvalidated MLLM judge that is used as both training reward and evaluation metric. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the atomic sub-question. Each CompAlign prompt is decomposed into n sub-questions, one per entity, each checking presence, attribute, and 3D position (for example, "Is there a green eraser on the left of the second row?"). A multimodal LLM returns a binary yes or no for each, and the Aggregated Compositional Accuracy is the fraction of yes answers. That same binary signal is thresholded to a win or lose label (win if ACA is at least 0.5) and fed into a KL-regularized binary preference objective, Equations 2 and 3, that updates the diffusion model's sampling policy.
What would settle it
Take a random sample of CompAlign prompts, have human annotators independently answer the same atomic questions, and compute agreement with gpt-4o-mini's answers; if agreement falls below roughly 85 percent, both the ACA metric and the preference labels derived from it lose their grounding. A cheaper probe is to test whether the judge credits an object as 'on the left' when that object also appears on the right side of the image.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a benchmark built around 3+ subjects, five escalating 3D layouts, and natural attribute bindings separates current text-to-image models sharply, with closed-source systems leading and U-Net diffusion models falling below 50% average accuracy, and that the gap is actionable: training SD1.5 and SD2 with binary preference labels derived from CompQuest's atomic feedback raises average compositional accuracy from 43.31 to 46.94 and from 46.28 to 53.08 respectively, beating checkpoints trained through the previous benchmark's pipeline. A secondary claim is that CompQuest's decompose-then-verify design is more interpretable and robust than direct scalar scoring by a multimodal LLM or VQA-based counting.
Load-bearing premise
The entire argument depends on gpt-4o-mini's binary answers actually tracking whether each entity, its attribute, and its position are correct in the generated image; the paper offers no human validation of this judge, and its own naturalness analysis shows that high scores can be earned by scenes that look like cut-and-paste collages.
Editorial extensions
If this is right
- Model performance degrades consistently as 3D-spatial layouts grow from one row of two subjects to two rows of three, so composition difficulty is monotone in layout complexity.
- Closed-source commercial models outperform open-weight models by a wide margin on CompAlign, with the best open-weight model at 83.92% and the best closed-source model at 93.51%.
- Fine-tuning with CompAlign feedback improves both SD1.5 and SD2 overall, with the largest gains on two-row spatial configurations such as +11.11% in the 2-row-by-1-subject setting, while checkpoints from the previous benchmark often underperform the base model.
- The alignment pipeline is data-agnostic and can be augmented with generations from stronger models such as SD3.5 and DALL-E 3 to bootstrap weaker diffusion models.
- Because CompQuest yields a per-image score rather than pairwise comparisons, the same pipeline can scale to larger training sets without human preference annotation.
Reading between the lines
- Since the judge's binary labels are the only training signal, the same recipe could be repurposed to optimize other decomposable prompt properties, such as style consistency or object counts, simply by rewriting the atomic questions.
- If the judge's yes/no answers are miscalibrated for rare objects or attributes, the preference signal would amplify that bias; checking whether alignment gains survive when the judge is swapped or human-validated would separate benchmark difficulty from judge artifacts.
- The paper's own naturalness analysis shows that collage-like scenes can score high ACA, so folding a naturalness sub-question into the atomic set would make the metric harder to game and could change which models rank best.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CompAlign, a 900-prompt benchmark for compositional text-to-image generation, emphasizing 3+ subjects, numerical/3D-spatial relations, and attribute bindings, together with CompQuest, an evaluation framework that decomposes prompts into atomic binary questions answered by an MLLM (gpt-4o-mini) and aggregates them into an ACA score. The authors evaluate 9 T2I models on CompAlign, finding that closed-source models outperform open-source ones and that performance degrades with spatial complexity. They then use ACA-based per-image win/lose signals (Eq. 2) to fine-tune SD1.5 and SD2 via a diffusion-alignment objective, reporting ACA improvements (SD1.5: 43.31 to 46.94; SD2: 46.28 to 53.08) that outperform the best CompBench++ checkpoint. The paper also includes an appendix case study showing that high-ACA images can be unnatural collages, and naturalness scores for five strong models.
Significance. If the ACA metric and the alignment results are valid, CompAlign would be a useful, more demanding benchmark for compositional generation, and the proposed feedback-driven alignment would offer a scalable, interpretable way to improve diffusion models. The benchmark construction is careful: 900 prompts balanced across five 3D-spatial configurations and multiple generation categories, with prompt templates that combine attributes and spatial layouts. The interpretability of atomic decomposition is a genuine strength, as is the reproducible, automated nature of the pipeline. However, the paper's central contribution rests on the accuracy of the gpt-4o-mini judge, which is used both as the training reward and as the evaluation metric. The paper provides no human validation, no agreement analysis, and its own Appendix D demonstrates that the metric can be gamed by models that produce collaged, unnatural scenes. Until the judge is validated against human judgments or an independent metric, the reported ACA numbers, including the claimed improvements, are not fully trustworthy evidence of compositional ability.
major comments (3)
- [§3.2, §4.1, §5.2 (Eq. 2 and Table 2)] The same MLLM (gpt-4o-mini) is used to construct the per-image win/lose preference labels in Eq. (2) and to compute the reported ACA evaluation scores in Table 2. The paper provides no human agreement analysis, no calibration against an independent metric, and no error bars. Appendix D shows that models with near-perfect ACA can produce unnatural, collaged scenes, which is direct evidence that the judge can be gamed. Consequently, the reported improvements (e.g., SD2 from 46.28 to 53.08) may partly measure how well the fine-tuned model satisfies this particular judge rather than genuine compositional ability. The authors should validate the atomic binary feedback against human judgments on a sampled subset of CompAlign and, ideally, evaluate the aligned models with an independent metric (e.g., human preference or a second, different judge). This validation is load-bearing because it affects both the benchmark's evaluation claim and the alignment method's effectiveness claim.
- [§5.2, Table 2] The claim that CompAlign 'effectively and consistently improves the performance for both base diffusion models' is contradicted by the table itself. For SD1.5, object_color drops from 34.78 to 31.00 and 2rows×2sub drops from 35.00 to 31.67; for SD2, object_color_bathroom and object_color_kitchen show no improvement (both 57.14 and 60.00, respectively). The paper should either soften the consistency claim or explain why these subcategories are not expected to improve. In addition, the evaluation of different subcategories is based on small samples (50 entries for people_only and bathroom/kitchen categories), so the reported percentage differences may not be statistically meaningful. The authors should provide confidence intervals or significance tests.
- [§3.3, Eq. (1), Appendix D] The ACA metric in Eq. (1) aggregates independent yes/no judgments about each entity's presence, attribute, and position. This formulation cannot capture compositional coherence, such as whether multiple entities appear together in a single unified scene rather than as separately positioned objects or collaged pieces. Appendix D demonstrates exactly this failure mode: models can achieve high ACA by placing entities in separate scenes while violating naturalness. The authors should either extend the metric to include relational or scene-level judgments, or explicitly state that ACA measures per-entity accuracy only and does not claim to measure overall image-prompt alignment. As written, the abstract and Section 3.3 describe ACA as a measure of 'alignment between generated images and compositional prompts,' which overstates what the metric captures.
minor comments (5)
- [§4.1] The threshold tau in Eq. (2) is introduced as adjustable, but the paper never reports a sensitivity analysis or states how the specific value 0.5 was chosen. Adding a short analysis of how tau affects the win/lose balance and downstream ACA would make the method more reproducible.
- [§5.2 and Table 2] The comparison baselines are limited to CompBench++ checkpoints provided by Huang et al. Several relevant alignment methods for diffusion models (e.g., D3PO, DPO-based diffusion alignment, ImageReward, DreamSync) are discussed in Section 6.2 but not compared experimentally. Including at least one additional baseline would strengthen the claim that the proposed approach outperforms previous methods.
- [Appendix D] The naturalness analysis is a valuable addition, but it relies solely on gpt-4o judgments without human validation. Since the main message of Appendix D is that automated metrics can be fooled, it would be more convincing to include human naturalness ratings or show at least a few human examples.
- [Throughout] There are several typos and formatting issues: 'leverageing' in Section 3, 'we proposeCompQuest' in Section 3, 'compositionally' in Section 5.2 and Appendix D, and missing spaces around references in Sections 5.2 and 6.1. These should be corrected.
- [§2.2, Table 4] The paper states that the 90-10 train-test split is 'balanced across sub-categories,' but does not describe the split procedure. For example, does each 3D-spatial configuration receive exactly 162 training and 18 test prompts? Adding the split details would facilitate reproduction and ensure no data leakage through prompt templates.
Circularity Check
Training reward and evaluation metric are the same ACA function from the same unvalidated gpt-4o-mini judge, so the reported post-alignment gains measure fit to the judge, not independent compositional accuracy; Appendix D concedes the metric can be hacked.
-
self definitional
[Section 3.3 Eq. (1); Section 4.1 Eq. (2); Section 5.2 Implementation Details/Evaluation Results]
"ACA is computed as the proportion of correctly-depicted entities, measured by the percentage of atomic questions with positive judgements by the MLLM. ... We utilize CompQuest's ACA scores to construct per-image binary preference signals. Specifically, we define an image to be a 'win' if its ACA score exceeds a predefined threshold τ: w(x) = 1 if ACA(x) ≥ τ, −1 otherwise. (2) ... At inference time, we report ACA outcomes on the held-out test set."
The reported outcome is the same scalar that defines the training signal: Eq. (2) turns ACA(x) directly into the win/lose preference w(x), and the alignment objective (3) is optimized against these preferences. The 'compositional accuracy' improvements in Table 2 are then measured by the same ACA function from the same gpt-4o-mini judge (Eq. 1). So a post-alignment ACA gain is, by construction, evidence that the model learned to answer the judge's binary questions, not independent evidence of compositional ability. The paper does not validate the judge against human judgments, and its own Appendix D shows that high-ACA images can be unnatural collages ('this might hack the accuracy score'), confirming that the shared metric can be gamed.
full rationale
The core benchmark and prompt construction are self-contained: the 900 prompts are manually templated from explicit spatial/attribute lists, and the comparison against CompBench++ checkpoints uses the same metric for both. There is no dependence on a self-citation chain or an imported uniqueness theorem. The circularity that does exist is in the alignment experiment: the training reward (Eq. 2) is a thresholded version of ACA, and the reported evaluation (Section 5.2) is ACA from the same CompQuest/gpt-4o-mini pipeline. Therefore the 'improvements' (SD1.5 43.31→46.94; SD2 46.28→53.08) are gains on the very function being optimized. The paper provides no human agreement or independent calibration for the judge, and its own Appendix D concedes that a model can 'hack' the automated score by producing separate collaged scenes, with naturalness scores (Table 8) far below ACA. This makes the central training-evaluation loop partially circular: it demonstrates optimization of the judge's binary feedback, not independently established compositional ability. Because the model is evaluated on a held-out test split and the judge is an external MLLM rather than a fitted parameter, the circularity is partial, not total, supporting score 6.
Assumptions & free parameters
free parameters (1)
- win threshold tau =
0.5
assumptions (3)
- domain assumption The MLLM (gpt-4o-mini) provides accurate binary feedback for atomic compositional questions.
- domain assumption The heuristic prompt templates yield natural and unambiguous compositional prompts.
- standard math The optimization objective from Li et al. [16] and KTO (Ethayarajh et al.) is a valid training method for diffusion models.
Cite this review
Pith. "Pith review of CompAlign: Improving Compositional Text-to-Image Generation with a Complex Benchmark and Fine-Grained Feedback." pith.science (2026). https://pith.science/paper/72SCPPF3
@misc{pith2026250511178,
author = {Pith},
title = {Pith review of: CompAlign: Improving Compositional Text-to-Image Generation with a Complex Benchmark and Fine-Grained Feedback},
year = {2026},
howpublished = {\url{https://pith.science/paper/72SCPPF3}},
note = {Machine review of arXiv:2505.11178}
}
read the original abstract
State-of-the-art T2I models are capable of generating high-resolution images given textual prompts. However, they still struggle with accurately depicting compositional scenes that specify multiple objects, attributes, and spatial relations. We present CompAlign, a challenging benchmark with an emphasis on assessing the depiction of 3D-spatial relationships, for evaluating and improving models on compositional image generation. CompAlign consists of 900 complex multi-subject image generation prompts that combine numerical and 3D-spatial relationships with varied attribute bindings. Our benchmark is remarkably challenging, incorporating generation tasks with 3+ generation subjects with complex 3D-spatial relationships. Additionally, we propose CompQuest, an interpretable and accurate evaluation framework that decomposes complex prompts into atomic sub-questions, then utilizes a MLLM to provide fine-grained binary feedback on the correctness of each aspect of generation elements in model-generated images. This enables precise quantification of alignment between generated images and compositional prompts. Furthermore, we propose an alignment framework that uses CompQuest's feedback as preference signals to improve diffusion models' compositional image generation abilities. Using adjustable per-image preferences, our method is easily scalable and flexible for different tasks. Evaluation of 9 T2I models reveals that: (1) models remarkable struggle more with compositional tasks with more complex 3D-spatial configurations, and (2) a noticeable performance gap exists between open-source accessible models and closed-source commercial models. Further empirical study on using CompAlign for model alignment yield promising results: post-alignment diffusion models achieve remarkable improvements in compositional accuracy, especially on complex generation tasks, outperforming previous approaches.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
Dominant vs. Dominated: Concept-Level Generative Collapse in Diffusion Models
A concept trained on visually uniform images tends to dominate and suppress a second concept in multi-concept text-to-image generation, a failure the authors name DvD.
Reference graph
Works this paper leans on
- [1]
- [2]
-
[3]
M. Chen, I. Laina, and A. Vedaldi. Training-free layout control with cross-attention guidance. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 5343–5353, 2024
work page 2024
- [4]
-
[5]
Esser, S
P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y . Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024
2024
-
[6]
K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024
arXiv 2024
- [7]
-
[8]
Y . Fan, O. Watkins, Y . Du, H. Liu, M. Ryu, C. Boutilier, P. Abbeel, M. Ghavamzadeh, K. Lee, and K. Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36:79858–79885, 2023
2023
Show all 25 references
-
[9]
W. Feng, X. He, T.-J. Fu, V . Jampani, A. Akula, P. Narayana, S. Basu, X. E. Wang, and W. Y . Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05032, 2022
2022 arXiv
-
[10]
Y . Hao, Z. Chi, L. Dong, and F. Wei. Optimizing prompts for text-to-image generation.Advances in Neural Information Processing Systems, 36:66923–66939, 2023
2023
-
[11]
Y . Hu, B. Liu, J. Kasai, Y . Wang, M. Ostendorf, R. Krishna, and N. A. Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20406–20417, 2023
2023
-
[12]
Huang, K
K. Huang, K. Sun, E. Xie, Z. Li, and X. Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation.Advances in Neural Information Processing Systems, 36:78723–78747, 2023
2023
-
[13]
Huang, C
K. Huang, C. Duan, K. Sun, E. Xie, Z. Li, and X. Liu. T2i-compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(5):3563–3579, 2025. doi: 10.1109/TPAMI.2025. 3531907
2025 doi
-
[14]
B. F. Labs. Flux. https://github.com/black-forest-labs/flux , 2024
2024
-
[15]
K. Lee, H. Liu, M. Ryu, O. Watkins, Y . Du, C. Boutilier, P. Abbeel, M. Ghavamzadeh, and S. S. Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192, 2023
2023 arXiv
-
[16]
S. Li, K. Kallidromitis, A. Gokul, Y . Kato, and K. Kozuka. Aligning diffusion models by optimizing human utility. In The Thirty-eighth Annual Conference on Neural Information Processing Systems
-
[17]
L. Lian, B. Li, A. Yala, and T. Darrell. Llm-grounded diffusion: Enhancing prompt un- derstanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305.13655, 2023. 10
2023 arXiv
-
[18]
Dall ·e 3 system card, Oct 2023
OpenAI. Dall ·e 3 system card, Oct 2023. URL https://openai.com/research/ dall-e-3-system-card
2023
-
[19]
D. H. Park, S. Azadi, X. Liu, T. Darrell, and A. Rohrbach. Benchmark for compositional text-to-image synthesis. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL https://datase...
2021
-
[20]
Prabhudesai, A
M. Prabhudesai, A. Goyal, D. Pathak, and K. Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation
-
[21]
Rombach, A
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022
2022
-
[22]
J. Sun, D. Fu, Y . Hu, S. Wang, R. Rassin, D.-C. Juan, D. Alon, C. Herrmann, S. van Steenkiste, R. Krishna, et al. Dreamsync: Aligning text-to-image generation with image understanding feedback. arXiv preprint arXiv:2311.17946, 2023
2023 arXiv
-
[23]
G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023
2023 arXiv
-
[24]
Q. Wu, Y . Liu, H. Zhao, T. Bui, Z. Lin, Y . Zhang, and S. Chang. Harnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7766–7776, 2023
2023
-
[25]
a blue cherry
J. Xu, X. Liu, Y . Wu, Y . Tong, Q. Li, M. Ding, J. Tang, and Y . Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation.Advances in Neural Information Processing Systems, 36:15903–15935, 2023. A Details on Prompt Construction for CompAlign W...
2023
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.