REVIEW 4 major objections 6 minor 14 references
TangPoetryBench: A Multi-Dimensional Benchmark and Rubric-Conditioned Evaluator for Poetry-to-Image Generation
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A new benchmark shows text-to-image models render a poem's scene but miss its emotion, and an open evaluator can grade the gap.
desk verdict A genuinely useful benchmark and a clever rubric-conditioned evaluator, but the headline parity claim needs stronger ground-truth uncertainty analysis before I'd trust it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the benchmark's two-stage annotation protocol: Phase 1 recognizability (a viewer identifies the correct poem from the image alone among four candidates) and Phase 2 per-dimension scoring on ten dimensions from safety and technical quality to core imagery and emotional resonance. The diagnostic that carries the argument is the depict-to-evoke gap, the consistent drop between near-ceiling surface dimensions (scene, cultural coherence, style) and meaning-bearing ones (core imagery, emotion). The mechanism attributed to this gap is face avoidance: models place figures in the right scene but omit a readable expression, and the rate of this failure tracks each model's emotion score. The evaluator argument is carried by rubric conditioning: PAE takes an image, its poem, and a written rubric defining score anchors, and outputs per-dimension scores after supervised fine-tuning plus a reinforcement stage on a Qwen3-VL-8B backbone.
What would settle it
Re-annotate a random sample of the 1,050 single-rated images with at least three raters and recompute the per-image quality scores; if PAE's per-dimension mean absolute error (0.129) and per-poem ranking τ (0.431) degrade substantially against this multi-rater ground truth, the parity-with-Claude claim is an artifact of annotation noise.
Extended reading notes
Core claim
On its own terms, the paper's central discovery is that current text-to-image models share a 'depict-to-evoke gap': they score near the ceiling on the visual surface of a Tang poem—scene, cultural coherence, artistic style—but drop sharply on core imagery and more sharply on emotional resonance, the dimension that correlates most with holistic human judgment. The paper attributes this to a concrete rendering failure: models avoid drawing a readable human face, which is the natural carrier of feeling, and the frequency of this avoidance tracks each model's emotion score. The paper further claims that this gap is invisible to standard scalar metrics, which reward literal correspondence and rank the best and worst models almost identically, and that an open, rubric-conditioned evaluator (PAE) reproduces human per-dimension judgments, reaches a per-poem ranking τ of 0.431 against 0.444 for a strong proprietary judge, and transfers to an unseen generator and to the Song Ci tradition.
Load-bearing premise
The quality scores that serve as PAE's training targets and as ground truth in the comparisons rest mostly on a single annotator per image: only 230 of the 1,280 images were multi-rated, and no correction for rater noise is applied to the rest.
Editorial extensions
If this is right
- Standard alignment metrics (CLIPScore, BLIPScore, VQAScore) cannot rank poetry illustrations and should be retired for this task in favor of per-dimension rubric scoring.
- Improving a model's ability to render expressive human faces should directly raise emotional resonance, the dimension that most strongly drives human overall-impression ratings.
- PAE's rubric conditioning means the benchmark can be extended to new dimensions, new generators, and new poetic traditions by supplying a rubric, not by retraining the evaluator.
- For model builders, the data say effort is better spent on core imagery and emotion than on already near-ceiling surface dimensions such as style and safety.
- Because poem difficulty tracks abstraction and allusion rather than length, the hardest poems are the natural target for knowledge-grounded or retrieval-augmented generation.
Reading between the lines
- The face-avoidance finding suggests an intervention the paper does not run: taking a high-scoring image and inserting a legible expressive face (or measuring face detectability) should shift emotional-resonance scores; this would test whether the bottleneck is causal rather than correlational.
- If the depict-to-evoke gap is a real property of current models, it likely reflects training on literal caption-image pairs, so a similar gap should appear in other implicit-meaning generation tasks (proverbs, abstract art, film stills) and could be tested with the same rubric-conditioned recipe.
- Because PAE matches the proprietary judge on ranking but not on absolute per-image error, its practical role is comparative evaluation—ordering models and versions—rather than certifying a single image's quality.
- All reported scores come from one fixed prompt per model, so the benchmark measures each generator at a single operating point; varying prompt specificity could reorder the models, a scoping limit the paper acknowledges and a natural next measurement.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces TangPoetryBench, a benchmark of 1,280 poem-to-image generations (320 Tang poems × 4 T2I models) annotated along ten dimensions, together with a diagnostic analysis of current T2I models and a rubric-conditioned evaluator, PoemAutoEvaluator (PAE). The benchmark uses a two-phase protocol: a poem-recognition phase and a multi-dimensional scoring phase, with 1,527 ratings from 191 annotators, of which 230 images are multi-rated. The diagnostic analysis reports a depict-to-evoke gap: models master visual surface dimensions but score lowest on emotional resonance and overall impression, with the paper attributing the emotion gap to a 'no readable face' failure mode. PAE fine-tunes Qwen3-VL-8B with LoRA and a GRPO stage on the human per-dimension scores, and is evaluated on held-out poems, reaching a per-poem ranking Kendall τ of 0.431, close to zero-shot Claude (0.444), while standard metrics such as CLIPScore, BLIPScore, and VQAScore show weak or no correlation with human ranking. The paper also reports generalization to an unseen generator (Kolors) and to a second poetic tradition (Song Ci, 20 images).
Significance. If the findings hold, TangPoetryBench would be a valuable resource for evaluating culturally grounded and affectively complex text-to-image generation, a domain that existing compositional benchmarks do not address. The manuscript is unusually transparent about its annotation process: it provides a detailed rubric, describes QC filters, adjudicates text-integrity labels, reports inter-annotator agreement, and includes bootstrap stability checks for the model ranking. The release of the benchmark, annotations, and an open evaluator is a practical contribution. The diagnostic claims—particularly the depict-to-evoke gap and the face-avoidance bottleneck—are actionable for model development. However, the strength of the paper's quantitative claims is currently limited because the per-image human scores that serve as PAE's training targets and as the evaluation ground truth are single-rater for 82% of images, no rater-noise floor is reported, and the cross-tradition generalization rests on a very small sample. These issues are fixable with additional analysis and reporting, so the central contributions are defensible but require revision.
major comments (4)
- [§3, Appendix D, Table 5] The per-image quality scores used as PAE's supervised training targets and as the human reference in Table 4 are single-rater for 82% of images (1,050 of 1,280), yet the manuscript reports only within-one-level inter-annotator agreement on the 230 multi-rated images. This statistic is lenient: on the 3–4 level rubrics, a one-level difference can be as large as 0.33, which is much larger than PAE's reported MAE of 0.129. To interpret the headline numbers, the paper must report a rater-noise floor on the multi-rated subset—for example, the mean absolute deviation of single ratings from the multi-rater mean, an ICC, or a weighted kappa—and then compare PAE's and Claude's MAE against that floor. Without this, the claims of 'reproduces human per-dimension judgment' and 'parity with Claude' are not established, because the apparent agreement may be dominated by the noise of the unstable ground truth.
- [§5, Appendix A] The generalization to Song Ci is based on only 20 images, each produced by a single generator, and the appendix does not state how many annotators rated these images or whether the same PAE checkpoint was used zero-shot or after any rubric-specific calibration. The main text claims that extending PAE to a new tradition 'required only a tradition-specific rubric, not retraining,' but the experimental detail needed to verify a zero-shot or minimally adapted transfer is missing. Please report the number of raters per Song Ci image, the exact protocol, and whether any calibration labels or additional fine-tuning were used; with 20 images, the current evidence is too thin to support the generalizable rubric-conditioning claim.
- [§4 ('What All Four Models Share')] The paper's central diagnostic finding—that the emotion bottleneck is caused by models avoiding a readable human face—is supported by a 'no readable face' category that does not appear in the rubric of Table 1 or Appendix C. No annotation protocol, definition, or inter-annotator reliability is given for this category, so the reader cannot tell whether the 15% figure and the per-model rates are reproducible expert judgments or informal post-hoc coding. Please specify who assigned this label, under what instructions, and with what agreement; this is load-bearing because the paper's main actionable guidance about expressive human faces rests on it.
- [Table 4, §5 ('Results')] The parity claim between PAE (τ=0.431) and Claude (τ=0.444) is based on a difference of 0.013, but no confidence intervals or significance tests are reported for any of the Kendall τ values. Given the small number of held-out poems (64 poems, 256 images), the difference between the two models could easily be within sampling noise, and the claim that PAE 'reaches parity' or 'matches' Claude is not yet supported. Please provide bootstrap confidence intervals for τ and, ideally, a paired test across poems, for PAE, the open baseline, and Claude.
minor comments (6)
- [Table 3] The table is garbled in the current text: 'Seedream368 44' and 'MJ 27173' should be formatted as separate columns with clear counts (e.g., Seedream: 36 leakage, 8 fake, 44 total; MJ: 27 leakage, 46 fake, 73 total).
- [§3 ('Annotation and adjudication')] The paper lists the QC filters (response-time filtering, pattern detection, accuracy floor, consistency checks) but does not report how many annotators or ratings were rejected by these filters; adding these counts would help the reader assess the quality of the retained annotations.
- [§3 ('Two reporting metrics')] The term 'Quality' for the unweighted mean over applicable dimensions is easy to confuse with the 'Overall Impression' dimension; consider renaming the aggregate to 'Mean Dimension Score' or 'Aggregate Quality' for clarity.
- [§3 ('Reliability of the model comparison')] The bootstrap resampling over images holds each image's rating fixed and therefore does not capture annotator variability; this should be stated explicitly in the text so the reliability claim is not over-interpreted.
- [§5, Table 4 caption] The caption should explicitly state that PAE is fine-tuned on the benchmark's training poems while the open baseline and Claude are zero-shot; this framing clarifies the comparison and prevents an unfair 'fine-tuned vs. zero-shot' misreading.
- [§1, Data release] The abstract and introduction state that the benchmark, annotations, and evaluator are released, but no URL or repository is provided in the manuscript; please add an availability statement.
Circularity Check
No significant circularity: PAE is trained and evaluated on disjoint poems and on genuinely new generators and a second poetic tradition, so the central claims are supervised generalization claims rather than derivations equivalent to their inputs.
full rationale
PAE's headline claims are not reductions to the benchmark's own inputs. The paper trains PAE on human per-dimension scores, then evaluates it on held-out poems: "We split by poem, not by image, so no poem appears in both training and test." It further tests on an unseen generator "with fresh human labels" and on Song Ci with a tradition-specific rubric. These are ordinary supervised-evaluation and transfer claims, not fitted parameters renamed as predictions. The benchmark's diagnostic findings (the depict-to-evoke gap, abstraction-not-length difficulty, opposite text-failure modes) are summaries of human annotations and cross-model comparisons; they do not depend on PAE. No load-bearing self-citation appears: the authors are independent researchers who cite external tools and prior benchmarks, not their own prior results or a self-authored uniqueness theorem. The one substantive caveat is a measurement-reliability concern rather than circularity: only 230 of 1,280 images are multi-rated, so the per-image human targets for the remaining 82% carry unmodeled single-rater noise, and the paper reports within-one-level agreement rather than ICC or weighted kappa. The paper itself acknowledges that "the residual subjectivity of emotion still sets a natural ceiling on any metric." That limitation affects the strength of the reported MAE and parity comparisons, but it does not make PAE's held-out evaluations equivalent to its training targets by construction.
Assumptions & free parameters
assumptions (4)
- domain assumption Human ratings are a valid ground truth for whether an image captures a poem's meaning, including implicit emotion.
- domain assumption The unweighted mean of applicable dimensions is a valid overall quality score.
- domain assumption One fixed prompt per model is representative of that model's poetry-illustration capability.
- ad hoc to paper The 'no readable face' category is a measurable failure mode for emotion.
Cite this review
Pith. "Pith review of TangPoetryBench: A Multi-Dimensional Benchmark and Rubric-Conditioned Evaluator for Poetry-to-Image Generation." pith.science (2026). https://pith.science/paper/O7ATE7CK
@misc{pith2026260811452,
author = {Pith},
title = {Pith review of: TangPoetryBench: A Multi-Dimensional Benchmark and Rubric-Conditioned Evaluator for Poetry-to-Image Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/O7ATE7CK}},
note = {Machine review of arXiv:2608.11452}
}
read the original abstract
Text-to-image (T2I) models are increasingly asked to illustrate literary and cultural content, yet we cannot measure how well an image renders the meaning of a poem. The task is many-sided: a good illustration must be visually sound, faithful to the poem's imagery and scene, culturally and stylistically apt, free of spurious text, and true to its emotion, and its deepest requirements, imagery and especially implicit emotion, are never stated in the words. Existing metrics (CLIPScore, BLIPScore, VQAScore) reward literal text-image correspondence and so cannot tell whether an illustration succeeds, let alone why, or even separate the best model from the worst. We introduce TangPoetryBench, a multi-dimensional benchmark of 1,280 images (320 classical Chinese Tang poems x 4 state-of-the-art T2I models) with quality-controlled human annotations across ten dimensions. Analyzing this data, we reveal the shared and model-specific strengths and weaknesses of current T2I models, including their ability to evoke a poem's implicit emotion. We further introduce PoemAutoEvaluator (PAE), an open, rubric-conditioned evaluator that reaches parity with a strong proprietary judge (Claude), generalizes to an unseen generator and a second poetic tradition (Song Ci), and lets the benchmark scale to new images without fresh human annotation. We release the benchmark, annotations, and evaluator.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Aghazadeh,A.;andKovashka,A.2025. CAP:Evaluationof PersuasiveandCreativeImageGeneration.InProceedingsof theIEEE/CVFInternationalConferenceonComputerVision (ICCV), 16970–16980. Betker, J.; Goh, G.; Jing, L.; Brooks, T.; Wang, J.; Li, L.; Ouyang,L.;Zhuang,J.;Lee,J.;Guo,Y.;etal.2023. Improv- ingImageGenerationwithBetterCaptions. OpenAItechni- cal report. http...
work page 2025
-
[4]
InAdvances in Neural Information Processing Systems (NeurIPS)
Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. InAdvances in Neural Information Processing Systems (NeurIPS). Ku,M.;Jiang,D.;Wei,C.;Yue,X.;andChen,W.2024. VI- EScore:TowardsExplainableMetricsforConditionalImage Synthesis Evaluation. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (AC...
work page 2024
-
[6]
arXiv preprint arXiv:2307.02971
On the Cultural Gap in Text-to-Image Generation. arXiv preprint arXiv:2307.02971. Malakouti, S.; Gong, B.; and Kovashka, A
-
[10]
arXiv preprint arXiv:2508.18265
InternVL3.5: Advancing Open-Source Mul- timodal Models in Versatility, Reasoning, and Efficiency. arXiv preprint arXiv:2508.18265. Wu,X.;Hao,Y.;Sun,K.;Chen,Y.;Zhu,F.;Zhao,R.;andLi, H.2023.HumanPreferenceScorev2:ASolidBenchmarkfor Evaluating Human Preferences of Text-to-Image Synthesis. arXiv preprint arXiv:2306.09341. Xu, C.; and Zhou, S
arXiv 2023
-
[12]
TCC-Bench: Benchmarking the Traditional Chinese Cul- ture Understanding Capabilities of MLLMs.arXiv preprint arXiv:2505.11275. Yu, J.; Xu, Y.; Koh, J. Y.; Luong, T.; Baid, G.; Wang, Z.; Vasudevan, V.; Ku, A.; Yang, Y.; Ayan, B. K.; et al
-
[13]
Zhang, C.; Feng, X.; Bai, Y.; Du, X.; Hou, J.; Deng, K.; Han, G.; Li, Q.; Wang, B.; Liu, J.; et al
Scalingautoregressivemodelsforcontent-richtext-to-image generation.arXiv preprint arXiv:2206.10789. Zhang, C.; Feng, X.; Bai, Y.; Du, X.; Hou, J.; Deng, K.; Han, G.; Li, Q.; Wang, B.; Liu, J.; et al
-
[14]
Can MLLMs Understand the Deep Implication Behind Chinese Images? InProceedingsofthe63rdAnnualMeetingoftheAssociation for Computational Linguistics, 14369–14402. Appendix A The Two Poetic Traditions TangPoetryBench is built on Tang regulated verse; to test generalizationwealsoevaluateSongCi(Section5).Thetwo traditions share much: both are classical Chinese...
work page 2025
-
[686]
ImageReward:LearningandEvaluating Human Preferences for Text-to-Image Generation
Xu, J.; Liu, X.; Wu, Y.; Tong, Y.; Li, Q.; Ding, M.; Tang, J.; andDong,Y.2023. ImageReward:LearningandEvaluating Human Preferences for Text-to-Image Generation. InAd- vancesinNeuralInformationProcessingSystems(NeurIPS). Xu, P.; Wang, Y.; Zhang, S.; Zhou, X.; Li, X.; Yuan, Y.; Li, F.; Zhou, S.; Wang, X.; Zhang, Y.; and Zhao, H
work page 2023
Show all 14 references
-
[2017]
In Advances in Neural Information Processing Systems
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems. Hu,E.J.;Shen,Y.;Wallis,P.;Allen-Zhu,Z.;Li,Y.;Wang,S.; Wang,L.;andChen,W.2022. LoRA:Low-RankAdaptation of Large Language Models. InInternatio...
2022
-
[2022]
InProceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, 10684– 10695
High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, 10684– 10695. Saharia,C.;Chan,W.;Saxena,S.;Li,L.;Whang,J.;Denton, E. L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.;Sa...
2022 arXiv
-
[2023]
InAdvances in Neural Information Processing Systems (NeurIPS)
T2I- CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-Image Generation. InAdvances in Neural Information Processing Systems (NeurIPS). Jamil, S.; Reddy, B. A.; Kumar, R.; Saha, S.; Goswami, K.; and Joseph, K. J. 2025a. Poemtale diffusion: Minimising in...
-
[2024]
Li, J.; Li, D.; Xiong, C.; and Hoi, S
Genai-bench: Evaluating and improving compositional text- to-visual generation.arXiv preprint arXiv:2406.13743. Li, J.; Li, D.; Xiong, C.; and Hoi, S
-
[2025]
Rombach,R.;Blattmann,A.;Lorenz,D.;Esser,P.;andOm- mer, B
Qwen3-VL Technical Report.arXiv preprint arXiv:2511.21631. Rombach,R.;Blattmann,A.;Lorenz,D.;Esser,P.;andOm- mer, B
-
[2026]
InInternational Conference on Learning Repre- sentations, volume 2026, 136235–136262
Culture in action: Evaluating text-to-image models through social activities. InInternational Conference on Learning Repre- sentations, volume 2026, 136235–136262. Otani, M.; Togashi, R.; Sawai, Y.; Ishigami, R.; Nakashima, Y.; Rahtu, E.; Heikkilä, J.; and Satoh, S
2026
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.