{"id":"42bec3d5-c99d-4715-ab63-7eb7419fda90","arxiv_id":"2506.08480","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Current image-text alignment metrics, including CLIPScore and DSGScore, produce unstable model rankings under random seeds and are highly sensitive to tiny image perturbations.","lead":"This paper argues that automatic scores for how well text-to-image models follow prompts should be judged by robustness and statistical significance, not only by agreement with human ratings. It shows that three common metrics give unstable rankings across random seeds and react strongly to tiny pixel changes, and asks researchers to treat such scores with caution.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Robustness-to-randomness argument conflates generation noise with metric unreliability: Table 1's rank flips may reflect sampling variability; the paper never checks whether flipped pairs have statistically indistinguishable means.","rationale":"The reader's weakest assumption correctly identifies the normative character of the perturbation-robustness criterion, but the more load-bearing vulnerability is in the randomness-robustness argument, which underpins Takeaway 1. The paper's definition of Robustness to Randomness makes a category error: it attributes seed-to-seed ranking variation to the metric even though the metric is deterministic and the generated images differ across seeds. A metric that perfectly measures alignment will still produce different averages on different random samples of images; requiring rank invariance across any three seeds is not a standard property of a trustworthy metric. The paper's own paired t-test machinery could have been used to show whether the flips in Table 1 are statistically meaningful, but it is not. This concern does not invalidate the paper's conceptual contribution or the perturbation-robustness experiments, which rest on a separate assumption about visual similarity. It does, however, weaken the strongest claim about CLIPScore and DSGScore. Because the paper is otherwise worth conditional acceptance and this issue could be addressed with additional experiments and more careful framing, I do not move the verdict; I would hold the authors to the concrete check above before accepting the randomness-based conclusion.","tokens_in":6712,"tokens_out":6934,"duration_ms":93042,"concrete_test":"On the same 1000 MSCOCO prompts, run CLIPScore over 30 independent generation seeds for all four models and compute, for each model pair, a bootstrap 95% confidence interval for the mean score difference and the fraction of seeds in which each pairwise ordering is observed. If every pairwise reversal occurs only when the confidence interval includes zero, then Table 1's flips are attributable to sampling variability of the stochastic generator rather than to a robustness defect of the metric; Takeaway 1 would need to be withdrawn or reformulated as a requirement to report confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main empirical support for the randomness half of the central claim is Section 2.2's 'Robustness to Randomness' and Table 1, but the argument treats finite-sample generation variability as if it were a defect of the metric. Section 2.2 states that because diffusion models depend on sampled noise, 'a trustworthy evaluation framework must maintain consistent model rankings despite this randomness.' Yet the metric J is deterministic given the image-text pair (p_i, I_i); changing the generation seed changes the images I_i, so the scores are evaluating different content, not the same content under metric noise. A perfect metric can rank SD-XL above Pixart on one 1000-image sample and below it on another independent sample if the true mean difference is small. Table 1 contains exactly such a near-tie: CLIPScore ranks Pixart (25.71) over SD1.5 (25.58) at seed 3407, but SD1.5 (25.76) over Pixart (25.75) at seed 5096. The paper's own Significance analysis in Section 4.2 provides paired t-tests that could determine whether such flips are meaningful, but the paper never uses those tests to interpret the rank reversals in Table 1. Therefore Takeaway 1, that CLIPScore and DSGScore are less trustworthy because they produce inconsistent rankings, is not established: the observed inconsistency may be a property of using only three seeds, not of the metrics themselves.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that trustworthiness of automatic image-text alignment metrics should be assessed not only by correlation with human judgments, but also by two additional properties: Robustness (stability under random seeds and under small image perturbations) and Significance (whether observed score differences reflect meaningful model superiority). The authors evaluate three metrics (CLIPScore, VQAScore, DSGScore) on four diffusion models (SD3, SD-XL, SD1.5, PixArt) using 1000 MSCOCO prompts and three random seeds. They report that CLIPScore and DSGScore change model rankings across seeds, that all three metrics exhibit large maximum score gaps under a one-pixel perturbation, and that paired t-tests can declare statistical significance even when the dominance ratio is near 50%. The paper concludes with recommendations for future evaluation practices.","tokens_in":6965,"tokens_out":2378,"duration_ms":29982,"significance":"If the empirical claims were fully established, this would be a useful contribution: the field mostly validates metrics against human ratings and largely ignores statistical stability and perturbation sensitivity. The paper also honestly states its main limitation (no improved evaluation framework is proposed). The strengths are the clean formulation of the two properties and the use of standard statistical tools (paired t-test, dominance ratio). However, the evidence base is thin: three seeds, one perturbation type, one generator for the perturbation analysis, and no error bars. The central claim about ranking instability currently conflates metric reliability with sampling variability, so the significance of the contribution is conditional on resolving that issue.","major_comments":[{"comment":"The argument that rank flips demonstrate metric unreliability is not established because the paper does not test whether the observed mean differences are statistically distinguishable. For instance, CLIPScore with seed 3407 gives Pixart 25.71 vs SD1.5 25.58, and seed 5096 gives SD1.5 25.76 vs Pixart 25.75; these could easily be near-ties. The paired t-test machinery introduced in Section 4.2 is never applied to the rank reversals in Table 1, so the flips may reflect generation sampling variability rather than inconsistency of the metric. Takeaway 1 therefore overstates what the data show.","section":"Section 2.2 and Table 1"},{"comment":"The robustness-to-randomness analysis uses only three seeds and reports no error bars or confidence intervals. The claim that 'significant robustness failures' occur even at 1000 prompts is not supported without a statistical model of ranking instability. With only three seeds, the probability of observing a rank flip under a true near-tie is high, so the observed flips do not imply a defect of CLIPScore or DSGScore. The paper should include more seeds (e.g., at least 10), bootstrap intervals on the mean scores, or a formal test of whether the ranking distribution is stable.","section":"Section 2.2 and Table 1"},{"comment":"The perturbation-based robustness conclusion depends entirely on the assumption that visually similar images should receive close scores. The paper states this as a 'fundamental assumption' but does not validate it against other plausible notions of robustness, nor does it compare with other minimal perturbations (e.g., -1 pixel shift, small blur, or JPEG compression). The large maximum gaps in Table 2 (e.g., DSGScore 50.00, VQAScore 10.36) could be artifacts of categorical or thresholded outputs of the metrics rather than a general robustness failure. Without additional perturbation families and a justification for why the '+1' direction is representative, the claim that this reveals 'potential for metric exploitation' is not fully supported.","section":"Section 2.2, Eqs. (1)-(2), and Table 2"}],"minor_comments":[{"comment":"The selection of the 1000 MSCOCO validation prompts is not described; no random seed or sampling procedure is given, which impedes reproducibility.","section":"Section 3"},{"comment":"The image perturbation experiment uses only SD-3 and a single fixed seed (42). A second generator or an additional seed would make the conclusion that 'all three metrics fail' more robust.","section":"Section 4.1 and Table 2"},{"comment":"Naming is inconsistent: 'SD-3' and 'SD3' both appear, and Takeaway 2 says 'worse case' instead of 'worst case'.","section":"Throughout"},{"comment":"The heatmaps lack axis labels and a clear caption; it is difficult to map rows and columns to the four models without consulting Table 1.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is worth publishing after the robustness-to-randomness claim is statistically grounded. The current evidence for Takeaway 1 is the weakest link; the perturbation analysis is more direct but also needs a defense of the normative assumption. The paper is within the scope of the journal and does not appear to hide limitations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper gets at something real: standard text-to-image evaluation practice often reports a single ranking of models, and this paper shows that ranking can flip across three random seeds and that a one-unit pixel perturbation can produce large score swings. Those observations are worth having. The demonstrations in Tables 1 and 2—CLIPScore and DSGScore swapping model ranks across seeds, and all three metrics showing large worst-case gaps under a simple perturbation—are new, and the dominance-ratio analysis is a useful reminder that a statistically significant mean advantage does not imply that the better model wins most prompts. The paper also deserves credit for stating two properties (robustness and significance) clearly and for honestly noting that it does not propose a fix. The soft spots are real, though. The stress-test note is right: the rank flips in Table 1 may be sampling variability rather than metric unreliability. The metric is deterministic given the generated image, and the images differ across seeds. With only three seeds and near-tied means, a perfect metric could produce such flips. The paper's own paired t-tests could adjudicate, but they are never linked to the rank reversals. The perturbation assumption—that visually similar images should receive close scores—is stated but not defended beyond that, and the 'one-unit pixel shift' is a global +1 shift, not a localized or semantically neutral perturbation; the manual check is not a substitute for a controlled calibration. No code or data is provided, the perturbation analysis lacks error bars and uses one generator, and the significance findings are mathematically expected with N=1000 (small effects become significant). The paper would be stronger if it compared the observed flip rate to what a stable metric would do over more seeds and if it reported confidence intervals. The central direction is sound, though. Even if the randomness argument is overstated, the empirical observations suggest that practitioners should use multiple seeds and report variance or confidence intervals. That is a useful, actionable message. The paper deserves serious peer review, but the referee should push for a larger seed count, better linking of significance tests to the robustness results, and more careful language about metric versus protocol reliability.","headline":"Useful empirical warning about seed and perturbation instability in T2I metrics, but the randomness argument overreaches by treating sampling variation as metric failure.","tokens_in":630,"tokens_out":1313,"would_cite":true,"duration_ms":47230,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current image-text alignment metrics do not reliably rank text-to-image models, because their scores shift under random seeds and tiny pixel changes, and small metric gaps can be statistically significant without meaning the winner is…","keywords":["image-text alignment","text-to-image generation","evaluation metrics","robustness","statistical significance","dominance ratio","CLIPScore","VQAScore"],"falsifier":"Run the three metrics over the same 1,000 prompts and four models with 100 random seeds instead of 3; if the pairwise model rankings stay fixed in nearly all cases, the claimed seed-robustness failure disappears. Similarly, if perturbing each image by plus-one pixel using many different starting offsets produces negligible maximum gaps, the claimed worst-case sensitivity is an artifact of the specific perturbation.","tokens_in":6507,"feed_emoji":"🖼️","tokens_out":4954,"duration_ms":52886,"temperature":0.7,"pith_summary":"Text-to-image evaluation is usually judged only by how well it agrees with human ratings. This paper argues that a trustworthy evaluation must also be stable under the randomness of diffusion generation and under tiny image changes, and that its reported differences must reflect actual per-prompt superiority. Testing CLIPScore, VQAScore, and DSGScore on four diffusion models with 1,000 MSCOCO prompts, the paper finds that CLIPScore and DSGScore change model rankings across random seeds, and all three metrics show large worst-case score jumps when a single pixel value is increased by one. It also finds that a score gap can be statistically significant even when the 'better' model only wins about 60% of prompts, and sometimes less. The contribution is diagnostic: the paper sets out two properties that future evaluation frameworks should satisfy, rather than proposing a new metric.","feed_headline":"Image-text scores flip rankings across random seeds","feed_subtitle":"One-pixel edits also swing scores; the paper makes robustness and significance required checks for evaluation.","key_machinery":"The carrying device is a two-property evaluation checklist applied to any metric $J$. Robustness is tested in two ways: running generation over three random seeds and checking whether model rankings stay fixed, and computing the absolute difference $|J(p_i, I_i) - J(p_i, I'_i)|$ for a perturbation that adds 1 to every non-255 pixel value. Significance is tested with a paired t-test over the per-prompt score sets and a dominance ratio $R = \\frac{1}{N} \\sum_{i=1}^N \\mathbb{I}[s^i_{M_1} > s^i_{M_2}]$, the empirical probability that one model beats another prompt-by-prompt. These tools turn 'is this metric trustworthy' into measurable questions about ranking stability, worst-case score gaps, and per-prompt win rates.","core_discovery":"The paper's central claim is that current mainstream image-text alignment metrics are not fully trustworthy evaluators because they fail two basic properties. Robustness requires that evaluation results stay consistent when the input is reasonably perturbed: under different random seeds of the same diffusion model, and under a one-unit pixel-value change to the generated image. Significance requires that an observed score difference reflect a meaningful, per-prompt superiority rather than only a statistical artifact. The empirical demonstration is that CLIPScore and DSGScore produce different model rankings for different seeds, and that CLIPScore, VQAScore, and DSGScore all have maximum absolute score gaps of 7.30, 10.36, and 50.00 points respectively under a visually imperceptible perturbation. Paired t-tests show small score gaps can be statistically significant, but dominance ratios show the 'better' model wins only about 60% of prompts at best, and sometimes less.","pith_inferences":["The paper tests three metrics; the same failure modes likely extend to other model-based and component-based metrics, since they share similar embedding or VQA backbones.","A practical repair would be to report worst-case perturbation gaps as error bars, or to average scores over several perturbed copies of each image.","The dominance-ratio result implies that claims like 'model A beats model B by 5 points' should be rephrased as 'model A wins on about 60% of prompts,' which changes how progress in text-to-image generation is discussed.","The one-pixel perturbation is a lower bound on perceptual sensitivity; adversarial perturbations tuned per image would likely produce even larger score swings, so the reported gaps are conservative."],"forward_implications":["Model rankings from CLIPScore and DSGScore should be reported with variance across seeds; a single seed can be misleading.","Any metric used for text-to-image comparison should be stress-tested with the one-pixel perturbation before its rankings are trusted.","A statistically significant metric gap should not be read as a per-prompt win; dominance ratios should accompany p-values.","Evaluation conclusions drawn from one generated image per prompt are especially fragile because diffusion randomness alone can flip rankings.","Future evaluation frameworks should treat robustness and significance as design requirements, not optional diagnostics."],"supporting_citations":[{"why":"Supplies CLIPScore, the embedding-cosine metric whose seed-inconsistent rankings are a central failure case.","marker":"Hessel et al., 2021"},{"why":"Supplies VQAScore, the VQA-logit metric whose worst-case perturbation gap is a central failure case.","marker":"Lin et al., 2024"},{"why":"Supplies DSGScore, the scene-graph decomposition metric whose seed-inconsistent rankings and 50-point perturbation gap are central failures.","marker":"Cho et al., 2023"},{"why":"Supplies Stable-Diffusion-3, the model used in the perturbation experiment and one of four ranked models.","marker":"Esser et al., 2024"},{"why":"Supplies Stable-Diffusion-XL, one of the four models whose ranking shifts under different seeds.","marker":"Podell et al., 2023"},{"why":"Supplies Stable-Diffusion-1.5, one of the ranked models.","marker":"Rombach et al., 2022"},{"why":"Supplies PixArt-Sigma-XL, one of the ranked models.","marker":"Chen et al., 2024a"},{"why":"Supplies the MSCOCO validation set from which the 1,000 evaluation prompts are selected.","marker":"Lin et al., 2014"}],"fun_headline_variants":["Image-text metrics flip rankings across seeds","One-pixel changes swing image-text alignment scores","Current image-text eval fails robustness and significance","Seeds and pixel edits expose image-text eval unreliability","Rethinking image-text alignment: score differences need significance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central premise is that images which look the same to a human must receive close evaluation scores, so a one-unit pixel shift is treated as a 'reasonable perturbation' that should not change a metric's assessment.","fun_headline_variants_meta":{"raw":{"variants":["Image-text metrics flip rankings across seeds","One-pixel changes swing image-text alignment scores","Current image-text eval fails robustness and significance","Seeds and pixel edits expose image-text eval unreliability","Rethinking image-text alignment: score differences need significance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000711,"raw_usage":{"total_tokens":3131,"prompt_tokens":807,"completion_tokens":2324,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":423,"completion_tokens_details":{"reasoning_tokens":2252}},"tokens_in":423,"tokens_out":2324,"duration_ms":19720,"temperature":1.0,"reasoning_tokens":2252,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:09:18.579943+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the three metrics over the same 1,000 prompts and four models with 100 random seeds instead of 3; if the pairwise model rankings stay fixed in nearly all cases, the claimed seed-robustness failure disappears. Similarly, if perturbing each image by plus-one pixel using many different starting offsets produces negligible maximum gaps, the claimed worst-case sensitivity is an artifact of the specific perturbation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies CLIPScore, the embedding-cosine metric whose seed-inconsistent rankings are a central failure case."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Stable-Diffusion-3, the model used in the perturbation experiment and one of four ranked models."}],"review_version":1}