{"id":"23b23803-b7ee-40a0-aa2d-494d2155f655","arxiv_id":"2505.16314","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"The NTIRE 2025 challenge report compares 20 methods for fine-grained text-to-image quality assessment, introduces the EvalMuse-Structure dataset, and finds every participating team outperformed the baselines.","lead":"This paper reports a 2025 competition for judging the quality of AI-generated images made from text prompts, with 20 teams in two tracks. It introduces a new dataset of images with labeled structural flaws and reports that every submitted method beat the official baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Winning alignment team IH-VQA trained on validation-set pseudo-labels while the test set contains part of the validation set; the reported win may reflect train/test leakage rather than generalization.","rationale":"The reader's weakest assumption correctly identified the validation/test overlap. I am partial rather than full agreement because the paper contains a stronger, explicit version of the problem: the winning team does not merely receive leaderboard feedback on the validation set; Section 5.1.1 says it generates pseudo-labels on the validation set and trains on them. If the test set is partly the validation set, that is direct train/test leakage for the top-ranked method. This does not require assuming bad faith; it follows from the protocol as written. The quantitative claims in Table 1 and Section 4 are otherwise internally consistent, and the methods sections give plausible technical descriptions, so I would not reject the report outright. The correct remedy is to make the partition public and recompute the ranking on the disjoint newly annotated portion. If the organizers cannot provide that partition, the headline result should be reported as unverified rather than as a win. Mechanical errors noted by the reader (video-track counts, 10K/12K mismatch, Appendix C title) are real but peripheral; they do not change this assessment.","tokens_in":22606,"tokens_out":9229,"duration_ms":72205,"concrete_test":"Obtain the organizers' partition of the test set into the newly annotated image-text pairs and the validation-derived pairs. Recompute all Table 1 Main Scores, for the 12 teams and the FGA-BLIP2 baseline, using only the newly annotated pairs (excluding every validation-derived image). If IH-VQA remains rank 1 and its margin over the baseline is comparable on this disjoint subset, the leakage concern is resolved; if not, the reported ranking is invalid. As a secondary check, list the validation images used in IH-VQA's pseudo-label augmentation and confirm none appear in the disjoint test partition.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing flaw is direct overlap between training and test data for the top alignment team. Section 3.1 states that the test set is 'additional image-text pairs... combined with a portion of the validation set,' and Section 3.3 says participants received the validation set with immediate leaderboard feedback. Section 5.1.1 then reports that IH-VQA used a 'validation set augmentation strategy' in which the model generates pseudo-labels on the validation set and merges them into the training set. Under the natural reading that the released validation set is the same split folded into the test set, IH-VQA can have trained on images that later appear in the test set. The reported Main Score of 0.8551 and the 0.1295 margin over FGA-BLIP2 therefore do not establish that IH-VQA generalizes best; the margin may be inflated by memorization of validation-derived test images. Because IH-VQA is the headline winner, this is not a cosmetic concern, and the same overlap threatens the exact ordering of the other teams if they also tuned to validation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This report describes the NTIRE 2025 challenge on text-to-image generation model quality assessment, with two tracks: an image-text alignment track using EvalMuse-40K and a structural distortion track using the new EvalMuse-Structure dataset. The evaluation protocol uses SRCC, PLCC (after third-order polynomial fitting), ACC for element-level alignment, and F1 for structural masks, combined into a weighted Main Score for each track. The paper reports that all participating teams outperformed the baselines (FGA-BLIP2 and RAHF), with IH-VQA winning the alignment track and HNU-VPAI winning the structure track. Section 5 provides brief descriptions of the 20 submitted methods, including fine-tuned MLLMs, ensembles, pseudo-labeling, and task-specific segmentation/regression architectures.","tokens_in":22638,"tokens_out":3853,"duration_ms":32388,"significance":"If the results are valid, the challenge provides a useful comparison for fine-grained T2I quality assessment, introduces EvalMuse-Structure as a new benchmark, and demonstrates that fine-tuned MLLM ensembles can surpass existing published baselines on alignment scoring and distortion localization. The paper includes standard metrics, full result tables, and team fact sheets, which are valuable for the community. However, the significance is conditional on the test set being a clean held-out measure; as discussed below, the alignment track test construction and the winning team's validation-set training strategy create a direct overlap concern that must be resolved before the headline ranking can be accepted.","major_comments":[{"comment":"The alignment track test set is not fully held out. Section 3.1 states that the test set is composed of 'additional image-text pairs... combined with a portion of the validation set,' while Section 3.3 says participants received the validation set (without annotations) and obtained 'immediate feedback' from the validation leaderboard. Section 5.1.1 then reports that the winning team IH-VQA used a 'validation set augmentation strategy' in which the model generates pseudo-labels on the validation set and merges them into the training set. Under the natural reading that the released validation set is the same split folded into the test set, IH-VQA trained on images that later appear in the test set. This means the reported margin of 0.1295 over the FGA-BLIP2 baseline (Section 4) does not establish generalization; it may reflect memorization of validation-derived test images. The authors must clarify whether the validation set used for pseudo-labeling is identical to the validation portion included in the test set, and must report rankings computed on the additional (cleanly held-out) test images only.","section":"§3.1, §3.3, §5.1.1"},{"comment":"Even for teams that did not explicitly pseudo-label the validation set, the challenge design allowed all participants to receive immediate feedback on the validation set, which is partly included in the test set. This permits validation-based tuning (e.g., checkpoint selection, hyperparameter choice, ensemble weighting) on images that later appear in the test set, potentially inflating scores and altering rankings for all alignment-track teams. The authors should quantify the effect by reporting results on the portion of the test set that was not derived from the released validation set, and should state whether any teams were restricted from using validation images in training. Without this, the claim in Section 4 that 'all submitted results... achieve better performance than the baseline' and the relative ordering of teams are not reliable evidence of generalization.","section":"§3.1, §3.3"},{"comment":"The reliability of the ground-truth annotations is not substantiated. The paper reports that each image-text pair is annotated by three annotators with re-annotation only for 'significant disagreements' in alignment scores, and that structural masks are derived from regions overlapping in at least two of three annotators, but no inter-annotator agreement statistics (e.g., Cohen's kappa, Krippendorff's alpha, or mask IoU between annotators) are provided. Since all challenge scores are evaluated against these annotations, the absence of agreement metrics leaves the quality of the benchmark itself unverified. The authors should report agreement statistics for both tracks, including agreement on element-level alignment labels and on structural distortion masks.","section":"§3.1"}],"minor_comments":[{"comment":"The introduction states '16 valid participating teams in the image track and 12 valid participating teams in the video track,' but the abstract and Section 4 report 12 teams in the alignment track and 8 teams in the structure track; 'image track' and 'video track' should be 'alignment track' and 'structure track.'","section":"§1"},{"comment":"The phrase 'Person Linear Correlation Coefficient' is a typo; it should be 'Pearson Linear Correlation Coefficient.'","section":"§3.2"},{"comment":"The team name is written as 'Tenryu Badu' in the heading of Section 5.2.8 and as 'Tenryu Babu' in the team list in Appendix C; the spelling should be consistent.","section":"§5.2.8 and Appendix C"},{"comment":"Several references are duplicated: [1] and [2] are the same Qwen2.5-VL technical report, [57] and [58] are the same Q-Align paper, and [69] and [70] are the same DeQA-Score paper. These should be consolidated into single entries.","section":"References"},{"comment":"The statement that 'they average the element-wise scores in original dataset to produce a single score per field' is unclear; it is not obvious what 'field' refers to, and the sentence should be rephrased for clarity.","section":"§5.1.3"},{"comment":"Adjacent teams in the structure track have very small Main Score differences (e.g., ranks 2–6 in Table 2 span only 0.0167), and no confidence intervals or significance tests are reported; a sentence acknowledging the uncertainty in close rankings would strengthen the presentation.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the apparent train/test overlap in the alignment track: the test set includes part of the validation set, and the winning team's described method trains on validation-set pseudo-labels. This directly threatens the headline result. If the authors can re-evaluate on the cleanly held-out test portion and show that IH-VQA still wins (or provide a clear statement that the validation set used for pseudo-labeling is disjoint from the test set), the paper could become acceptable after minor revisions. Otherwise, the challenge's main ranking claim is not supported. The lack of inter-annotator agreement statistics is a secondary but important weakness for a benchmark paper. The paper fits the NTIRE challenge report format, but the integrity of the evaluation protocol must be clarified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a standard NTIRE challenge report with one genuinely useful new resource—EvalMuse-Structure, a 12K-image dataset with structural distortion masks—and a systematic comparison of 20 methods. The tables support the qualitative claim that fine-tuned MLLMs beat the provided baselines. But there's a load-bearing problem: the test set is partly made of the validation set that participants saw with immediate leaderboard feedback (Sections 3.1 and 3.3), and the winning alignment team, IH-VQA, explicitly trained on pseudo-labels generated for that validation set (Section 5.1.1). So IH-VQA's 0.1295-point margin over the FGA-BLIP2 baseline may be inflated by direct train/test overlap. This doesn't sink the whole paper—the baseline comparisons are still informative—but it means the exact ranking, and especially the claimed win, is not established.\n\nWhat's good: the EvalMuse-Structure dataset is new; the annotation protocol (three annotators, masks agreed by at least two) is reasonable; the metrics are standard; and the method descriptions in Section 5 give a useful snapshot of what works in practice. The baseline numbers are clearly reported.\n\nSoft spots, in rough order of severity. First, the validation/test overlap above. Second, no significance testing or confidence intervals, and several between-team gaps are tiny (0.0007 in Table 2), so fine ordering is fragile. Third, no inter-annotator agreement stats are reported, which matters because the masks are ground truth for the F1 metric. Fourth, the manuscript is not fully reconciled: abstract says 10K images for EvalMuse-Structure, body says 12K; the introduction mentions a nonexistent 'video track' with wrong team counts; and Appendix C copy-pastes the wrong team title. These are cosmetic but signal sloppy production.\n\nThe central claim that all 20 teams beat the baselines holds up numerically. The exact ordering does not necessarily.\n\nWho this is for: someone working on AI-generated image quality assessment, especially fine-grained element alignment and distortion localization. The dataset alone is worth a look.\n\nFor peer review: yes, send it out rather than desk-reject. The resource and comparative data are valuable, and the leakage issue is fixable (e.g., re-evaluate on a clean test split or report results separately for the clean vs. validation-derived portions). But the current version needs major revision before I'd trust the rankings.","headline":"A useful new dataset and benchmark, but the alignment-track winner trained on validation images that are partly in the test set, so the headline ranking is not established.","tokens_in":23847,"tokens_out":4135,"would_cite":true,"duration_ms":23792,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"All submitted challenge systems beat the published baselines on fine-grained text-to-image quality assessment.","keywords":["text-to-image generation","image quality assessment","image-text alignment","structural distortion detection","challenge report","multimodal large language models","EvalMuse-40K","EvalMuse-Structure"],"falsifier":"Re-run the test phase with the validation-contaminated portion removed and a freshly annotated held-out set, before teams see any feedback, then compare rankings; if the margins of the top teams shrink or reverse, the reported superiority is partly an artifact of leaderboard tuning.","tokens_in":1589,"feed_emoji":"🏆","tokens_out":5846,"duration_ms":84039,"temperature":0.7,"pith_summary":"This paper reports a structured competition on fine-grained quality assessment of text-to-image (T2I) generation models. Its central claim is that current assessment methods can be pushed well past published baselines: every finalist beat the FGA-BLIP2 baseline on element-level image-text alignment or the RAHF baseline on structural distortion localization, and the top methods improved the main score by 0.1295 and 0.0862 respectively. The winning approaches fine-tune large multimodal models and combine them with specialized detection or segmentation components. The paper also contributes EvalMuse-Structure, a 10K-image dataset with bounding-box structural distortion annotations.","feed_headline":"All 20 teams beat baselines in text-to-image quality challenge","feed_subtitle":"Top alignment team adds 0.13 main-score points; structure winner beats baseline by 0.086.","key_machinery":"The central machinery is the paired evaluation protocol, which uses Spearman Rank-order Correlation Coefficient (SRCC), Pearson Linear Correlation Coefficient (PLCC), element-presence accuracy (ACC), and the F1 score between predicted and human distortion masks. For alignment, Main Score = $0.5 \\times \\text{ACC} + 0.25 \\times (\\text{SRCC} + \\text{PLCC})$; for structure, Main Score = $0.7 \\times \\text{F1} + 0.15 \\times (\\text{SRCC} + \\text{PLCC})$. The winning methods carry the argument through fine-tuned multimodal large language models that output both global alignment scores and per-element presence predictions, and through hybrid CNN-Transformer plus instance-segmentation pipelines for structural distortion detection.","core_discovery":"The challenge establishes empirically that fine-grained T2I quality assessment is tractable and that published baselines are not ceilings. In the alignment track, the top team, an ensemble of fine-tuned multimodal language models with instruction-augmented element matching, reaches a main score of 0.8551 versus the FGA-BLIP2 baseline of 0.7256; six alignment teams exceed 0.8. In the structure track, the winning team, combining a CNN-Transformer hybrid quality scorer with instance segmentation for distortion localization, reaches 0.6927 versus the RAHF baseline of 0.6065. Across both tracks, all 20 submitted systems outperform their track's baseline.","pith_inferences":["Because the test set includes validation images whose leaderboard scores were shown to participants during development, part of the reported gap over baselines may reflect leaderboard tuning rather than pure generalization; a fully withheld test set would separate these effects.","The element-presence accuracy metric rewards marking an element present if any ensemble member votes yes, which can inflate recall; reporting precision-recall tradeoffs per element type would sharpen the comparison.","The F1 mask metric derived from majority-overlap annotation may under-penalize small or boundary-detail distortions; instance-level average precision or boundary-aware metrics could reorder the structure track rankings.","The reported improvements are aggregate over prompts and generative models; it remains unknown whether gains are uniform or concentrated in specific element categories such as quantity, spatial relations, or color."],"forward_implications":["Fine-grained element-level alignment scoring improves substantially when multimodal models are fine-tuned with element-specific prompts, chain-of-thought queries, pseudo-label augmentation, and ensembling.","Structural distortion detection is more effective when framed as instance segmentation or heatmap prediction with separate quality-score regression than as a single semantic segmentation task.","Scoring and localization can be decoupled: the top structure team uses separate models for the quality score and the distortion mask, suggesting the two subtasks have different failure modes.","The new EvalMuse-Structure dataset provides a reusable benchmark that combines a structure score and a localization mask under one evaluation protocol.","Because every final team beat the published baselines, the baseline methods leave clear headroom in both fine-grained alignment and structural distortion detection."],"supporting_citations":[{"why":"Supplies the EvalMuse-40K dataset with element-level alignment annotations and the FGA-BLIP2 baseline used in the alignment track.","marker":"[17]"},{"why":"Supplies the RAHF baseline and the fine-grained structure-distortion annotation approach that motivates the structure track.","marker":"[35]"},{"why":"Q-Align's weighted-probability scoring over discrete text-defined levels is adopted by several alignment-track teams.","marker":"[58]"},{"why":"Qwen2.5-VL serves as the backbone for the winning alignment team and several other top methods.","marker":"[2]"},{"why":"Ovis2 models are used in the winning alignment ensemble for both global and element matching.","marker":"[40]"},{"why":"Co-DETR instance segmentation is used by the winning structure team for distortion region detection.","marker":"[78]"},{"why":"LoRA low-rank adaptation is the parameter-efficient fine-tuning technique used by multiple alignment teams.","marker":"[20]"},{"why":"LoDa's local distortion feature enhancement inspires the CNN-Transformer hybrid used by the winning structure team.","marker":"[64]"},{"why":"Study group learning is used to reduce annotation noise in the structure track by one participating team.","marker":"[77]"}],"fun_headline_variants":["All 20 finalists top baselines in T2I quality challenge","Text-to-image quality: every submitted model beats baseline","NTIRE 2025: 20 teams beat T2I model quality baselines","Quality challenge: winners beat baselines by 0.13 and 0.086","Every T2I model in challenge beats its baseline"],"cache_read_input_tokens":25344,"weakest_assumption_plain":"The test set is a fair measure of generalization even though part of it is drawn from the validation set whose leaderboard scores were shown to participants during development, so teams could have tuned to those images.","fun_headline_variants_meta":{"raw":{"variants":["All 20 finalists top baselines in T2I quality challenge","Text-to-image quality: every submitted model beats baseline","NTIRE 2025: 20 teams beat T2I model quality baselines","Quality challenge: winners beat baselines by 0.13 and 0.086","Every T2I model in challenge beats its baseline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000737,"raw_usage":{"total_tokens":3301,"prompt_tokens":962,"completion_tokens":2339,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":2245}},"tokens_in":578,"tokens_out":2339,"duration_ms":14722,"temperature":1.0,"reasoning_tokens":2245,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:04:48.599923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the test phase with the validation-contaminated portion removed and a freshly annotated held-out set, before teams see any feedback, then compare rankings; if the margins of the top teams shrink or reverse, the reported superiority is partly an artifact of leaderboard tuning.","supporting_citations":[{"cited_title":"Evalmuse-40k: A reliable and fine-grained benchmark with comprehensive human annotations for text- to-image generation model evaluation, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the EvalMuse-40K dataset with element-level alignment annotations and the FGA-BLIP2 baseline used in the alignment track."},{"cited_title":"Rich human feedback for text-to-image generation","cited_arxiv_id":null,"evidence_quote":"Supplies the RAHF baseline and the fine-grained structure-distortion annotation approach that motivates the structure track."},{"cited_title":"Detrs with col- laborative hybrid assignments training","cited_arxiv_id":null,"evidence_quote":"Co-DETR instance segmentation is used by the winning structure team for distortion region detection."},{"cited_title":"LoRA: Low-rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"LoRA low-rank adaptation is the parameter-efficient fine-tuning technique used by multiple alignment teams."},{"cited_title":"Boosting image quality assessment through efficient transformer adaptation with lo- cal feature enhancement","cited_arxiv_id":null,"evidence_quote":"LoDa's local distortion feature enhancement inspires the CNN-Transformer hybrid used by the winning structure team."},{"cited_title":"Study group learning: Improving retinal vessel segmentation trained with noisy labels","cited_arxiv_id":null,"evidence_quote":"Study group learning is used to reduce annotation noise in the structure track by one participating team."}],"review_version":1}