{"id":"f6719ba4-3b02-4e45-8be7-ff94b3379ac4","arxiv_id":"2509.06413","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new SR image quality dataset focused on modern GAN and diffusion super-resolution outputs, plus benchmark results from four teams achieving SRCC above 0.90, is presented.","lead":"This paper reports a challenge on assessing the quality of AI-generated super-resolution images, using a new dataset of 720 images produced by GAN and diffusion models. It provides a public benchmark and shows that four submitted methods predict human quality ratings with correlations above 0.9.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ISRGen-QA split granularity is unspecified: if train/test share HR/LR sources, the >0.9 challenge scores likely reflect content memorization, not general SR-IQA ability.","rationale":"The reader's weakest assumption is the split granularity; I agree that it is the most load-bearing. The dataset's 19 HR sources make this non-negotiable: with a 10% test split, a random SR-level split will place variants of nearly every HR/LR in both train and test. The paper's Section 2.1 gives no indication that the split is content-disjoint, and Section 2.3.2 confirms that test-phase participants received HR/LR references for test images, so reference-based methods could match them against training references. This is not a question of fraud—organizers may have made a disjoint split—but the manuscript as written does not establish it. A second concern is the lack of baselines for the 'SOTA' claim; that weakens the interpretation but does not invalidate the internal ranking. The polynomial-fit issue in Figure 2 is worth noting but is not the primary vulnerability: SRCC is rank-based and unaffected by monotone fitting, and the text only mentions the fourth-order fit for scatter visualization. Therefore the recommended verdict stays conditional: the dataset contribution is real, but the headline SOTA claim should be accepted only after the split is confirmed content-disjoint and, ideally, baselines are added.","tokens_in":14036,"tokens_out":7235,"duration_ms":87356,"concrete_test":"Download the official ISRGen-QA split lists and map each train/val/test SR image to its HR/LR source IDs. If any HR or LR source appears in both train and test, retrain/re-evaluate the submitted methods on the subset of test images whose content was unseen in training; if that subset is empty or SRCC/PLCC drop materially (e.g., >0.05), the headline numbers are leakage-inflated. If the split is already content-disjoint, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's SOTA claim (all teams >0.9, Sec. 3.1) rests on the test set being a valid measure of generalization. Sec. 2.1 reports only 19 HR references and 76 LR references for 720 SR images, with a 576/72/72 train/val/test split, but never states whether the split is at the SR-image level or at the HR/LR-content level. With 19 content sources, an SR-level split almost guarantees the same HR/LR content appears in train and test (as different SR-algorithm outputs). That gives the reference-based 2077 Agent method (Sec. 4.4) and the LR-residual-based ydy method (Sec. 4.2) direct access to train/test content correspondences, and even SR-only models can memorize content-specific artifact patterns. If so, the reported SRCC/PLCC values are inflated by content leakage rather than measuring a general ability to score unseen SR content. The absence of any baseline NR-IQA comparison in Sec. 3 further prevents calibrating these absolute numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports the organization and results of the VQualA 2025 ISRGC-Q challenge on quality assessment of super-resolved (SR) images produced by modern generative models. It introduces the ISRGen-QA dataset (720 SR images from 15 SR algorithms, derived from 19 HR and 76 LR references, with MOS from 21 valid participants), defines an evaluation protocol based on a 0.6·SRCC + 0.4·PLCC score, and describes the four participating methods. All four teams report overall scores above 0.9 on a 72-image test set, and the abstract claims these results demonstrate state-of-the-art performance. The paper also provides architecture and training details for each method and makes the dataset publicly available.","tokens_in":14290,"tokens_out":4869,"duration_ms":55383,"significance":"If the reported numbers are valid, the challenge demonstrates that learned models can predict human perceptual quality ratings on modern GAN- and diffusion-based SR content with high correlation. The public ISRGen-QA dataset is a potentially valuable resource, as existing SR-IQA datasets predate the current generation of generative SR methods. The paper also provides detailed method descriptions and a reproducible challenge protocol, which are strengths. However, the central SOTA claim is not yet supported because no comparison with existing NR-IQA or SR-IQA baselines is provided, and the 72-image test set without uncertainty quantification makes the absolute scores difficult to interpret.","major_comments":[{"comment":"The dataset split is described only as 'training (576 images, 80%), validation (72 images, 10%), and test (72 images, 10%)' without specifying whether the split is at the SR-image level or at the level of the 19 HR / 76 LR content sources. Because each content source yields multiple SR outputs (different algorithms and scales), an SR-level split is likely to place different versions of the same underlying content in training and test. This would let reference-based methods (2077 Agent, Sec. 4.4, uses HR; ydy, Sec. 4.2, uses LR) exploit content correspondences, and would allow even SR-only models to memorize content-specific artifact patterns. This is load-bearing for the validity of the reported test correlations. Please state the split granularity; if it is SR-level, re-split by content and re-report all results.","section":"Section 2.1"},{"comment":"The abstract and Section 3.1 describe the submitted methods as 'state-of-the-art' with 'exceptional performance' based on overall scores exceeding 0.9. No existing NR-IQA or SR-IQA baselines are evaluated on the same test set. Without comparisons to, e.g., MUSIQ, NIQE, BRISQUE, or SR-specific metrics, the absolute SRCC/PLCC values are uncalibrated and cannot support a SOTA claim. Please add a baseline comparison on the same test split.","section":"Abstract / Section 3.1"},{"comment":"All conclusions in Section 3.1 are based on 72 test images with no confidence intervals or significance tests. The differences among teams (e.g., SRCC 0.9588 vs. 0.9277) are small relative to the sampling variability of Spearman correlation at n=72. The claims that PLCC consistently exceeds SRCC and the ranking of methods are not statistically established. Report bootstrap confidence intervals or significance tests, and temper the 'consistent pattern' conclusions accordingly.","section":"Section 3.1"},{"comment":"The evaluation protocol defines PLCC but does not state whether the Pearson correlation is computed on raw predicted scores or after a nonlinear monotonic mapping (e.g., a logistic fit), which is standard in IQA. This affects the reported PLCC values and comparability with prior SR-IQA work. Please specify the exact computation and whether any nonlinear regression was applied before computing PLCC.","section":"Section 2.2"}],"minor_comments":[{"comment":"There is an inconsistency: Section 2.3.2 says '5 teams submitted their final results' but the abstract and Section 3 say only 4 teams submitted valid solutions and fact sheets. Please clarify the count.","section":"Section 2.3.2"},{"comment":"The 'Testing Details' paragraph contains a duplicated sentence: 'Subsequently, center cropping is performed to extract 448×448 resolution images...' appears twice, with the second occurrence being incomplete. Please fix the editing error.","section":"Section 4.1"},{"comment":"The description 'anomaly filtering yielding valid scores from 21 participants' lacks detail. Specify how anomalies were detected and how many participants were excluded.","section":"Section 2.1"},{"comment":"The word 'reweighting' is misleading; Eq. (1) is a weighted combination of SRCC and PLCC. Also clarify whether PLCC is computed after any nonlinear fitting, as noted in the major comments.","section":"Eq. (1)"},{"comment":"The caption notes that GFlops are 'calculated relative to Input Size' but the precise meaning is unclear. State whether this is the multiply-accumulate count for the listed input resolution and how it was computed.","section":"Table 1"},{"comment":"The fourth-order polynomial fitting is mentioned only for visualization. If this fit is also used in PLCC computation, it must be described in Section 2.2; otherwise, clarify that it is only for plotting.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The key uncertainty is the data split granularity. If the authors cannot confirm a content-disjoint split, the challenge results may be invalid. I recommend requesting the split specification, baseline comparisons, and statistical uncertainty quantification before considering publication. The manuscript is appropriate as a challenge report, but the SOTA claim currently exceeds the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset is the real contribution here. ISRGen-QA is the first SR-IQA set to include diffusion-based and transformer-based super-resolution outputs, and it's public. The 720 images, 15 modern algorithms, and per-image MOS from 21 subjects fill a gap that the older QADS, Waterloo, and SISAR sets leave open. The challenge protocol is clearly written, the train/val/test split is explicit, and the per-team descriptions are concise but point to companion papers. Credit where due: the organizers did the hard part, which is building and annotating the set.\n\nThe soft spots are mostly around what the paper claims from the challenge results. The 'state-of-the-art performance' claim is not backed by any comparison to existing NR-IQA or SR-IQA baselines. On a 72-image test set, absolute SRCC/PLCC numbers without baselines or confidence intervals don't tell us much. A bigger issue is the split granularity. The paper says 720 SR images come from 19 HR references and 76 LR references, with a 576/72/72 split, but never says whether the split is at the content level. With only 19 HR sources, an image-level split almost guarantees the same HR/LR content appears in both training and test, just generated by different SR algorithms. The two teams that use LR or HR references (ydy and 2077 Agent) would then have direct access to content correspondences at test time, and even SR-only models can memorize content-specific artifacts. That would inflate the reported correlations substantially. I'm not saying that's what happened, but the paper doesn't rule it out, and the claim hangs on it.\n\nThe 0.6/0.4 weighting of SRCC/PLCC is a protocol choice; it's arbitrary but not harmful.\n\nBottom line: the dataset deserves to become a standard benchmark, but the challenge results as reported should be treated as illustrative until the split is clarified and baselines are added. This deserves a serious referee, with revision likely. I'd bring it to a reading group for the dataset and the discussion of generative-SR artifacts.","headline":"A useful new public dataset for SR-IQA on modern generative SR, but the SOTA claim and test results are weakened by an unspecified content split and no baselines.","tokens_in":14925,"tokens_out":2427,"would_cite":true,"duration_ms":25290,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This challenge paper reports that four quality-assessment methods predict human perceptual scores on super-resolved images from modern GAN and diffusion models, all exceeding 0.9 on the ISRGen-QA test set.","keywords":["image super-resolution","quality assessment","generative models","GAN artifacts","diffusion SR","mean opinion score","SRCC","PLCC"],"falsifier":"Regroup the 720 SR images by their high-resolution parent image and split so that all SR images from any single HR reference stay in exactly one split; retrain the submitted models on the content-disjoint training fold. If the test correlations drop substantially below the reported 0.9, the original numbers were inflated by reference leakage. A cheaper check is to verify whether any same-HR image pairs straddle the current train/test boundary.","tokens_in":13960,"feed_emoji":"🖼️","tokens_out":3294,"duration_ms":39479,"temperature":0.7,"pith_summary":"The paper presents the first benchmark challenge specifically for assessing the perceptual quality of images super-resolved by recent generative models — GANs, diffusion models, and transformer-based upscalers. It claims that all four submitted teams achieved overall scores above 0.9, meaning their predicted quality ratings closely track human mean opinion scores. This matters because older SR-IQA datasets contain few or no generative SR outputs, so existing metrics may not capture the artifacts of current super-resolution technology. The paper argues that blind, no-reference methods can succeed on this new benchmark, and the winning method needed only the SR image itself, not the high-resolution reference.","feed_headline":"Four methods top 0.9 on generative SR quality benchmark","feed_subtitle":"Blind models predict human opinion scores on GAN and diffusion upscaled images, with the best scoring 0.964.","key_machinery":"The central object is the ISRGen-QA dataset itself, built from 19 high-resolution references and 76 low-resolution images downsampled at four scales, then super-resolved by 15 recent GAN, diffusion, transformer, flow, and CNN methods to yield 720 SR images at roughly 2K resolution with MOS from 21 human raters. The evaluation machinery is the weighted score Score = 0.6×SRCC + 0.4×PLCC, which jointly rewards rank-order and linear agreement with human scores. Each submitted method is a distinct learned regressor mapping SR images (and sometimes the LR reference) to quality scores, trained on the 576-image development split.","core_discovery":"The paper's central claim is that the ISRGen-QA dataset — 720 approximately 2K SR images from 15 recent super-resolution algorithms (ESRGAN, Real-ESRGAN, SR3, SwinIR, and others), rated by 21 valid subjects — supports reliable automatic quality prediction, and that the four challenge submissions achieve state-of-the-art correlation with human opinion. The best method (MICV) reached an overall score of 0.9638, combining SRCC 0.9588 and PLCC 0.9714. All teams' PLCC values exceeded their SRCC values, indicating the models capture linear relationships well but occasionally misorder closely rated images. The paper further describes each method: a hybrid ViT-CNN that uses only the SR image, a BLIP","pith_inferences":["The reported numbers rest on a train/test split that is not shown to be disjoint at the level of the 19 high-resolution reference images; if the same reference contributes both training and test SR images, models could memorize reference-specific artifacts and inflate correlations.","With only 72 test images and one MOS value per image, the 0.9+ correlations may carry wide confidence intervals; a content-disjoint split or repeated resampling would reveal how stable the ranking is.","The consistent PLCC-over-SRCC gap across all four teams hints that the test set may contain distinctive quality clusters rather than a smooth spectrum, a pattern worth checking on other generative SR datasets.","A natural extension is to test whether these models transfer across SR datasets with different generative backbones (e.g., from GAN-heavy to diffusion-heavy outputs), since all four teams trained and tested on the same small dataset."],"forward_implications":["If the central claim holds, blind no-reference quality metrics can reliably grade images produced by modern generative super-resolution, removing the need for high-resolution references in quality monitoring.","The benchmark provides a reproducible test bed for comparing SR-IQA methods on the artifact types actual users encounter today, such as hallucinated textures and unnatural reconstruction patterns.","The patterns in the results — high PLCC with slightly lower SRCC — suggest that future methods should focus on fine-grained rank ordering among similar-quality images, where current models are weakest.","Because the winning method ignores LR and HR references, lightweight SR-only quality models could be deployed in real-world pipelines without extra reference data.","The dataset's coverage of four upscaling factors and 15 algorithms gives a concrete target for improving generalization across SR methods."],"supporting_citations":[{"why":"Supplies the 19 high-resolution reference images (from DIV2K) from which all SR images in ISRGen-QA are derived.","marker":"[1]"},{"why":"The MICV team's method paper, describing the hybrid ViT-CNN that predicts quality from SR-only input and achieved the top overall score.","marker":"[14]"},{"why":"The ydy team's method paper, introducing BLIP-2-assisted residual-guided quality assessment with a dual ResNet50 backbone.","marker":"[29]"},{"why":"The QA-Veteran team's method paper, proposing a resolution-adaptive vision-language model based on SigLIP2-NaFlex.","marker":"[49]"},{"why":"The 2077 Agent team's method paper, presenting UltraR-IQA, a region-weighted MOS histogram prediction approach.","marker":"[27]"},{"why":"Supplies the SigLIP2-NaFlex pretrained vision-language backbone that the QA-Veteran method relies on for resolution-adaptive quality prediction.","marker":"[31]"},{"why":"Provides the textual template idea ('a photo with {q} quality') used by the QA-Veteran method to map visual embeddings to quality scores.","marker":"[48]"}],"fun_headline_variants":["VQualA challenge: 4 methods beat 0.9 on generative SR quality","Top generative SR quality score hits 0.964 in VQualA challenge","ISRGen-QA challenge: four models lead generative SR quality assessment","Blind IQA for GAN/diffusion SR: VQualA challenge results","Four methods set SOTA on generative SR quality (VQualA 2025)"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The train/validation/test split of the 720 SR images is not shown to be disjoint at the level of the 19 high-resolution reference images, so artifacts specific to a particular reference could in principle appear in both training and test and inflate the reported correlations.","fun_headline_variants_meta":{"raw":{"variants":["VQualA challenge: 4 methods beat 0.9 on generative SR quality","Top generative SR quality score hits 0.964 in VQualA challenge","ISRGen-QA challenge: four models lead generative SR quality assessment","Blind IQA for GAN/diffusion SR: VQualA challenge results","Four methods set SOTA on generative SR quality (VQualA 2025)"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000586,"raw_usage":{"total_tokens":2584,"prompt_tokens":733,"completion_tokens":1851,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":1746}},"tokens_in":477,"tokens_out":1851,"duration_ms":12628,"temperature":1.0,"reasoning_tokens":1746,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:36:57.953025+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Regroup the 720 SR images by their high-resolution parent image and split so that all SR images from any single HR reference stay in exactly one split; retrain the submitted models on the content-disjoint training fold. If the test correlations drop substantially below the reported 0.9, the original numbers were inflated by reference leakage. A cheaper check is to verify whether any same-HR image pairs straddle the current train/test boundary.","supporting_citations":[{"cited_title":"NTIRE 2017 Chal- lenge on Single Image Super-Resolution: Dataset and Study","cited_arxiv_id":null,"evidence_quote":"Supplies the 19 high-resolution reference images (from DIV2K) from which all SR images in ISRGen-QA are derived."},{"cited_title":"Hybrid Vision Transformer and Convolutional Neural Network for Super- Resolution Image Quality Assessment","cited_arxiv_id":null,"evidence_quote":"The MICV team's method paper, describing the hybrid ViT-CNN that predicts quality from SR-only input and achieved the top overall score."},{"cited_title":"BLIP-2 Assisted Residual- Guided Quality Assessment for Super-Resolution Images","cited_arxiv_id":null,"evidence_quote":"The ydy team's method paper, introducing BLIP-2-assisted residual-guided quality assessment with a dual ResNet50 backbone."},{"cited_title":"Blind Super-resolution Quality Assessment based on a Resolution-adaptive Vision-language Model","cited_arxiv_id":null,"evidence_quote":"The QA-Veteran team's method paper, proposing a resolution-adaptive vision-language model based on SigLIP2-NaFlex."},{"cited_title":"Ultra-High- Resolution Image Quality Assessment","cited_arxiv_id":null,"evidence_quote":"The 2077 Agent team's method paper, presenting UltraR-IQA, a region-weighted MOS histogram prediction approach."},{"cited_title":"SigLIP 2: Multilingual Vision- Language Encoders with Improved Semantic Understanding, Localization, and Dense Features, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the SigLIP2-NaFlex pretrained vision-language backbone that the QA-Veteran method relies on for resolution-adaptive quality prediction."},{"cited_title":"Blind Image Quality Assessment via Vision- Language Correspondence: A Multitask Learning Perspec- tive","cited_arxiv_id":null,"evidence_quote":"Provides the textual template idea ('a photo with {q} quality') used by the QA-Veteran method to map visual embeddings to quality scores."}],"review_version":1}