REVIEW 4 major objections 6 minor 2 cited by
ICME 2025 Grand Challenge on Video Super-Resolution for Video Conferencing
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper argues that for general-purpose and talking-head video super-resolution, PSNR and SSIM correlate weakly with subjective quality, so rankings should rely on subjective evaluation, while screen-content super-resolution is won by…
desk verdict A useful challenge report with a genuine new dataset, but the central metric-correlation claim is statistically thinner than it looks. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the crowdsourced implementation of ITU-T Rec. P.910 Comparison Category Rating (CCR), which produces CMOS scores by having raters compare each processed clip directly against the ground-truth source. For screen content, this is augmented by a composite challenge score that averages normalized CMOS with character error rate (CER), computed from OCR over approximately 210,000 annotated characters. The paper then uses model-level Pearson, Spearman, Kendall's Tau-b, and a confidence-interval-adjusted Tau-b95 to compare these subjective scores against PSNR, SSIM, VMAF, LPIPS, and CER.
What would settle it
Re-run the subjective evaluation with a larger panel of trained raters on the same test clips and compute the Spearman correlation between CMOS and PSNR or SSIM per track; if in Track 1 or Track 2 that correlation rises above about 0.8, the paper's claim that objective metrics rank poorly would be contradicted. Alternatively, if a held-out set of 50 or more models shows PSNR ranking matching subjective ranking, the result would not generalize.
Extended reading notes
Core claim
The authors claim that objective metrics and subjective quality scores are only weakly correlated for general-purpose and talking-head video super-resolution, making subjective evaluation necessary for ranking models in these contexts. For Track 1, the Pearson correlation between CMOS and PSNR is -0.212 and with SSIM is -0.073, while LPIPS reaches -0.893 Pearson but only -0.800 Spearman; Track 2 shows a similar pattern, with PSNR Pearson at 0.232, SSIM at 0.316, and LPIPS at -0.732 Pearson and -0.720 Spearman. In contrast, for screen content, PSNR and SSIM show strong correlations with CMOS (Pearson 0.885 and 0.894, respectively), and CER correlates strongly as well. The paper also reports that in the general-purpose and talking-head tracks, pretrained image-based baseline models ranked first in subjective tests, while in the screen-content track, participating teams that used OCR and CER-based methods significantly outperformed all baselines.
Load-bearing premise
The conclusions assume the crowdsourced CCR subjective ratings are accurate enough on these 300-frame clips that model-level correlations with about ten models per track are meaningful, even though the paper reports no rater counts, inter-rater agreement, or per-clip confidence.
Editorial extensions
If this is right
- Benchmarks that rank video super-resolution models by PSNR or SSIM on general or talking-head content will not reflect human preference; organizers should budget for subjective testing.
- LPIPS is the strongest objective proxy in these tracks, but its Spearman correlation of about 0.8 still leaves enough disagreement that final rankings should be subjective.
- For screen content, PSNR and SSIM are usable development metrics, and CER should be reported alongside them whenever text legibility matters.
- Screen-content super-resolution systems should include OCR or text-recovery components; the top two teams used them and clearly separated from generic baselines.
- The newly released screen-content and extended talking-head datasets allow future training and evaluation outside the challenge.
Reading between the lines
- Since every model's CMOS is negative in all three tracks, none of the submissions improved on the ground-truth source: the practical task was damage minimization, and the challenge measured which model degraded least, a framing the paper does not itself emphasize.
- The low correlation in Tracks 1 and 2 may be partly an artifact of the narrow model sample, about ten pipelines per track, many of them off-the-shelf baselines, so the observed coefficients are estimates over a small, nonrandom set.
- A natural testable extension is to add a CER-style loss or OCR post-processing to general-purpose and talking-head models, since the screen-content result suggests that training to preserve semantically identifiable structures may also improve perceptual quality.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports on the ICME 2025 Grand Challenge on Video Super-Resolution for Video Conferencing, covering three tracks (general-purpose, talking head, screen content). It describes the datasets, the causal low-delay setup, the crowdsourced CCR subjective evaluation, and the results, including baseline and participant models. The main findings are that objective metrics (PSNR, SSIM, VMAF, LPIPS) correlate only weakly with subjective CMOS for general-purpose and talking-head content, while for screen content the top-performing teams used OCR- and CER-aware approaches. The paper also open-sources a new screen-content dataset and an extension of the VCD talking-head dataset.
Significance. If the correlation finding is statistically robust, the paper would provide practical evidence that common objective metrics mislead model ranking for video-conferencing SR, and that screen-content SR is best evaluated with text-recovery measures. The release of the datasets is a valuable contribution. The paper also includes a sensible test-retest control for the CER measure (0.0024) and transparently discloses that one baseline violates the causal constraint. However, the correlation claim is currently supported only by small-sample point estimates without confidence intervals, and the subjective scoring procedure lacks rater-level statistics, so the significance of the headline finding is not yet established.
major comments (4)
- [III-A, Tables IV and V] The central claim that objective metrics correlate weakly with subjective quality in Tracks 1 and 2 rests on model-level correlation coefficients computed from only 9 models (Track 1) and 10 models (Tracks 2 and 3). No confidence intervals, significance tests, or leave-one-out sensitivity analyses are provided. Given the narrow CMOS range (e.g., -2.216 to -2.712 in Track 1), the point estimates are fragile; for instance, the Track 1 VMAF Pearson coefficient of 0.581 could plausibly include zero, and the LPIPS Pearson coefficient of -0.893 may be driven by the single lowest-CMOS model. The authors should supply bootstrap (or jackknife) confidence intervals for all coefficients in Tables IV and V, or at minimum temper the conclusion to acknowledge that the low correlations are not established beyond this small, narrow-range model set.
- [II-B] The reliability of the subjective scores is not documented. The paper reports per-model CMOS confidence intervals in Table II but no rater counts, no inter-rater agreement (e.g., ICC or Krippendorff's alpha), and no per-clip confidence values. This matters because the tie-ranking procedure in Section III depends on 'no significant difference between the distributions of CMOS values,' and because the correlation analysis in Section III-A relies on the precision of the CMOS values. Add the number of raters per clip, the rating design, and an inter-rater reliability statistic.
- [III] The rule for assigning tied ranks in Table II is not operationalized. The text states that two consecutive models are tied when 'there is no significant difference between the distributions of CMOS values,' but no statistical test is named, and no threshold is given. If the method is the CI-overlap approach described in Ref. [28], state that explicitly and describe how the 95% CIs are used. Without this, the ranks, on which several qualitative conclusions depend, are not reproducible.
- [II-B, Eq. (1)] The Track 3 challenge score in Eq. (1) is ambiguous as typeset. The text calls it a 'normalized average of CMOS and CER,' but the equation can be read either as (CMOS/-3) + (CER/2) or as CMOS/(-3 + CER/2). The ranking in Table II is consistent with the latter interpretation, but the equation should be typeset as a single unambiguous fraction (e.g., score = CMOS / (-3 + CER/2)) so that the scoring rule is reproducible from the text alone.
minor comments (6)
- [II-A, Table I] The Track 3 training-set counts (1520 GT, 9072 LR+H.265) do not match the expected six QP levels per clip (1520x6=9120). Please clarify whether this is due to encoding failures or to a different number of QPs for some clips.
- [II-A] The specific H.265 QP values used are not listed. Providing the QP numbers would improve reproducibility.
- [II-B] The CER measurement pipeline is under-specified; the OCR engine and the frame/text-region selection criteria should be stated, as CER is a key outcome for Track 3.
- [III-A, Table IV] The phrase 'Spearman correlation coefficient of ≤0.8' is confusing; in Track 1 the value is -0.800, so it should be phrased as 'an absolute value no greater than 0.8.'
- [III] The statement that participating teams 'significantly outperformed' baselines in Track 3 is not backed by a significance test; the non-overlapping CIs in Table II are suggestive, but the paper should either perform a formal test or refer to the CI-based tie-breaking rule.
- [Table VI] The FLOPs value for BVIVSR appears twice as 455.16; please verify whether the FLOPs are identical for Track 1 and Track 2, and fix the formatting of the table.
Circularity Check
No significant circularity: the central correlation and ranking claims are empirical, and the organizers' self-citations are instrumental rather than load-bearing.
full rationale
The paper's central claims are empirical observations from a challenge: CMOS subjective scores are correlated with objective metrics, and model rankings are reported. The CMOS scores come from a crowdsourced CCR implementation of ITU-T P.910 [23], which is a measurement procedure that produces independent human judgments, not a mathematical consequence of the paper's definitions. The objective metrics (PSNR, SSIM, VMAF, LPIPS, CER) are standard computed quantities. Equation 1 defines the Track 3 challenge score as a weighted combination of CMOS and CER, but this is an evaluation rule chosen by the organizers, not a derived prediction; the subsequent correlation analysis treats CMOS and CER as separate measured variables. The self-citations to the VCD dataset [22] and the crowdsourcing method [23] are relevant because the organizers built those resources, but neither supplies a conclusion by construction: the subjective ratings are external human measurements, and the datasets are test material. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no result is derived from an assumption that already contains the conclusion. Concerns about small sample size, narrow quality range, or the statistical strength of the correlations are legitimate robustness or correctness issues, but they are not circularity. Therefore the derivation chain is self-contained, and the low circularity score reflects only the presence of non-load-bearing self-citations.
Assumptions & free parameters
free parameters (1)
- Track 3 challenge score coefficients =
CER coefficient 0.5; CMOS normalized by -3
assumptions (4)
- domain assumption Crowdsourced CCR (ITU-T P.910) ratings yield valid, unbiased quality labels despite 300-frame clips and worker variability.
- domain assumption OCR-based CER measured on annotated frames is a valid proxy for text readability in screen content.
- domain assumption Encoding LR inputs with H.265 at six fixed QPs simulates realistic conferencing compression.
- domain assumption Roughly ten models per track is sufficient for rank-correlation inference between subjective and objective metrics.
Cite this review
Pith. "Pith review of ICME 2025 Grand Challenge on Video Super-Resolution for Video Conferencing." pith.science (2026). https://pith.science/paper/D5K7W2XG
@misc{pith2026250612269,
author = {Pith},
title = {Pith review of: ICME 2025 Grand Challenge on Video Super-Resolution for Video Conferencing},
year = {2026},
howpublished = {\url{https://pith.science/paper/D5K7W2XG}},
note = {Machine review of arXiv:2506.12269}
}
read the original abstract
Super-Resolution (SR) is a critical task in computer vision, focusing on reconstructing high-resolution (HR) images from low-resolution (LR) inputs. The field has seen significant progress through various challenges, particularly in single-image SR. Video Super-Resolution (VSR) extends this to the temporal domain, aiming to enhance video quality using methods like local, uni-, bi-directional propagation, or traditional upscaling followed by restoration. This challenge addresses VSR for conferencing, where LR videos are encoded with H.265 at fixed QPs. The goal is to upscale videos by a specific factor, providing HR outputs with enhanced perceptual quality under a low-delay scenario using causal models. The challenge included three tracks: general-purpose videos, talking head videos, and screen content videos, with separate datasets provided by the organizers for training, validation, and testing. We open-sourced a new screen content dataset for the SR task in this challenge. Submissions were evaluated through subjective tests using a crowdsourced implementation of the ITU-T Rec P.910.
Figures
Forward citations
Cited by 2 Pith papers
-
4KAgent: Agentic Any Image to 4K Super-Resolution
An agentic pipeline that plans and executes image restoration from a toolbox of pretrained models to upscale arbitrary images to 4K, reporting state-of-the-art results on many benchmarks.
-
Towards Adaptive Super-Resolution and Quality Assessment via Test-Time Adaptation
A doctoral summary reports consistent perceptual-quality gains from test-time adaptation for video super-resolution, screen-content super-resolution, and no-reference video quality assessment, using the author's previ...
Reference graph
Works this paper leans on
-
[28]
Transformation of mean opinion scores to avoid misleading of ranked based statistical techniques,
Babak Naderi and Sebastian M ¨oller, “Transformation of mean opinion scores to avoid misleading of ranked based statistical techniques,” in2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2020, pp. 1–4
work page 2020
-
[1]
Ntire 2021 challenge on video super-resolution,
Sanghyun Son, Suyoung Lee, Seungjun Nah, Radu Timofte, and Ky- oung Mu Lee, “Ntire 2021 challenge on video super-resolution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 166–181
work page 2021
-
[2]
NTIRE 2024 Challenge on Image Super-Resolution (x4): Methods and Results
Zheng Chen, Zongwei Wu, Eduard Zamfir, Kai Zhang, Yulun Zhang, Radu Timofte, Xiaokang Yang, Hongyuan Yu, Cheng Wan, Yuxin Hong, et al., “NTIRE 2024 Challenge on Image Super-Resolution (x4): Methods and Results,”arXiv preprint arXiv:2404.09790, 2024
work page Pith review arXiv 2024
-
[3]
NTIRE 2023 Challenge on Image Super-Resolution (x4): Methods and results,
Yulun Zhang, Kai Zhang, Zheng Chen, Yawei Li, Radu Timofte, Junpei Zhang, Kexin Zhang, Rui Peng, Yanbiao Ma, Licheng Jia, et al., “NTIRE 2023 Challenge on Image Super-Resolution (x4): Methods and results,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1865–1884
work page 2023
-
[4]
Bsraw: Improving blind raw image super-resolution,
Marcos V Conde, Florin Vasluianu, and Radu Timofte, “Bsraw: Improving blind raw image super-resolution,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 8500–8510
work page 2024
-
[5]
Ntire 2020 challenge on real-world image super-resolution: Methods and results,
Andreas Lugmayr, Martin Danelljan, and Radu Timofte, “Ntire 2020 challenge on real-world image super-resolution: Methods and results,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 494–495
work page 2020
-
[6]
The ninth ntire 2024 efficient super-resolution challenge report,
Bin Ren, Yawei Li, Nancy Mehta, Radu Timofte, Hongyuan Yu, Cheng Wan, Yuxin Hong, Bingnan Han, Zhuoyuan Wu, Yajun Zou, et al., “The ninth ntire 2024 efficient super-resolution challenge report,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 6595–6631
work page 2024
-
[7]
Efficient deep models for real-time 4k image super-resolution. ntire 2023 benchmark and report,
Marcos V Conde, Eduard Zamfir, Radu Timofte, Daniel Motilla, Cen Liu, Zexin Zhang, Yunbo Peng, Yue Lin, Jiaming Guo, Xueyi Zou, et al., “Efficient deep models for real-time 4k image super-resolution. ntire 2023 benchmark and report,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 1495– 1521
work page 2023
Show all 29 references
-
[8]
Aim 2020 challenge on efficient super-resolution: Methods and results,
Kai Zhang, Martin Danelljan, Yawei Li, Radu Timofte, Jie Liu, Jie Tang, Gangshan Wu, Yu Zhu, Xiangyu He, Wenjie Xu, et al., “Aim 2020 challenge on efficient super-resolution: Methods and results,” in Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceed...
2020
-
[9]
Basicvsr: The search for essential components in video super- resolution and beyond,
Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy, “Basicvsr: The search for essential components in video super- resolution and beyond,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 4947–4956
2021
-
[10]
Ntire 2021 challenge on quality enhancement of compressed video: Methods and results,
Ren Yang, “Ntire 2021 challenge on quality enhancement of compressed video: Methods and results,” inProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2021, pp. 647–666
2021
-
[11]
Rebotnet: Fast real-time video enhancement,
Jeya Maria Jose Valanarasu, Rahul Garg, Andeep Toor, Xin Tong, Weijuan Xi, Andreas Lugmayr, Vishal M Patel, and Anne Menini, “Rebotnet: Fast real-time video enhancement,”arXiv preprint arXiv:2303.13504, 2023
2023 arXiv
-
[12]
Ntire 2019 challenge on video super-resolution: Methods and results,
Seungjun Nah, Radu Timofte, Shuhang Gu, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, and Kyoung Mu Lee, “Ntire 2019 challenge on video super-resolution: Methods and results,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Work...
2019
-
[13]
Aim 2024 challenge on efficient video super-resolution for av1 compressed content,
Marcos V Conde, Zhijun Lei, Wen Li, Christos Bampis, Ioannis Katsavounidis, and Radu Timofte, “Aim 2024 challenge on efficient video super-resolution for av1 compressed content,”arXiv preprint arXiv:2409.17256, 2024
2024 arXiv
-
[14]
Gonzalez and R
R. Gonzalez and R. Woods,Digital image processing, Prentice Hall, 3rd edition, 2006
2006
-
[15]
Image quality assessment: from error visibility to structural similarity,
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,”IEEE Transactions on Image Processing, vol. 13, pp. 600–612, Apr. 2004
2004
-
[16]
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, June 2018, pp. 586–595, IEEE
2018
-
[17]
Toward A Practical Perceptual Video Quality Metric.,
Z. Li, A. Aaron, I. Katsavounidis, A. Moorthy, and M. Manohara, “Toward A Practical Perceptual Video Quality Metric.,” Tech. Rep., 2016
2016
-
[18]
A Subjective Study to Evaluate Video Quality Assessment Algorithms,
Kalpana Seshadrinathan, Rajiv Soundararajan, Alan Bovik, and Lawrence Cormack, “A Subjective Study to Evaluate Video Quality Assessment Algorithms,” inHuman Vision and Electronic Imaging, 2010
2010
-
[19]
Investigating tradeoffs in real-world video super-resolution,
Kelvin CK Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy, “Investigating tradeoffs in real-world video super-resolution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5962–5971
2022
-
[20]
Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study,
Seungjun Nah, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee, “Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study,” inCVPR Workshops, June 2019
2019
-
[21]
Openvid-1m: A large- scale high-quality dataset for text-to-video generation,
Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai, “Openvid-1m: A large- scale high-quality dataset for text-to-video generation,”arXiv preprint arXiv:2407.02371, 2024
2024 arXiv
-
[22]
Vcd: A video conferencing dataset for video compression,
Babak Naderi, Ross Cutler, Nabakumar Singh Khongbantabam, Yasaman Hosseinkashi, Henrik Turbell, Albert Sadovnikov, and Quan Zou, “Vcd: A video conferencing dataset for video compression,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Process...
2024
-
[23]
A crowdsourcing approach to video quality assessment,
Babak Naderi and Ross Cutler, “A crowdsourcing approach to video quality assessment,” inICASSP, 2024
2024
-
[24]
Swinir: Image restoration using swin transformer,
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte, “Swinir: Image restoration using swin transformer,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 1833–1844
2021
-
[25]
Real-world super-resolution via kernel estimation and noise injection,
Xiaozhong Ji, Yun Cao, Ying Tai, Chengjie Wang, Jilin Li, and Feiyue Huang, “Real-world super-resolution via kernel estimation and noise injection,” inproceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 2020, pp. 466–467
2020
-
[26]
Realviformer: Investigating attention for real-world video super-resolution,
Yuehan Zhang and Angela Yao, “Realviformer: Investigating attention for real-world video super-resolution,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 412–428
2024
-
[27]
Basicvsr++: Improving video super-resolution with enhanced propagation and alignment,
Kelvin CK Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy, “Basicvsr++: Improving video super-resolution with enhanced propagation and alignment,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 5972–5981
2022
-
[29]
Arbitrary- steps image super-resolution via diffusion inversion,
Zongsheng Yue, Kang Liao, and Chen Change Loy, “Arbitrary- steps image super-resolution via diffusion inversion,”arXiv preprint arXiv:2412.09013, 2024
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.