REVIEW 3 major objections 5 minor 43 references
RankSRGAN: Generative Adversarial Networks with Ranker for Image Super-Resolution
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A learned Ranker lets super-resolution GANs optimize perceptual scores directly, and the paper reports best NIQE/PI on three benchmarks.
desk verdict A genuine, well-ablated approach to optimizing non-differentiable perceptual metrics in SR; the main gap is how the frozen Ranker behaves on the generator's shifting output distribution. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Ranker is a Siamese CNN that takes two images and outputs scalar ranking scores, trained with margin-ranking loss to match the pair-wise order of a chosen perceptual metric. It converts an indifferentiable metric into a differentiable objective; the concrete hook is the rank-content loss $L_R = \mathrm{sigmoid}(R(G(x_i)))$, which lets gradients flow from the predicted rank back into the generator. The metric-rank labeling scheme, rather than model classification, is what lets the framework exceed the upper bound of any single SR algorithm.
What would settle it
Take a trained RankSRGAN and compare its Ranker scores against the true perceptual metric on the generator's own outputs: if the Spearman correlation drops well below the reported validation value of 0.88, or if a model trained with the rank-content loss improves its Ranker score without improving the actual metric, the Ranker has stopped tracking the target.
Extended reading notes
Core claim
The central claim is that any non-differentiable perceptual metric can be optimized through a learned ranking surrogate. The Ranker is a Siamese VGG-style network trained to reproduce the pair-wise ordering induced by a metric on outputs of several SR algorithms; training uses margin-ranking loss rather than regression, so the surrogate only needs ordinal agreement. The rank-content loss $L_R = \mathrm{sigmoid}(R(G(x_i)))$ is added to the standard SRGAN objective, and lowering it raises the predicted perceptual rank. Because the rank dataset contains images where SRGAN and ESRGAN each win in mixed order, the generator is guided to adopt the better parts of either algorithm. The reported experiments give the best NIQE and PI among the compared methods on Set14, BSD100, and PIRM-Test, preserve PSNR on PIRM-Test, and user studies prefer RankSRGAN over both baselines.
Load-bearing premise
The Ranker is trained only on super-resolved outputs from SRResNet, SRGAN, and ESRGAN under a fixed perceptual metric, and the method assumes its ranking signal stays valid for the generator's changing output distribution during adversarial training.
Editorial extensions
If this is right
- Swapping the labels in the rank dataset swaps the target metric: RankSRGAN trained with NIQE, Ma, or PI labels improves that metric, so the same framework can chase any pair-wise rankable quality score.
- Because the Ranker learns order rather than absolute values, it separates SR algorithms whose metric values are close (for example SRGAN vs ESRGAN on NIQE), giving the generator a usable gradient where regression would collapse.
- The metric-rank upper bound can exceed every SR algorithm in the rank dataset, so adding better or complementary guiding algorithms, even ground-truth HR, should continue to improve perceptual quality.
- The rank-content loss can be weighted against MSE to navigate the perception-distortion tradeoff, letting practitioners choose a preferred balance between visual quality and fidelity.
- The generator's parameter count and inference cost stay the same as SRGAN; only training time grows by about 1.18 times.
Reading between the lines
- An unstated consequence is that any non-differentiable evaluation score expressible as pair-wise preferences, not just NR-IQA metrics, could be optimized with the same Ranker mechanism, including full-reference quality measures or task-specific scores.
- A risk the paper does not address is reward hacking: the generator may exploit the Ranker's learned scoring function rather than the true metric if training pushes outputs outside the rank dataset's distribution, and the reported validation SROCC of 0.88 does not guarantee the same ordinal accuracy on the generator's own outputs.
- A testable extension is to build rank datasets from more diverse or deliberately adversarial SR models, or to update the Ranker periodically during GAN training, to see whether the ranking signal stays aligned with the target metric under distribution shift.
- The metric-rank versus model-classification comparison suggests an ensemble view: a Ranker over many specialist generators could yield a single model that inherits each specialist's best behavior per image.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. RankSRGAN proposes a super-resolution framework that augments SRGAN with a learned Ranker, a Siamese CNN trained with margin-ranking loss to mimic the ordering behavior of non-differentiable perceptual metrics such as NIQE, PI, and Ma. The Ranker is trained on rank datasets constructed from the outputs of SRResNet, SRGAN, and ESRGAN on DIV2K/Flickr2K. Once trained, the Ranker's scalar output is used to define a rank-content loss, and the generator is optimized by a weighted sum of perceptual, adversarial, and rank-content losses. Experiments on Set14, BSD100, and PIRM-Test report lower NIQE/PI than SRGAN and ESRGAN, with ablations on the rank dataset composition, the choice of perceptual metric, ranking versus regression for the Ranker, and the loss combination.
Significance. If the mechanism is sound, the idea of using a learned ranker as a differentiable surrogate for non-differentiable perceptual quality metrics is a useful addition to perceptual SR, and the paper demonstrates that this can combine the strengths of different SR algorithms. The ablations (rank vs. regression, different metrics, different rank datasets) are thoughtful and provide some evidence that the Ranker is not merely a black-box regularizer. The paper also reports SROCC of the Ranker and includes convergence curves, which is helpful. However, the central claim that the rank-content loss 'optimizes the generator in the direction of perceptual metrics' rests on an unvalidated assumption: that the Ranker's absolute score is a valid loss signal for images that fall outside its training distribution. The empirical gains are small and reported without error bars or significance tests, so the strength of the claim currently exceeds the evidence.
major comments (3)
- [Sec. 3.3, Eq. (9)] The rank-content loss in Eq. (9) is the sigmoid of the Ranker's raw output, but the Ranker is trained with a margin-ranking loss (Eq. (4)) that determines scores only up to an arbitrary monotonic transformation. Without calibration or a mechanism anchoring the absolute scale of R to the target perceptual metric, minimizing sigmoid(R(G(x))) is not theoretically guaranteed to minimize the metric. The paper should provide either a calibration step or empirical evidence that lower R(G(x)) consistently corresponds to better metric values on a validation set of images produced during training.
- [Sec. 4.1-4.2] The Ranker is trained only on outputs of SRResNet, SRGAN, and ESRGAN, and its accuracy is evaluated only via SROCC on a held-out split of that same rank dataset. During adversarial training, the generator's output distribution shifts (as seen in Figure 6), yet the paper does not measure how the Ranker's ranking accuracy behaves on the generator's intermediate or final outputs. Without this distribution-shift analysis, the key claim that the rank-content loss 'optimizes the generator in the direction of perceptual metrics' is not fully supported; it is possible that the Ranker acts as a regularizer that only biases outputs toward the training SR methods. Please evaluate the Ranker on generated images from the trained RankSRGAN and report the correlation between R(G(x)) and the metric.
- [Tables 2-7] All quantitative results are reported as single-run means without error bars or significance tests. Given that the improvements over ESRGAN are often within a few hundredths of a NIQE/PI point (e.g., Table 2, Set14: NIQE 3.28 vs 3.28, PI 2.61 vs 2.61), it is unclear whether the claimed gains are statistically meaningful. I request variance across at least three independent training runs or a statistical significance test on the test datasets.
minor comments (5)
- [Sec. 4.1, Eq. (11)] The SROCC definition is written only for the case of no tied ranks; please clarify how ties are handled or state that the data contain no ties.
- [Sec. 3.4, Eq. (10)] The notation 'PMSR2-L + PMSR2-H' and 'PMSR1-L' is confusing; the subscripts should be defined explicitly.
- [Figure 3] The caption does not match the bars; please indicate which bar corresponds to 'Metric Rank', 'Model Classification', 'ESRGAN', and 'SRGAN'.
- [Sec. 4.4, Table 7] The values of alpha_1 and alpha_2 are given as '{1,5}' but it is unclear which alpha corresponds to which row; please label the rows explicitly.
- [Throughout] There are minor typos: 'beheviour' in Section 3.1, 'traning' in Section 4.2, and 'indifferentiable' should likely be 'non-differentiable' in the introduction.
Circularity Check
No significant circularity: the Ranker is a learned surrogate, while final evaluation uses the original metrics on held-out sets.
full rationale
The paper's central claim is that RankSRGAN optimizes a generator in the direction of otherwise non-differentiable perceptual metrics by training a Ranker to imitate metric rankings and then using the Ranker output as a loss. This is a surrogate-optimization procedure, not a circular derivation. The Ranker is trained on outputs of SRResNet, SRGAN, and ESRGAN with labels from NIQE, Ma, or PI, and then the rank-content loss (Eq. 9) minimizes sigmoid of the Ranker score. If the Ranker perfectly mimicked the metric, minimizing it would indeed improve the metric, but the paper does not define the target metric as the Ranker. The final evaluations in Table 2 use the original NIQE and PI values computed directly on Set14, BSD100, and PIRM-Test, which are external to the Ranker training data (DIV2K/Flickr2K). Thus the reported improvements are not forced by construction; they depend on whether the Ranker generalizes. The Ranker's SROCC of 0.88 is reported on a held-out split of the rank dataset, which is evidence of ordering ability but does not by itself guarantee distribution-shift robustness. That is a correctness or generalization concern, not a circularity. The paper also cites ESRGAN, which shares overlapping authors, but ESRGAN is used as an external baseline and data source, not as an unverified premise that carries the argument. No equation in the paper reduces to its own inputs, and no fitted parameter is relabeled as a prediction. The upper-bound analysis in Section 3.4 is descriptive and is separately validated by experiments comparing RankSRGAN-MR and RankSRGAN-MC. Overall, the derivation chain is self-contained with respect to the tested metrics; the main limitations are empirical robustness issues rather than circular reasoning.
Assumptions & free parameters
free parameters (3)
- margin epsilon in margin-ranking loss =
0.5
- adversarial loss weight in Eq. 12 =
0.005
- rank-content loss weight in Eq. 12 =
0.03
assumptions (4)
- domain assumption NIQE, PI and Ma are valid proxies for human perceptual quality.
- domain assumption A Siamese CNN trained with margin-ranking loss can learn the ordering behavior of a non-differentiable perceptual metric.
- domain assumption The rank dataset built from SRResNet, SRGAN and ESRGAN outputs covers the relevant distribution of perceptual quality for training.
- domain assumption The SRGAN base model and adversarial training setup are reliable.
Cite this review
Pith. "Pith review of RankSRGAN: Generative Adversarial Networks with Ranker for Image Super-Resolution." pith.science (2026). https://pith.science/paper/FBWKQTQ7
@misc{pith2026190806382,
author = {Pith},
title = {Pith review of: RankSRGAN: Generative Adversarial Networks with Ranker for Image Super-Resolution},
year = {2026},
howpublished = {\url{https://pith.science/paper/FBWKQTQ7}},
note = {Machine review of arXiv:1908.06382}
}
read the original abstract
Generative Adversarial Networks (GAN) have demonstrated the potential to recover realistic details for single image super-resolution (SISR). To further improve the visual quality of super-resolved results, PIRM2018-SR Challenge employed perceptual metrics to assess the perceptual quality, such as PI, NIQE, and Ma. However, existing methods cannot directly optimize these indifferentiable perceptual metrics, which are shown to be highly correlated with human ratings. To address the problem, we propose Super-Resolution Generative Adversarial Networks with Ranker (RankSRGAN) to optimize generator in the direction of perceptual metrics. Specifically, we first train a Ranker which can learn the behavior of perceptual metrics and then introduce a novel rank-content loss to optimize the perceptual quality. The most appealing part is that the proposed method can combine the strengths of different SR methods to generate better results. Extensive experiments show that RankSRGAN achieves visually pleasing results and reaches state-of-the-art performance in perceptual metrics. Project page: https://wenlongzhang0724.github.io/Projects/RankSRGAN
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[1]
Ntire 2017 challenge on single image super-resolution: Dataset and study
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , volume 3, page 2, 2017
work page 2017
-
[2]
2018 pirm challenge on perceptual image super-resolution
Yochai Blau, Roey Mechrez, Radu Timofte, Tomer Michaeli, and Lihi Zelnik-Manor. 2018 pirm challenge on perceptual image super-resolution. arXiv preprint arXiv:1809.07517 , 2018
arXiv 2018
-
[3]
The perception-distortion tradeoff
Yochai Blau and Tomer Michaeli. The perception-distortion tradeoff. In Proc. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, Utah, USA , pages 6228--6237, 2018
work page 2018
-
[4]
Deep neural networks for no-reference and full-reference image quality assessment
Sebastian Bosse, Dominique Maniry, Klaus-Robert M \"u ller, Thomas Wiegand, and Wojciech Samek. Deep neural networks for no-reference and full-reference image quality assessment. IEEE Transactions on Image Processing , 27(1):206--219, 2018
work page 2018
-
[5]
a ckinger, and Roopak Shah. Signature verification using a
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard S \"a ckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. In Advances in neural information processing systems , pages 737--744, 1994
work page 1994
-
[6]
Super-resolution with deep convolutional sufficient statistics
Joan Bruna, Pablo Sprechmann, and Yann LeCun. Super-resolution with deep convolutional sufficient statistics. arXiv preprint arXiv:1511.05666 , 2015
arXiv 2015
-
[7]
Learning to rank using gradient descent
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning , pages 89--96. ACM, 2005
work page 2005
-
[8]
Deep Learning-based Image Super-Resolution Considering Quantitative and Perceptual Quality
Jun-Ho Choi, Jun-Hyuk Kim, Manri Cheon, and Jong-Seok Lee. Deep learning-based image super-resolution considering quantitative and perceptual quality. arXiv preprint arXiv:1809.04789 , 2018
work page Pith review arXiv 2018
Show all 43 references
-
[9]
Learning a similarity metric discriminatively, with application to face verification
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on , volume 1, pages 539--546. IEEE, 2005
2005
-
[10]
Learning a deep convolutional network for image super-resolution
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional network for image super-resolution. In European conference on computer vision , pages 184--199. Springer, 2014
2014
-
[11]
Accelerating the super-resolution convolutional neural network
Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the super-resolution convolutional neural network. In European Conference on Computer Vision , pages 391--407. Springer, 2016
2016
-
[12]
Generating images with perceptual similarity metrics based on deep networks
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems , pages 658--666, 2016
2016
-
[13]
Suppressing model overfitting for image super-resolution networks
Ruicheng Feng, Jinjin Gu, Yu Qiao, and Chao Dong. Suppressing model overfitting for image super-resolution networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 0--0, 2019
2019
-
[14]
Blind super-resolution with iterative kernel correction
Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong. Blind super-resolution with iterative kernel correction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1604--1613, 2019
2019
-
[15]
Deep backprojection networks for super-resolution
Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. Deep backprojection networks for super-resolution. In Conference on Computer Vision and Pattern Recognition , 2018
2018
-
[16]
Modulating image restoration with continual levels via adaptive feature modification layers
Jingwen He, Chao Dong, and Yu Qiao. Modulating image restoration with continual levels via adaptive feature modification layers. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019
2019
-
[17]
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026--1034, 2015
2015
-
[18]
Perceptual losses for real-time style transfer and super-resolution
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision , pages 694--711. Springer, 2016
2016
-
[19]
Convolutional neural networks for no-reference image quality assessment
Le Kang, Peng Ye, Yi Li, and David Doermann. Convolutional neural networks for no-reference image quality assessment. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1733--1740, 2014
2014
-
[20]
Accurate image super-resolution using very deep convolutional networks
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1646--1654, 2016
2016
-
[21]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014
2014 arXiv
-
[22]
Photo-realistic single image super-resolution using a generative adversarial network
Christian Ledig, Lucas Theis, Ferenc Husz \'a r, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR , volume 2, pa...
2017
-
[23]
Blind image quality assessment using a general regression neural network
Chaofeng Li, Alan Conrad Bovik, and Xiaojun Wu. Blind image quality assessment using a general regression neural network. IEEE Transactions on Neural Networks , 22(5):793--799, 2011
2011
-
[24]
Enhanced deep residual networks for single image super-resolution
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In The IEEE conference on computer vision and pattern recognition (CVPR) workshops , volume 1, page 4, 2017
2017
-
[25]
Rankiqa: Learning from rankings for no-reference image quality assessment
Xialei Liu, Joost van de Weijer, and Andrew D Bagdanov. Rankiqa: Learning from rankings for no-reference image quality assessment. Computer Vision and Pattern Recognition, https://arxiv. org/abs/1707.08347 v1 , 2017
2017 arXiv
-
[26]
Learning a no-reference quality metric for single-image super-resolution
Chao Ma, Chih-Yuan Yang, Xiaokang Yang, and Ming-Hsuan Yang. Learning a no-reference quality metric for single-image super-resolution. Computer Vision and Image Understanding , 158:1--16, 2017
2017
-
[27]
A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Computer Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE Internationa...
2001
-
[28]
Conditional generative adversarial nets
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 , 2014
2014 arXiv
-
[29]
completely blind
Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making a" completely blind" image quality analyzer. IEEE Signal Process. Lett. , 20(3):209--212, 2013
2013
-
[30]
Relative attributes
Devi Parikh and Kristen Grauman. Relative attributes. In Computer Vision (ICCV), 2011 IEEE International Conference on , pages 503--510. IEEE, 2011
2011
-
[31]
Enhancenet: Single image super-resolution through automated texture synthesis
Mehdi SM Sajjadi, Bernhard Sch \"o lkopf, and Michael Hirsch. Enhancenet: Single image super-resolution through automated texture synthesis. In Computer Vision (ICCV), 2017 IEEE International Conference on , pages 4501--4510. IEEE, 2017
2017
-
[32]
Ranking cgans: Subjective control over semantic image attributes
Yassir Saquil, Kwang In Kim, and Peter Hall. Ranking cgans: Subjective control over semantic image attributes. In Proc. of British Machine Vision Conference (BMVC) , 2018
2018
-
[33]
Very deep convolutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014
2014 arXiv
-
[34]
Recovering realistic texture in image super-resolution by deep spatial feature transform
Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. arXiv preprint arXiv:1804.02815 , 2018
2018 arXiv
-
[35]
Esrgan: Enhanced super-resolution generative adversarial networks
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In The European Conference on Computer Vision (ECCV) Workshops , September 2018
2018
-
[36]
Deep relative attributes
Xiaoshan Yang, Tianzhu Zhang, Changsheng Xu, Shuicheng Yan, M Shamim Hossain, and Ahmed Ghoneim. Deep relative attributes. IEEE Transactions on Multimedia , 18(9):1832--1842, 2016
2016
-
[37]
No-reference image quality assessment based on visual codebook
Peng Ye and David S Doermann. No-reference image quality assessment based on visual codebook. In ICIP , pages 3089--3092. Citeseer, 2011
2011
-
[38]
Learning to compare image patches via convolutional neural networks
Sergey Zagoruyko and Nikos Komodakis. Learning to compare image patches via convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4353--4361, 2015
2015
-
[39]
On single image scale-up using sparse-representations
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse-representations. In International conference on curves and surfaces , pages 711--730. Springer, 2010
2010
-
[40]
Single image super-resolution with non-local means and steering kernel regression
Kaibing Zhang, Xinbo Gao, Dacheng Tao, Xuelong Li, et al. Single image super-resolution with non-local means and steering kernel regression. Image , 11:12, 2012
2012
-
[41]
An edge-guided image interpolation algorithm via directional filtering and data fusion
Lei Zhang and Xiaolin Wu. An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE transactions on Image Processing , 15(8):2226--2238, 2006
2006
-
[42]
Residual dense network for image super-resolution
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018
2018
-
[43]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.