{"id":"2384b768-0b95-4832-bcfd-fa3e481df861","arxiv_id":"1908.06382","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RankSRGAN uses a learned ranking network as a differentiable surrogate for perceptual metrics, enabling GAN-based super-resolution to be optimized directly toward NIQE, Ma, or PI.","lead":"This paper trains a 'Ranker' network to imitate non-differentiable image quality scores like NIQE, then uses it as a loss to guide a super-resolution GAN. The result is a flexible framework that improves perceptual quality scores and can combine the best parts of different super-resolution models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The rank-content loss in Eq. 9 is never validated on the generator's output distribution; the Ranker's pairwise training leaves its absolute score unanchored, so minimizing that score may not minimize the true perceptual metric.","rationale":"The reader's verdict is CONDITIONAL, and my read keeps that verdict. I agree with the identified weakness: distribution shift is the load-bearing risk. I sharpen it into two testable components: (1) the Ranker is trained for pairwise order, so its raw output is not anchored and may not be a valid scalar loss; (2) adversarial training moves the generator off the rank dataset's support. The paper does substantial empirical work: three-level rank dataset, ablations over rank datasets, metrics, rank-vs-regression, loss combinations, convergence curves, and a user study. These support the conclusion that RankSRGAN yields state-of-the-art NIQE/PI and preferred images, and they rule out some alternative explanations (e.g., pure regression labels do worse). However, none of them measures the rank loss's gradient alignment with the true metric on images that the generator actually produces. The proposed checkpoint-correlation test would settle this directly. If it passes, the conditional concerns are largely resolved; if it fails, the paper's central causal claim about the Ranker would need substantial revision.","tokens_in":18332,"tokens_out":5841,"duration_ms":66097,"concrete_test":"Re-run the Sec. 4.2 training and, at every checkpoint (e.g., every 50k iterations), feed current generator outputs on a held-out set such as PIRM-Test into both the frozen Ranker and the true perceptual metric (NIQE or the metric used for labeling). Compute the Spearman rank correlation between Ranker scores and true metric scores on these generated images. If the correlation remains high (near the 0.88 level) while rank-content loss decreases, the surrogate is tracking the metric and the concern is mitigated. If the correlation degrades substantially or changes sign while Eq. 9 still decreases, the Ranker is being exploited and the central causal claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that RankSRGAN optimizes the generator in the direction of perceptual metrics via the rank-content loss. The Ranker is trained with margin-ranking loss (Eq. 4-5) on pairs of super-resolved outputs from SRResNet, SRGAN and ESRGAN, and its only reported validation is SROCC 0.88 on a held-out split of that same rank dataset (Sec. 4.1). That validates ordering within the training distribution, not the use of the Ranker's raw score as a loss. A ranking objective determines scores only up to monotonic transformations; Eq. 9 minimizes sigmoid(R(G(x))) as if the score were an absolute quality measure, but nothing anchors the absolute scale to the metric. Moreover, the generator is trained adversarially (Sec. 4.2), so its output distribution shifts during training; the Ranker is applied to inputs increasingly unlike the three fixed SR methods in the rank dataset, with no distribution-shift analysis or adaptation. The final NIQE/PI gains and the ablation tables demonstrate that RankSRGAN produces better-scoring images, but they do not show that the rank-content loss is the cause; the same outcome could arise if the frozen Ranker acts as a regularizer biasing outputs toward SRGAN/ESRGAN-like images. Directional correctness of Eq. 9 on generated images is never directly measured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RankSRGAN proposes a super-resolution framework that augments SRGAN with a learned Ranker, a Siamese CNN trained with margin-ranking loss to mimic the ordering behavior of non-differentiable perceptual metrics such as NIQE, PI, and Ma. The Ranker is trained on rank datasets constructed from the outputs of SRResNet, SRGAN, and ESRGAN on DIV2K/Flickr2K. Once trained, the Ranker's scalar output is used to define a rank-content loss, and the generator is optimized by a weighted sum of perceptual, adversarial, and rank-content losses. Experiments on Set14, BSD100, and PIRM-Test report lower NIQE/PI than SRGAN and ESRGAN, with ablations on the rank dataset composition, the choice of perceptual metric, ranking versus regression for the Ranker, and the loss combination.","tokens_in":18625,"tokens_out":5569,"duration_ms":53225,"significance":"If the mechanism is sound, the idea of using a learned ranker as a differentiable surrogate for non-differentiable perceptual quality metrics is a useful addition to perceptual SR, and the paper demonstrates that this can combine the strengths of different SR algorithms. The ablations (rank vs. regression, different metrics, different rank datasets) are thoughtful and provide some evidence that the Ranker is not merely a black-box regularizer. The paper also reports SROCC of the Ranker and includes convergence curves, which is helpful. However, the central claim that the rank-content loss 'optimizes the generator in the direction of perceptual metrics' rests on an unvalidated assumption: that the Ranker's absolute score is a valid loss signal for images that fall outside its training distribution. The empirical gains are small and reported without error bars or significance tests, so the strength of the claim currently exceeds the evidence.","major_comments":[{"comment":"The rank-content loss in Eq. (9) is the sigmoid of the Ranker's raw output, but the Ranker is trained with a margin-ranking loss (Eq. (4)) that determines scores only up to an arbitrary monotonic transformation. Without calibration or a mechanism anchoring the absolute scale of R to the target perceptual metric, minimizing sigmoid(R(G(x))) is not theoretically guaranteed to minimize the metric. The paper should provide either a calibration step or empirical evidence that lower R(G(x)) consistently corresponds to better metric values on a validation set of images produced during training.","section":"Sec. 3.3, Eq. (9)"},{"comment":"The Ranker is trained only on outputs of SRResNet, SRGAN, and ESRGAN, and its accuracy is evaluated only via SROCC on a held-out split of that same rank dataset. During adversarial training, the generator's output distribution shifts (as seen in Figure 6), yet the paper does not measure how the Ranker's ranking accuracy behaves on the generator's intermediate or final outputs. Without this distribution-shift analysis, the key claim that the rank-content loss 'optimizes the generator in the direction of perceptual metrics' is not fully supported; it is possible that the Ranker acts as a regularizer that only biases outputs toward the training SR methods. Please evaluate the Ranker on generated images from the trained RankSRGAN and report the correlation between R(G(x)) and the metric.","section":"Sec. 4.1-4.2"},{"comment":"All quantitative results are reported as single-run means without error bars or significance tests. Given that the improvements over ESRGAN are often within a few hundredths of a NIQE/PI point (e.g., Table 2, Set14: NIQE 3.28 vs 3.28, PI 2.61 vs 2.61), it is unclear whether the claimed gains are statistically meaningful. I request variance across at least three independent training runs or a statistical significance test on the test datasets.","section":"Tables 2-7"}],"minor_comments":[{"comment":"The SROCC definition is written only for the case of no tied ranks; please clarify how ties are handled or state that the data contain no ties.","section":"Sec. 4.1, Eq. (11)"},{"comment":"The notation 'PMSR2-L + PMSR2-H' and 'PMSR1-L' is confusing; the subscripts should be defined explicitly.","section":"Sec. 3.4, Eq. (10)"},{"comment":"The caption does not match the bars; please indicate which bar corresponds to 'Metric Rank', 'Model Classification', 'ESRGAN', and 'SRGAN'.","section":"Figure 3"},{"comment":"The values of alpha_1 and alpha_2 are given as '{1,5}' but it is unclear which alpha corresponds to which row; please label the rows explicitly.","section":"Sec. 4.4, Table 7"},{"comment":"There are minor typos: 'beheviour' in Section 3.1, 'traning' in Section 4.2, and 'indifferentiable' should likely be 'non-differentiable' in the introduction.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is from a reputable group and the core idea is appealing, but the main concern is the lack of validation on the generator's output distribution and the absence of statistical rigor. I would recommend major revision. Additionally, the comparison is limited to SRGAN and ESRGAN; a broader comparison with other perceptual SR methods (e.g., EnhanceNet, SFTGAN) would strengthen the state-of-the-art claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RankSRGAN is a genuine contribution to perceptual super-resolution, and it deserves to go through peer review. The core idea is clean: train a Siamese Ranker on pairs of SR outputs from different methods, labeled by a non-differentiable metric like NIQE or PI, then use the Ranker's score as a loss to guide the generator. That is a practical way to optimize metrics that otherwise can't be backpropped through.\n\nWhat's best about the paper is the experimental discipline. The ablations cover the important axes: rank vs regression (the histograms in the supplement show why ranking helps separate SRGAN from ESRGAN), different metrics, different rank datasets, and different loss combinations. The metric-rank vs model-classification comparison is particularly convincing because it shows the Ranker is learning the metric's ordering, not just the source model. Held-out results on Set14, BSD100, and PIRM-Test are consistent, and the user study supports the perceptual improvement.\n\nThe soft spots are real but not fatal. First, the Ranker is only validated on the rank dataset distribution; there's no analysis of how its score relates to the true metric on the generator's output distribution, which shifts during adversarial training. I think the stress-test concern about the unanchored absolute score is less serious than it looks—for gradient descent, the ranking direction matters more than the scale—but the distribution-shift issue is a legitimate gap. The paper could have checked this by directly measuring the Ranker's SROCC against the metric on generated images during training.\n\nSecond, all quantitative tables are single-run means without error bars or significance tests. That's common in this subfield, but it limits how strongly one can claim \"state of the art.\" The loss weights in Eq. 12 are empirically tuned, which is fine, but it would be good to know how sensitive the results are.\n\nFinally, the paper doesn't release code or checkpoints, which is a shame for a method like this. Still, the formal setup is coherent, the empirical story is consistent, and the method is a real step forward for optimizing perceptual metrics in SR. I'd send this to a serious referee, and I'd recommend the authors address the distribution-shift point and add error bars in revision.","headline":"A genuine, well-ablated approach to optimizing non-differentiable perceptual metrics in SR; the main gap is how the frozen Ranker behaves on the generator's shifting output distribution.","tokens_in":19152,"tokens_out":3984,"would_cite":true,"duration_ms":39290,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A learned Ranker lets super-resolution GANs optimize perceptual scores directly, and the paper reports best NIQE/PI on three benchmarks.","keywords":["RankSRGAN","single image super-resolution","perceptual metrics","learning to rank","Siamese network","generative adversarial network","NIQE","no-reference image quality assessment"],"falsifier":"Take a trained RankSRGAN and compare its Ranker scores against the true perceptual metric on the generator's own outputs: if the Spearman correlation drops well below the reported validation value of 0.88, or if a model trained with the rank-content loss improves its Ranker score without improving the actual metric, the Ranker has stopped tracking the target.","tokens_in":18139,"feed_emoji":"🖼️","tokens_out":7105,"duration_ms":62260,"temperature":0.7,"pith_summary":"Perceptual super-resolution is judged by no-reference metrics such as NIQE, Ma, and PI, but these metrics are not differentiable, so they cannot directly train a generator. RankSRGAN adds a Ranker, a Siamese CNN trained with margin-ranking loss on pairs of super-resolved images labeled by the chosen metric, to supply a differentiable rank-content loss. The generator is then optimized toward higher perceptual scores while keeping the standard SRGAN objective. The authors show this lets a standard SRGAN beat SRGAN and ESRGAN on NIQE and PI on Set14, BSD100, and PIRM-Test, and that the framework generalizes across metrics and rank datasets.","feed_headline":"Ranker lets super-resolution GANs optimize perceptual scores","feed_subtitle":"A Siamese Ranker learns NIQE/PI orderings and guides a standard SRGAN to beat SRGAN and ESRGAN on three benchmarks.","key_machinery":"The Ranker is a Siamese CNN that takes two images and outputs scalar ranking scores, trained with margin-ranking loss to match the pair-wise order of a chosen perceptual metric. It converts an indifferentiable metric into a differentiable objective; the concrete hook is the rank-content loss $L_R = \\mathrm{sigmoid}(R(G(x_i)))$, which lets gradients flow from the predicted rank back into the generator. The metric-rank labeling scheme, rather than model classification, is what lets the framework exceed the upper bound of any single SR algorithm.","core_discovery":"The central claim is that any non-differentiable perceptual metric can be optimized through a learned ranking surrogate. The Ranker is a Siamese VGG-style network trained to reproduce the pair-wise ordering induced by a metric on outputs of several SR algorithms; training uses margin-ranking loss rather than regression, so the surrogate only needs ordinal agreement. The rank-content loss $L_R = \\mathrm{sigmoid}(R(G(x_i)))$ is added to the standard SRGAN objective, and lowering it raises the predicted perceptual rank. Because the rank dataset contains images where SRGAN and ESRGAN each win in mixed order, the generator is guided to adopt the better parts of either algorithm. The reported experiments give the best NIQE and PI among the compared methods on Set14, BSD100, and PIRM-Test, preserve PSNR on PIRM-Test, and user studies prefer RankSRGAN over both baselines.","pith_inferences":["An unstated consequence is that any non-differentiable evaluation score expressible as pair-wise preferences, not just NR-IQA metrics, could be optimized with the same Ranker mechanism, including full-reference quality measures or task-specific scores.","A risk the paper does not address is reward hacking: the generator may exploit the Ranker's learned scoring function rather than the true metric if training pushes outputs outside the rank dataset's distribution, and the reported validation SROCC of 0.88 does not guarantee the same ordinal accuracy on the generator's own outputs.","A testable extension is to build rank datasets from more diverse or deliberately adversarial SR models, or to update the Ranker periodically during GAN training, to see whether the ranking signal stays aligned with the target metric under distribution shift.","The metric-rank versus model-classification comparison suggests an ensemble view: a Ranker over many specialist generators could yield a single model that inherits each specialist's best behavior per image."],"forward_implications":["Swapping the labels in the rank dataset swaps the target metric: RankSRGAN trained with NIQE, Ma, or PI labels improves that metric, so the same framework can chase any pair-wise rankable quality score.","Because the Ranker learns order rather than absolute values, it separates SR algorithms whose metric values are close (for example SRGAN vs ESRGAN on NIQE), giving the generator a usable gradient where regression would collapse.","The metric-rank upper bound can exceed every SR algorithm in the rank dataset, so adding better or complementary guiding algorithms, even ground-truth HR, should continue to improve perceptual quality.","The rank-content loss can be weighted against MSE to navigate the perception-distortion tradeoff, letting practitioners choose a preferred balance between visual quality and fidelity.","The generator's parameter count and inference cost stay the same as SRGAN; only training time grows by about 1.18 times."],"supporting_citations":[{"why":"provides the base SRGAN generator, discriminator, adversarial loss, and perceptual loss that RankSRGAN extends","marker":"[22]"},{"why":"supplies ESRGAN as the second SR algorithm in the rank dataset and as the main state-of-the-art baseline","marker":"[35]"},{"why":"defines the NIQE perceptual metric whose ranking behavior the Ranker learns in the main experiments","marker":"[29]"},{"why":"introduces the PI metric and PIRM-Test benchmark, with human-correlation evidence that motivates optimizing perceptual metrics","marker":"[2]"},{"why":"provides the Siamese-network learning-to-rank approach for no-reference image quality assessment that the Ranker architecture draws on","marker":"[25]"},{"why":"supplies the margin-ranking loss used to train the Ranker from pair-wise ordering labels","marker":"[7]"},{"why":"defines the Ma perceptual metric used to test generalization of the Ranker to another target","marker":"[26]"},{"why":"provides the DIV2K and Flickr2K datasets used to generate the rank dataset and to train RankSRGAN","marker":"[1]"}],"fun_headline_variants":["Ranker-based GAN optimizes perceptual metrics for super-resolution","Super-resolution GANs learn to rank perceptual quality","Siamese Ranker guides SRGAN to better NIQE and PI scores","Optimizing non-differentiable perceptual metrics via learned ranking","RankSRGAN: a ranker to improve GAN-based super-resolution"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Ranker is trained only on super-resolved outputs from SRResNet, SRGAN, and ESRGAN under a fixed perceptual metric, and the method assumes its ranking signal stays valid for the generator's changing output distribution during adversarial training.","fun_headline_variants_meta":{"raw":{"variants":["Ranker-based GAN optimizes perceptual metrics for super-resolution","Super-resolution GANs learn to rank perceptual quality","Siamese Ranker guides SRGAN to better NIQE and PI scores","Optimizing non-differentiable perceptual metrics via learned ranking","RankSRGAN: a ranker to improve GAN-based super-resolution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000311,"raw_usage":{"total_tokens":1762,"prompt_tokens":928,"completion_tokens":834,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":746}},"tokens_in":544,"tokens_out":834,"duration_ms":8115,"temperature":1.0,"reasoning_tokens":746,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:46:23.329094+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained RankSRGAN and compare its Ranker scores against the true perceptual metric on the generator's own outputs: if the Spearman correlation drops well below the reported validation value of 0.88, or if a model trained with the rank-content loss improves its Ranker score without improving the actual metric, the Ranker has stopped tracking the target.","supporting_citations":[{"cited_title":"Photo-realistic single image super-resolution using a generative adversarial network","cited_arxiv_id":null,"evidence_quote":"provides the base SRGAN generator, discriminator, adversarial loss, and perceptual loss that RankSRGAN extends"},{"cited_title":"Esrgan: Enhanced super-resolution generative adversarial networks","cited_arxiv_id":null,"evidence_quote":"supplies ESRGAN as the second SR algorithm in the rank dataset and as the main state-of-the-art baseline"},{"cited_title":"completely blind","cited_arxiv_id":null,"evidence_quote":"defines the NIQE perceptual metric whose ranking behavior the Ranker learns in the main experiments"},{"cited_title":"RankIQA: Learning from Rankings for No-reference Image Quality Assessment","cited_arxiv_id":"1707.08347","evidence_quote":"provides the Siamese-network learning-to-rank approach for no-reference image quality assessment that the Ranker architecture draws on"},{"cited_title":"Learning to rank using gradient descent","cited_arxiv_id":null,"evidence_quote":"supplies the margin-ranking loss used to train the Ranker from pair-wise ordering labels"},{"cited_title":"Learning a no-reference quality metric for single-image super-resolution","cited_arxiv_id":null,"evidence_quote":"defines the Ma perceptual metric used to test generalization of the Ranker to another target"},{"cited_title":"Ntire 2017 challenge on single image super-resolution: Dataset and study","cited_arxiv_id":null,"evidence_quote":"provides the DIV2K and Flickr2K datasets used to generate the rank dataset and to train RankSRGAN"}],"review_version":1}