Algorithm Selection for Image Quality Assessment

Dietmar Saupe; Hanhe Lin; Markus Wagner; Shujun Li

REVIEW 3 major objections 4 minor 11 references

Algorithm Selection for Image Quality Assessment

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Algorithm selection can't beat the best single image-quality method

desk verdict First real test of algorithm selection for blind IQA; the negative result is plausible and the AutoFolio experiment is the strong witness, but the deep-learning protocol is too undertrained to carry much weight. read the letter →

arxiv 1908.06911 v1 pith:V2DZVOPH submitted 2019-08-19 cs.CV

Markus Wagner , Hanhe Lin , Shujun Li , Dietmar Saupe This is my paper · ORCID

classification cs.CV

keywords blindimagequalityassessmentalgorithmselectionvirtualbestmethodAutoFoliodeeplearningKonIQ-10kSiamesenetworknoisyevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a learned oracle—an algorithm selector that picks, for each input image, the best among eight blind image-quality assessment methods—can outperform the single best method on the KonIQ-10k benchmark. An ideal oracle would raise Spearman correlation from 0.921 to 0.978, but both an AutoFolio-based selector and two deep-learning classifiers failed to beat KonCept512, the best single method. The authors attribute the failure to noisy quality evaluation: perceptually equivalent images can receive different numerical scores, and the oracle's advantage may be mostly an artifact of that noise rather than learnable structure. If this is right, the practical ceiling for blind IQA may already be near the best single method, and research should shift to measuring and reducing the noisiness of IQA methods.

What carries the argument

The central devices are the virtual best method (VBM) and the algorithm-selection pipeline built to mimic it. The VBM acts as the oracle: for each test image it picks the one of eight methods whose quality estimate is closest to the human mean opinion score, defining an upper bound that any selector would need to learn. AutoFolio searches over selector models and hyperparameters using instance features and per-method errors; the deep approach uses InceptionResNetV2 as a shared CNN base, either as a direct eight-way classifier or as a Siamese error predictor with eight output modules. The paper's explanatory mechanism is the noise model: an IQA method evaluated on perceptually indistinguishable images yields a distribution of scores, so the oracle can appear to win by choosing the method whose noise happens to land closest to the human rating, an effect no training set can teach a selector.

What would settle it

Rate multiple perceptually indistinguishable versions of each KonIQ-10k test image with the same eight methods and human subjects, then recompute the oracle's SROCC and MAE against the averaged human scores. If the oracle advantage over KonCept512 mostly disappears, the noise-exploitation explanation is confirmed; if it persists, the failure of the trained selectors is unexplained and a better selector should be demonstrable.

Watch

Extended reading notes

Core claim

On the KonIQ-10k test set, the paper shows that the gap between the single best method (KonCept512, SROCC 0.921, MAE 4.154) and the virtual best method—the oracle that always selects the method closest to the human mean opinion score—is large (SROCC 0.978, MAE 2.069). It then attempts to learn that oracle with AutoFolio, a state-of-the-art automatic algorithm selector, and with two deep networks: an InceptionResNetV2 classifier assigning each image to one of the eight methods, and a Siamese error-prediction network that estimates each method's absolute error. None of these surpassed KonCept512: AutoFolio degenerated to always choosing KonCept512, and the deep classifiers yielded SROCCs of 0.871 and 0.908. The paper concludes that algorithm selection for blind IQA does not improve over the single best method, and conjectures that the oracle's apparent superiority reflects noise in image-quality evaluation rather than predictable variation across images.

Load-bearing premise

The load-bearing premise is that the training protocols—24 hours of AutoFolio search over 500 models and only 10 epochs of fine-tuning for the deep networks—gave the learned selectors a fair chance to capture any learnable image-to-method structure; if these regimes underfit, the failure is an artifact of training effort, not a property of the problem.

Editorial extensions

If this is right

If no learned selector can beat KonCept512 on KonIQ-10k, then a single well-chosen BIQA method is the practical default for this benchmark, and adding a portfolio of eight methods does not help by itself.
The large VBM gap should not be read as an achievable performance target for algorithm selection; part or all of it may be due to the evaluation noise floor.
Measuring the noisiness of individual IQA methods becomes a first-class problem: methods with lower noise may be preferable even when their mean accuracy is similar.
Denoising IQA predictions—for instance by averaging over perceptually equivalent inputs—could shrink the oracle gap and make algorithm selection a more meaningful exercise.
The negative result carries a methodological lesson: before declaring that an oracle gap is learnable, one should check whether the gap survives under a noise-corrected evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same noise argument may explain why the deep classifiers performed worse than AutoFolio: if per-image labels (which method wins) are dominated by noise, training a classifier on those labels teaches the network the noise pattern, not a stable mapping. A testable extension would be to train on labels averaged over multiple perceptually equivalent renditions of each image.
A natural extension is to apply the same VBM-versus-selector comparison to full-reference IQA or to distortion-specific subsets; if the oracle gap is broader than KonIQ-10k's authentic distortions, the conclusion would generalize, and if it shrinks, the noise floor is dataset-specific.
If the noisiness conjecture is correct, then the VBM's SROCC of 0.978 is not a meaningful upper bound for algorithmic performance on this dataset; a sharper bound would come from the best achievable method under a noise-removed ground truth, which one could construct by repeated rating of visually identical images and averaging the human scores.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The one thing to know: this is the first attempt to apply formal algorithm selection to blind IQA, and the negative result is real but narrower than the abstract implies. On KonIQ-10k, an oracle selector would beat KonCept512 by a large margin, but AutoFolio, after 24 hours of search over 500 models, learned to pick KonCept512 for every image, and two deep classifiers fell short. The paper's central claim—algorithm selection as tried here does not beat the single best method—is supported by the reported experiments.

What's new: the formulation of BIQA as algorithm selection with VBM/SBM, the use of AutoFolio, a siamese error-prediction network, and a noisiness conjecture. The noisiness idea is genuinely testable: if the oracle advantage is partly noise exploitation, then denoising individual methods should shrink the gap. The paper is honest to label it as a conjecture.

What it does well: clear setup, a sensible portfolio of methods, and the AutoFolio experiment is convincing. The oracle is used only as an upper bound, not as training signal, so no circularity. The citation pattern looks fine.

Soft spots: the deep training protocol is too thin to rule out underfitting. Ten epochs of fine-tuning InceptionResNetV2, with no learning curves, convergence checks, or data augmentation, is a live alternative explanation for those two failures. That matters because the paper counts those failures as part of the negative conclusion. The AutoFolio result, however, does not depend on that protocol and is the strongest witness. Also: one dataset, no confidence intervals or significance tests, and no released code or data. The noise conjecture is speculative, but it is clearly framed as speculation.

The central negative claim holds up as reported for AutoFolio. The deep-learning leg is weaker but not load-bearing. The paper deserves peer review: a serious referee would ask for a second dataset, more training budget or convergence evidence, and a plan for testing noisiness.

For a reading group, I would bring it as a cautionary example of oracle gaps not being learnable structure. I would not cite it in my own work next year unless I were directly working on BIQA algorithm selection.

Referee Report

3 major / 4 minor

Summary. The paper addresses whether algorithm selection (AS) can improve blind image quality assessment (BIQA) on the KonIQ-10k dataset. Using a portfolio of eight BIQA methods, the authors first show that a virtual best method (VBM) that chooses the best method per image would achieve SROCC 0.978 versus 0.921 for the single best method, KonCept512. They then attempt to learn this oracle in three ways: AutoFolio with hand-crafted features, a CNN classifier directly predicting the best method, and a Siamese network predicting per-method absolute errors. All attempts fail to beat the single best method on the test set: AutoFolio collapses to always selecting KonCept512, and the deep approaches yield SROCCs of 0.871 and 0.908. The paper concludes that algorithm selection for BIQA does not improve over the single best method and conjectures that the VBM advantage is partly due to unavoidable noise in per-image quality evaluation.

Significance. If the negative result is accepted, this is a useful cautionary data point for the algorithm-selection community and for BIQA researchers: despite a large oracle gap, state-of-the-art selectors may fail on perceptual quality tasks because the per-image advantage of the VBM may be partly noise. The paper is honest about the failure and makes the AutoFolio search systematic (24 hours, over 500 models). It also provides a clear, reproducible setup based on the public KonIQ-10k dataset and an explicit comparison of SBM, VBM, and AS. However, the evidence base is narrow (a single dataset), the deep-learning training protocol is not shown to be sufficient, and no statistical significance tests are reported; these weaknesses currently limit the strength of the general conclusion.

major comments (3)

[Section IV] The deep-learning experiments do not demonstrate that the selectors were trained to convergence. InceptionResNetV2 was fine-tuned for exactly 10 epochs with SGD at alpha=0.0001, no learning curves or convergence checks are shown, and no data augmentation is mentioned. Given that the network is large and the task is a new 8-way labeling problem, underfitting is a plausible alternative explanation for the failure of Approach 1 and Approach 2. These results therefore cannot support the paper's negative conclusion unless the training protocol is shown to suffice or the conclusion is explicitly restricted to the AutoFolio experiments.
[Section III, Table II] The claim that algorithm selection 'did not yield a significant improvement' is not backed by any significance test or confidence interval. The only case where AS numerically beats the single best method is the MAE comparison after excluding KonCept512 (6.665 versus 6.792), while the corresponding SROCC is worse (0.784 versus 0.805). Without paired bootstrap or permutation tests, these differences may be pure noise. Please add statistical tests or soften the 'significant' wording.
[Sections II and V] The central negative claim is established on a single dataset, the KonIQ-10k test set, yet the conclusion is stated generally: 'algorithm selection for BIQA does not yield an improvement over the single best method.' Because BIQA methods and instance-level selectors may behave differently on other databases, either add experiments on at least one additional dataset (e.g., LIVEC or another authentic-distortion benchmark) or explicitly restrict the claim to KonIQ-10k and note that generalization remains untested.

minor comments (4)

[Section III] The phrase 'principle component analysis' should be 'principal component analysis'.
[Section V] The 'noisiness' conjecture is presented as a potential explanation, but it is not quantified or tested. Please either add an experiment (e.g., measuring the variance of each IQA method under small pixel perturbations or using repeated MOS data) or clearly label the conjecture as a hypothesis for future work rather than as an observed property.
[Section IV] For Approach 1, the paper reports a classification accuracy of 29.3% on an 8-class problem, but no baseline or confusion matrix is given. Reporting the majority-class baseline and per-class accuracies would make this result interpretable.
[General] The paper does not state whether the selector scripts or feature computation code are publicly available; releasing them would strengthen reproducibility, since the underlying dataset is public.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the algorithm-selection experiments are standard supervised evaluations, and the oracle is used only as an unreachable upper bound.

full rationale

The paper's derivation chain is not circular. The virtual best method (VBM) is defined as the per-image argmin of |M(I)-MOS(I)| and is used only as an upper bound in Table I; it is not supplied as a training signal to AutoFolio or the deep networks. AutoFolio is trained on instance features and training-set performance matrices and evaluated on the held-out KonIQ-10k test set, so its failure to beat KonCept512 is an empirical result, not a consequence of any fitted parameter being renamed as a prediction. The deep classifiers (Approaches 1 and 2) are trained on ground-truth-derived labels and errors, validated on a 1,000-image split, and tested on the held-out test set; this is standard supervised benchmarking. The 5-parameter logistic calibration is fit on the training set and applied to test predictions, which is standard preprocessing rather than a fitted 'prediction.' The self-citation to KonIQ-10k [4] is a public benchmark dataset with human MOS values, not an unverified theorem, so it is not load-bearing circularity. The paper explicitly labels its noise explanation as a conjecture in Section V, which is honest speculation rather than a disguised derivation. The possible underfitting of the 10-epoch deep training is a correctness risk or alternative explanation for the negative result, but it is not a circularity because the central claim does not reduce by construction to its inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim rests on standard supervised learning and one dataset. Four free parameters enter through preprocessing and training choices; one ad hoc conjecture (noisiness) does not support the negative result but motivates future work.

free parameters (4)

5-parameter logistic alignment coefficients = Not reported (one fit per method on the KonIQ-10k training set)
Section II: predictions of all eight methods are aligned to [0,100] MOS via the 5-parameter logistic; MAE comparisons depend on this fitted alignment.
PCA component counts for CORNIA, HOSA, KonCept512 = 100 components for each of the three methods
Section III: to limit selection bias and complexity, the authors reduce each method's features to 100 principal components, a hand-chosen threshold that changes the selector's input.
Deep network hyperparameters = learning rate 0.0001, 10 epochs, batch size 64, dropout 0.25 and 0.5
Section IV: chosen by validation performance; the negative result depends on these training choices.
AutoFolio configuration search = 24 hours of search, over 500 models
Section III: the algorithm selector's internal hyperparameters are tuned on training data; the conclusion that selection fails is conditional on this search budget.

assumptions (4)

domain assumption MOS labels in KonIQ-10k are reliable ground truth for subjective quality.
The entire evaluation treats mean opinion scores as the target and assumes the test labels are not themselves dominated by annotation noise.
domain assumption Train and test sets are drawn from the same distribution, and features generalize.
The selectors are trained on KonIQ-10k training images and evaluated on its test set; if the split is unrepresentative, negative results may not transfer.
standard math The algorithm selection framework of Rice applies with cost defined as absolute error against MOS.
Section I formalizes the selection problem; this is a standard definition, not an empirical assumption.
ad hoc to paper The noisiness conjecture: every image has perceptually indistinguishable variants whose IQA scores form a distribution.
Section V introduces this as a post hoc explanation for the oracle gap; it is not measured or tested, only conjectured.

invented entities (1)

Noisiness of IQA methods
purpose: Explains why the virtual best method beats the single best method and why learned selectors cannot match the oracle.
Section V proposes that an IQA score contains an added noise term due to visually equivalent images. No direct measurement or falsifiable prediction is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Algorithm Selection for Image Quality Assessment." pith.science (2026). https://pith.science/paper/V2DZVOPH

@misc{pith2026190806911,
  author       = {Pith},
  title        = {Pith review of: Algorithm Selection for Image Quality Assessment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V2DZVOPH}},
  note         = {Machine review of arXiv:1908.06911}
}

read the original abstract

Subjective perceptual image quality can be assessed in lab studies by human observers. Objective image quality assessment (IQA) refers to algorithms for estimation of the mean subjective quality ratings. Many such methods have been proposed, both for blind IQA in which no original reference image is available as well as for the full-reference case. We compared 8 state-of-the-art algorithms for blind IQA and showed that an oracle, able to predict the best performing method for any given input image, yields a hybrid method that could outperform even the best single existing method by a large margin. In this contribution we address the research question whether established methods to learn such an oracle can improve blind IQA. We applied AutoFolio, a state-of-the-art system that trains an algorithm selector to choose a well-performing algorithm for a given instance. We also trained deep neural networks to predict the best method. Our results did not give a positive answer, algorithm selection did not yield a significant improvement over the single best method. Looking into the results in depth, we observed that the noise in images may have played a role in why our trained classifiers could not predict the oracle. This motivates the consideration of noisiness in IQA methods, a property that has so far not been observed and that opens up several interesting new research questions and applications.

Figures

Figures reproduced from arXiv: 1908.06911 by the authors.

**Figure 2.** The proposed siamese network architecture for error prediction. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 10 canonical work pages

[1]

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page
[2]

J. R. Rice, ``The algorithm selection problem,'' in Advances in Computers. 1em plus 0.5em minus 0.4em Elsevier, 1976, vol. 15, pp. 65--118

work page 1976
[3]

L. Xu, W. Lin, and C.-C. J. Kuo, ``Metrics fusion,'' in Visual Quality Assessment by Machine Learning, ser. SpringerBriefs in Electrical and Computer Engineering. 1em plus 0.5em minus 0.4em Springer Singapore, 2015, ch. 5, pp. 93--122

work page 2015
[4]

Oszust, ``Decision fusion for image quality assessment using an optimization approach,'' IEEE Signal Processing Letters, vol

M. Oszust, ``Decision fusion for image quality assessment using an optimization approach,'' IEEE Signal Processing Letters, vol. 23, no. 1, pp. 65--69, 2016

work page 2016
[5]

H. Lin, V. Hosu, and D. Saupe, `` KonIQ-10K : T owards an ecologically valid and large-scale IQA database,'' arXiv:1803.08489 (cs.CV), 2018

arXiv 2018
[6]

H. R. Sheikh, M. F. Sabir, and A. C. Bovik, ``A statistical evaluation of recent full reference image quality assessment algorithms,'' IEEE Transactions on Image Processing, vol. 15, no. 11, pp. 3440--3451, 2006

work page 2006
[7]

Bischl, P

B. Bischl, P. Kerschke, L. Kotthoff, M. Lindauer, Y. Malitsky, A. Frech\'ette, H. Hoos, F. Hutter, K. Leyton-Brown, K. Tierney, and J. Vanschoren, ``Aslib: A benchmark library for algorithm selection,'' Artificial Intelligence Journal (AIJ), vol. 237, pp. 41--58, 2016

work page 2016
[8]

Lindauer, H

M. Lindauer, H. Hoos, F. Hutter, and T. Schaub, ``Autofolio: An automatically configured algorithm selector,'' Journal of Artificial Intelligence Research, vol. 53, pp. 745--778, 2015

work page 2015

Show all 11 references

[9]

Szegedy, S

C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, ``Inception-v4, I nception- R es N et and the impact of residual connections on learning.'' in AAAI Conference on Artificial Intelligence (AAAI), vol. 4, 2017, p. 12

2017
[10]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, ``Image N et: A large-scale hierarchical image database,'' in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009, pp. 248--255

2009
[11]

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEco...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page

[2] [2]

J. R. Rice, ``The algorithm selection problem,'' in Advances in Computers. 1em plus 0.5em minus 0.4em Elsevier, 1976, vol. 15, pp. 65--118

work page 1976

[3] [3]

L. Xu, W. Lin, and C.-C. J. Kuo, ``Metrics fusion,'' in Visual Quality Assessment by Machine Learning, ser. SpringerBriefs in Electrical and Computer Engineering. 1em plus 0.5em minus 0.4em Springer Singapore, 2015, ch. 5, pp. 93--122

work page 2015

[4] [4]

Oszust, ``Decision fusion for image quality assessment using an optimization approach,'' IEEE Signal Processing Letters, vol

M. Oszust, ``Decision fusion for image quality assessment using an optimization approach,'' IEEE Signal Processing Letters, vol. 23, no. 1, pp. 65--69, 2016

work page 2016

[5] [5]

H. Lin, V. Hosu, and D. Saupe, `` KonIQ-10K : T owards an ecologically valid and large-scale IQA database,'' arXiv:1803.08489 (cs.CV), 2018

arXiv 2018

[6] [6]

H. R. Sheikh, M. F. Sabir, and A. C. Bovik, ``A statistical evaluation of recent full reference image quality assessment algorithms,'' IEEE Transactions on Image Processing, vol. 15, no. 11, pp. 3440--3451, 2006

work page 2006

[7] [7]

Bischl, P

B. Bischl, P. Kerschke, L. Kotthoff, M. Lindauer, Y. Malitsky, A. Frech\'ette, H. Hoos, F. Hutter, K. Leyton-Brown, K. Tierney, and J. Vanschoren, ``Aslib: A benchmark library for algorithm selection,'' Artificial Intelligence Journal (AIJ), vol. 237, pp. 41--58, 2016

work page 2016

[8] [8]

Lindauer, H

M. Lindauer, H. Hoos, F. Hutter, and T. Schaub, ``Autofolio: An automatically configured algorithm selector,'' Journal of Artificial Intelligence Research, vol. 53, pp. 745--778, 2015

work page 2015