{"id":"c4b11346-b6da-4fb9-8cc8-942a6bfbd8dd","arxiv_id":"1908.06911","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Algorithm selection, via AutoFolio or deep networks, fails to improve blind image quality assessment on KonIQ-10k over the single best method, and the oracle's edge is attributed to irreducible label noise.","lead":"What happens if you try to train a computer to pick, for each image, the best of eight quality-scoring algorithms? The trained selectors never beat simply using the best single algorithm. The paper reports this negative result and suggests that the apparent advantage of an ideal oracle comes from unpredictable noise in image quality scores.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The negative claim depends on the untested sufficiency of the deep training protocol: 10 epochs of fine-tuning with no convergence check leaves underfitting as a live alternative explanation.","rationale":"The paper's headline negative result is an empirical universal: none of the tested selectors beat KonCept512. My read of the experiments is that AutoFolio's 24-hour, 500-model search is a credible probe of classic algorithm selection, and its failure with the full portfolio is real evidence. However, the deep learning attempts are not yet at the same evidentiary level. Section IV specifies 10 epochs and no convergence diagnostics; without knowing whether validation performance had plateaued, underfitting is not merely a theoretical possibility. The fact that Approach 2 reaches 0.908 SROCC, close to KonCept512's 0.921, makes it particularly sensitive to training budget: a few more epochs could cross the threshold. The paper's own noise conjecture is plausible but explicitly conjectural and cannot be used to rule out a training-artifact explanation. Since the reader's CONDITIONAL verdict already flags training sufficiency as a condition, I do not change the verdict; my concrete check would resolve the open question. I do not see evidence of internal inconsistency or overreach beyond the depth of the training protocol; the paper is appropriately tentative in saying 'It seems.'","tokens_in":5485,"tokens_out":9801,"duration_ms":114865,"concrete_test":"Retrain both deep approaches (Section IV) to convergence, defined by a plateau in validation SROCC/MAE for at least 10 epochs, with early stopping, cosine LR schedule from 1e-4, random horizontal flips/crops, and the same 8,058/1,000/2,015 split. If the best convergent model fails to beat KonCept512 in SROCC and MAE on the test set (with a paired bootstrap or Wilcoxon test), the underfitting concern is resolved; if it beats KonCept512 on either metric, the paper's negative claim is falsified as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section V: 'algorithm selection for BIQA does not yield an improvement over the single best method') is supported by three experiments, but only the AutoFolio experiment (Section III) is a strong witness for an established selector. The deep selectors in Section IV are fine-tuned for exactly 10 epochs with SGD at alpha=0.0001 on roughly 7,000 training images (after the 1,000-image validation split), and the paper reports the best validation model without showing learning curves or convergence evidence. For a large InceptionResNetV2 network, 10 epochs can be too short for a new 8-way labeling task, especially when labels are noisy; there is also no mention of data augmentation. If the networks are underfit, the failure to beat KonCept512 (SROCC 0.921; deep approaches 0.871 and 0.908) may reflect training budget rather than the non-learnability of the oracle selection. AutoFolio's failure narrows but does not eliminate this possibility, because the paper advances the deep results as part of the same negative conclusion. The noise conjecture at the end of Section V is explicitly a conjecture and does not independently establish that no additional training would help.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses whether algorithm selection (AS) can improve blind image quality assessment (BIQA) on the KonIQ-10k dataset. Using a portfolio of eight BIQA methods, the authors first show that a virtual best method (VBM) that chooses the best method per image would achieve SROCC 0.978 versus 0.921 for the single best method, KonCept512. They then attempt to learn this oracle in three ways: AutoFolio with hand-crafted features, a CNN classifier directly predicting the best method, and a Siamese network predicting per-method absolute errors. All attempts fail to beat the single best method on the test set: AutoFolio collapses to always selecting KonCept512, and the deep approaches yield SROCCs of 0.871 and 0.908. The paper concludes that algorithm selection for BIQA does not improve over the single best method and conjectures that the VBM advantage is partly due to unavoidable noise in per-image quality evaluation.","tokens_in":5753,"tokens_out":4620,"duration_ms":49956,"significance":"If the negative result is accepted, this is a useful cautionary data point for the algorithm-selection community and for BIQA researchers: despite a large oracle gap, state-of-the-art selectors may fail on perceptual quality tasks because the per-image advantage of the VBM may be partly noise. The paper is honest about the failure and makes the AutoFolio search systematic (24 hours, over 500 models). It also provides a clear, reproducible setup based on the public KonIQ-10k dataset and an explicit comparison of SBM, VBM, and AS. However, the evidence base is narrow (a single dataset), the deep-learning training protocol is not shown to be sufficient, and no statistical significance tests are reported; these weaknesses currently limit the strength of the general conclusion.","major_comments":[{"comment":"The deep-learning experiments do not demonstrate that the selectors were trained to convergence. InceptionResNetV2 was fine-tuned for exactly 10 epochs with SGD at alpha=0.0001, no learning curves or convergence checks are shown, and no data augmentation is mentioned. Given that the network is large and the task is a new 8-way labeling problem, underfitting is a plausible alternative explanation for the failure of Approach 1 and Approach 2. These results therefore cannot support the paper's negative conclusion unless the training protocol is shown to suffice or the conclusion is explicitly restricted to the AutoFolio experiments.","section":"Section IV"},{"comment":"The claim that algorithm selection 'did not yield a significant improvement' is not backed by any significance test or confidence interval. The only case where AS numerically beats the single best method is the MAE comparison after excluding KonCept512 (6.665 versus 6.792), while the corresponding SROCC is worse (0.784 versus 0.805). Without paired bootstrap or permutation tests, these differences may be pure noise. Please add statistical tests or soften the 'significant' wording.","section":"Section III, Table II"},{"comment":"The central negative claim is established on a single dataset, the KonIQ-10k test set, yet the conclusion is stated generally: 'algorithm selection for BIQA does not yield an improvement over the single best method.' Because BIQA methods and instance-level selectors may behave differently on other databases, either add experiments on at least one additional dataset (e.g., LIVEC or another authentic-distortion benchmark) or explicitly restrict the claim to KonIQ-10k and note that generalization remains untested.","section":"Sections II and V"}],"minor_comments":[{"comment":"The phrase 'principle component analysis' should be 'principal component analysis'.","section":"Section III"},{"comment":"The 'noisiness' conjecture is presented as a potential explanation, but it is not quantified or tested. Please either add an experiment (e.g., measuring the variance of each IQA method under small pixel perturbations or using repeated MOS data) or clearly label the conjecture as a hypothesis for future work rather than as an observed property.","section":"Section V"},{"comment":"For Approach 1, the paper reports a classification accuracy of 29.3% on an 8-class problem, but no baseline or confusion matrix is given. Reporting the majority-class baseline and per-class accuracies would make this result interpretable.","section":"Section IV"},{"comment":"The paper does not state whether the selector scripts or feature computation code are publicly available; releasing them would strengthen reproducibility, since the underlying dataset is public.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The AutoFolio experiment is the strongest part of the paper and, in itself, provides a credible negative result for feature-based algorithm selection on KonIQ-10k. The main risk is that the broad negative claim is supported by a weak deep-learning protocol and no significance testing. I would not recommend rejection if the authors add statistical tests and either strengthen or explicitly delimit the deep-learning claims. The paper is a short workshop-style report, so the additional evidence need not be extensive, but it must be sufficient to rule out the most obvious alternative explanations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this is the first attempt to apply formal algorithm selection to blind IQA, and the negative result is real but narrower than the abstract implies. On KonIQ-10k, an oracle selector would beat KonCept512 by a large margin, but AutoFolio, after 24 hours of search over 500 models, learned to pick KonCept512 for every image, and two deep classifiers fell short. The paper's central claim—algorithm selection as tried here does not beat the single best method—is supported by the reported experiments.\n\nWhat's new: the formulation of BIQA as algorithm selection with VBM/SBM, the use of AutoFolio, a siamese error-prediction network, and a noisiness conjecture. The noisiness idea is genuinely testable: if the oracle advantage is partly noise exploitation, then denoising individual methods should shrink the gap. The paper is honest to label it as a conjecture.\n\nWhat it does well: clear setup, a sensible portfolio of methods, and the AutoFolio experiment is convincing. The oracle is used only as an upper bound, not as training signal, so no circularity. The citation pattern looks fine.\n\nSoft spots: the deep training protocol is too thin to rule out underfitting. Ten epochs of fine-tuning InceptionResNetV2, with no learning curves, convergence checks, or data augmentation, is a live alternative explanation for those two failures. That matters because the paper counts those failures as part of the negative conclusion. The AutoFolio result, however, does not depend on that protocol and is the strongest witness. Also: one dataset, no confidence intervals or significance tests, and no released code or data. The noise conjecture is speculative, but it is clearly framed as speculation.\n\nThe central negative claim holds up as reported for AutoFolio. The deep-learning leg is weaker but not load-bearing. The paper deserves peer review: a serious referee would ask for a second dataset, more training budget or convergence evidence, and a plan for testing noisiness.\n\nFor a reading group, I would bring it as a cautionary example of oracle gaps not being learnable structure. I would not cite it in my own work next year unless I were directly working on BIQA algorithm selection.","headline":"First real test of algorithm selection for blind IQA; the negative result is plausible and the AutoFolio experiment is the strong witness, but the deep-learning protocol is too undertrained to carry much weight.","tokens_in":6229,"tokens_out":2690,"would_cite":false,"duration_ms":28300,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Algorithm selection can't beat the best single image-quality method","keywords":["blind image quality assessment","algorithm selection","virtual best method","AutoFolio","deep learning","KonIQ-10k","Siamese network","noisy evaluation"],"falsifier":"Rate multiple perceptually indistinguishable versions of each KonIQ-10k test image with the same eight methods and human subjects, then recompute the oracle's SROCC and MAE against the averaged human scores. If the oracle advantage over KonCept512 mostly disappears, the noise-exploitation explanation is confirmed; if it persists, the failure of the trained selectors is unexplained and a better selector should be demonstrable.","tokens_in":5289,"feed_emoji":"🖼️","tokens_out":6153,"duration_ms":56699,"temperature":0.7,"pith_summary":"The paper asks whether a learned oracle—an algorithm selector that picks, for each input image, the best among eight blind image-quality assessment methods—can outperform the single best method on the KonIQ-10k benchmark. An ideal oracle would raise Spearman correlation from 0.921 to 0.978, but both an AutoFolio-based selector and two deep-learning classifiers failed to beat KonCept512, the best single method. The authors attribute the failure to noisy quality evaluation: perceptually equivalent images can receive different numerical scores, and the oracle's advantage may be mostly an artifact of that noise rather than learnable structure. If this is right, the practical ceiling for blind IQA may already be near the best single method, and research should shift to measuring and reducing the noisiness of IQA methods.","feed_headline":"Algorithm selection can't beat the best single image-quality method","feed_subtitle":"AutoFolio and two deep classifiers failed to beat KonCept512; the oracle edge may be noise, not signal.","key_machinery":"The central devices are the virtual best method (VBM) and the algorithm-selection pipeline built to mimic it. The VBM acts as the oracle: for each test image it picks the one of eight methods whose quality estimate is closest to the human mean opinion score, defining an upper bound that any selector would need to learn. AutoFolio searches over selector models and hyperparameters using instance features and per-method errors; the deep approach uses InceptionResNetV2 as a shared CNN base, either as a direct eight-way classifier or as a Siamese error predictor with eight output modules. The paper's explanatory mechanism is the noise model: an IQA method evaluated on perceptually indistinguishable images yields a distribution of scores, so the oracle can appear to win by choosing the method whose noise happens to land closest to the human rating, an effect no training set can teach a selector.","core_discovery":"On the KonIQ-10k test set, the paper shows that the gap between the single best method (KonCept512, SROCC 0.921, MAE 4.154) and the virtual best method—the oracle that always selects the method closest to the human mean opinion score—is large (SROCC 0.978, MAE 2.069). It then attempts to learn that oracle with AutoFolio, a state-of-the-art automatic algorithm selector, and with two deep networks: an InceptionResNetV2 classifier assigning each image to one of the eight methods, and a Siamese error-prediction network that estimates each method's absolute error. None of these surpassed KonCept512: AutoFolio degenerated to always choosing KonCept512, and the deep classifiers yielded SROCCs of 0.871 and 0.908. The paper concludes that algorithm selection for blind IQA does not improve over the single best method, and conjectures that the oracle's apparent superiority reflects noise in image-quality evaluation rather than predictable variation across images.","pith_inferences":["The same noise argument may explain why the deep classifiers performed worse than AutoFolio: if per-image labels (which method wins) are dominated by noise, training a classifier on those labels teaches the network the noise pattern, not a stable mapping. A testable extension would be to train on labels averaged over multiple perceptually equivalent renditions of each image.","A natural extension is to apply the same VBM-versus-selector comparison to full-reference IQA or to distortion-specific subsets; if the oracle gap is broader than KonIQ-10k's authentic distortions, the conclusion would generalize, and if it shrinks, the noise floor is dataset-specific.","If the noisiness conjecture is correct, then the VBM's SROCC of 0.978 is not a meaningful upper bound for algorithmic performance on this dataset; a sharper bound would come from the best achievable method under a noise-removed ground truth, which one could construct by repeated rating of visually identical images and averaging the human scores."],"forward_implications":["If no learned selector can beat KonCept512 on KonIQ-10k, then a single well-chosen BIQA method is the practical default for this benchmark, and adding a portfolio of eight methods does not help by itself.","The large VBM gap should not be read as an achievable performance target for algorithm selection; part or all of it may be due to the evaluation noise floor.","Measuring the noisiness of individual IQA methods becomes a first-class problem: methods with lower noise may be preferable even when their mean accuracy is similar.","Denoising IQA predictions—for instance by averaging over perceptually equivalent inputs—could shrink the oracle gap and make algorithm selection a more meaningful exercise.","The negative result carries a methodological lesson: before declaring that an oracle gap is learnable, one should check whether the gap survives under a noise-corrected evaluation."],"supporting_citations":[{"why":"Formalizes the algorithm selection problem that the paper applies to BIQA, defining single best method and virtual best method.","marker":"[1]"},{"why":"Provides the KonIQ-10k dataset and the eight IQA method predictions used for all experiments.","marker":"[4]"},{"why":"Supplies the 5-parameter logistic function used to calibrate predictions to the joint quality scale before selection.","marker":"[5]"},{"why":"Is the AutoFolio system whose selector-design search is tested in the first experiment.","marker":"[7]"},{"why":"Provides the InceptionResNetV2 architecture used as the base of both deep-learning selectors.","marker":"[8]"},{"why":"Provides the ImageNet pretrained weights from which the deep classifiers are fine-tuned.","marker":"[9]"}],"fun_headline_variants":["Oracle edge in image quality may be noise, not learnable","No algorithm selector beats the best single image-quality method","Why the 'best' image-quality oracle may just be noise","AI selection fails to beat the best single image-quality method"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the training protocols—24 hours of AutoFolio search over 500 models and only 10 epochs of fine-tuning for the deep networks—gave the learned selectors a fair chance to capture any learnable image-to-method structure; if these regimes underfit, the failure is an artifact of training effort, not a property of the problem.","fun_headline_variants_meta":{"raw":{"variants":["Oracle edge in image quality may be noise, not learnable","No algorithm selector beats the best single image-quality method","Why the 'best' image-quality oracle may just be noise","AI selection fails to beat the best single image-quality method"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00077,"raw_usage":{"total_tokens":3434,"prompt_tokens":995,"completion_tokens":2439,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":2371}},"tokens_in":611,"tokens_out":2439,"duration_ms":18756,"temperature":1.0,"reasoning_tokens":2371,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:30:32.070832+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rate multiple perceptually indistinguishable versions of each KonIQ-10k test image with the same eight methods and human subjects, then recompute the oracle's SROCC and MAE against the averaged human scores. If the oracle advantage over KonCept512 mostly disappears, the noise-exploitation explanation is confirmed; if it persists, the failure of the trained selectors is unexplained and a better selector should be demonstrable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Formalizes the algorithm selection problem that the paper applies to BIQA, defining single best method and virtual best method."},{"cited_title":"Oszust, ``Decision fusion for image quality assessment using an optimization approach,'' IEEE Signal Processing Letters, vol","cited_arxiv_id":null,"evidence_quote":"Provides the KonIQ-10k dataset and the eight IQA method predictions used for all experiments."},{"cited_title":"Bischl, P","cited_arxiv_id":null,"evidence_quote":"Is the AutoFolio system whose selector-design search is tested in the first experiment."},{"cited_title":"Lindauer, H","cited_arxiv_id":null,"evidence_quote":"Provides the InceptionResNetV2 architecture used as the base of both deep-learning selectors."},{"cited_title":"Szegedy, S","cited_arxiv_id":null,"evidence_quote":"Provides the ImageNet pretrained weights from which the deep classifiers are fine-tuned."}],"review_version":1}