{"id":"2cd92b81-ca31-4467-9ed8-4d9b5e0b8788","arxiv_id":"2411.18372","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An uncertainty-based sampling method selects a small subset of image pairs for human evaluation and uses a deep model for the rest, preserving accuracy with fewer comparisons.","lead":"This paper trains a deep neural network to predict which of two images a human would prefer, and uses the network's uncertainty to decide which image pairs still need human judgment. The result is a cheaper, shorter subjective image quality test that keeps most of the accuracy of a full test.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LBPS-EIC is evaluated with ground-truth preference probabilities for selected pairs, while active baselines get only finite binary samples; the fairness of the comparison is the load-bearing issue.","rationale":"The paper's claimed contribution is that uncertainty-driven offline sampling reduces the number of human comparisons while preserving accuracy. The evidence for this is entirely the PLCC/SROCC/RMSE comparison in Tables I and II. The load-bearing condition is that all methods receive the same amount and type of label information under the stated trial budget. That condition fails: Section V-C explicitly uses ground-truth preference probabilities for LBPS-EIC's selected pairs, while Section V-D uses Bernoulli samples or single random judgments for the active baselines. The ground-truth probability is the aggregate of many human judgments; entering it directly is equivalent to using a label oracle, not a fixed number of trials. This gives LBPS-EIC an information advantage that can explain the large PLCC gaps, making the central claim unsupported. The reader's concern about MC-dropout calibration is secondary; even if sigma_m is a perfect error proxy, the reported comparisons do not establish the method's advantage because of the asymmetric protocol. In addition, the text's assertion of 'highest PLCC at all budgets' is contradicted by Table II at 50% on PC-IQA, where Hybrid-MST reaches 0.99. A fair re-evaluation with matched binary judgments or matched aggregated probabilities is required before the claim can be accepted.","tokens_in":19230,"tokens_out":10727,"duration_ms":85943,"concrete_test":"Re-run the state-of-the-art comparison with a symmetric label protocol: for every pair selected by LBPS-EIC, draw exactly B binary judgments from the ground-truth preference (e.g., B=15), enter the empirical proportion into the PCM, and keep the budget at X% of total trials. If the LBPS-EIC PLCC at 10% falls by more than ~0.05 or no longer exceeds ASAP/Hybrid-MST, the claimed advantage is an artifact of oracle label access. Equivalently, give the anchors access to the same aggregated ground-truth probabilities for the pairs they select; if the gap closes, the comparison was unfair.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim rests on Table I/II comparisons where LBPS-EIC and the active anchors are not given comparable label information. In Section V-C, the estimated PCM for LBPS methods is populated with model predictions and, for pairs selected for human evaluation, with 'ground truth scores' (the true preference probabilities). In Section V-D, the anchors instead receive simulated Bernoulli binary judgments (PieAPP) or a randomly selected single binary judgment (PC-IQA), and their PCM entries are empirical proportions from the finite trials allocated by the sampler. The trial budget is defined as a percentage of n(n-1)/2 x 15 trials, but the ground-truth probability for a selected pair encodes the full aggregated human preference data for that pair, which is at least as informative as all trials in the budget for that pair. Thus LBPS-EIC effectively receives oracle labels for the pairs it selects, while active methods must infer those probabilities from sparse noisy samples. The reported PLCC gains (0.93 at 10%, 0.99 at 50% on PieAPP) may simply reflect the amount of ground-truth information injected into the PCM, not the quality of the uncertainty criterion. A second, independent inconsistency: Table II at 50% on PC-IQA shows Hybrid-MST PLCC 0.99 vs LBPS-EIC 0.98, contradicting the text's claim of highest PLCC at every budget.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an offline, uncertainty-driven sampling method for pairwise comparison subjective image quality assessment. A deep network estimates the quality distribution of each image and hence the probability of preference for each pair, together with aleatoric uncertainty (the variance of the quality difference) and epistemic uncertainty (MC-dropout variance). Three selection criteria are compared: pure data uncertainty, pure model uncertainty, and a new expected information change (EIC) criterion that combines both. The selected pairs are replaced by ground-truth preference probabilities, the remaining pairs by model predictions, and the resulting pairwise comparison matrix is aggregated with the Bradley-Terry model. Experiments on the PieAPP test set and PC-IQA dataset report that LBPS-EIC achieves the highest PLCC among compared methods, e.g., 0.93 at a 10% trial budget and 0.99 at 50% on PieAPP, and that it outperforms active sampling baselines.","tokens_in":19542,"tokens_out":4420,"duration_ms":38400,"significance":"The paper addresses a practically important problem: reducing the cost of pairwise comparison subjective tests while preserving the accuracy of the inferred quality scores. The proposed combination of aleatoric and epistemic uncertainty for offline pair selection is well motivated, and the EIC criterion is a reasonable adaptation of expected-information-gain active sampling. The model is trained on a disjoint split of PieAPP and evaluated on two datasets, and the code is publicly available. However, the central empirical claim of superiority over active baselines is currently supported by an unequal experimental comparison, because LBPS-EIC receives aggregated ground-truth probabilities for the pairs it selects while the anchors receive only sparse binary judgments. Until that asymmetry is fixed, the reported quantitative gains cannot be attributed to the uncertainty-driven selection criterion. With a fair comparison, the method would be a useful contribution to efficient subjective testing. I do not see a circularity problem: the preference model is evaluated on content and distortion types disjoint from its training set, and the EIC selection uses only model outputs.","major_comments":[{"comment":"The comparison between LBPS-EIC and the active anchors is not information-fair. In Section V-C, the estimated PCM for the LBPS methods is populated with model predictions, and for pairs selected for human evaluation, with the ground-truth preference probability (the aggregated human score). In Section V-D, the anchors instead receive simulated Bernoulli binary judgments on PieAPP and a randomly selected single binary judgment on PC-IQA, and their PCM entries are empirical proportions over the finite trials allocated by the sampler. Because the budget in Tables I and II is defined as a percentage of n(n-1)/2 x 15 trials, a selected pair in LBPS-EIC injects the full aggregate information of all 15 trials for that pair at once, while an anchor must spread the same budget over binary trials. The reported PLCC gains (e.g., 0.93 at 10% on PieAPP) could therefore reflect the amount of oracle information injected rather than the quality of the EIC selection criterion. I ask the authors to re-run the comparison under equal information conditions, e.g., by sampling binary judgments for the LBPS-EIC selected pairs from the same ground-truth probabilities, or by giving anchors the same aggregated probability information for their selected pairs.","section":"Section V-C, V-D, Tables I and II"},{"comment":"The text states that LBPS-EIC 'consistently exhibits the highest PLCC correlation across all trial budgets compared to other methods for both datasets,' but Table II shows Hybrid-MST with PLCC 0.99 at the 50% budget versus LBPS-EIC's 0.98 on PC-IQA. This contradicts the claimed consistency. Please correct the claim or reconcile the discrepancy.","section":"Table II, Section V-E3"},{"comment":"The EIC criterion depends on the perturbation max(delta, sigma_m^2) with delta set to 0.3, and the MC-dropout inference uses a fixed 200 iterations and dropout ratio 0.2. No ablation or sensitivity analysis is provided for delta, even though the paper states only that the update 'works best.' Because Tables I and II compare LBPS-EIC against anchors, the reported advantage of EIC over the simpler LBPS-Data and LBPS-Model criteria could be sensitive to this hand-tuned value. Please add an ablation over delta and, ideally, over MC-dropout iterations, or justify the chosen value from a validation set.","section":"Equation (5), Section V-E"}],"minor_comments":[{"comment":"The paper says the standard deviation over the 25 runs is not significant and is not reported; please report the standard deviations or provide a statistical test, especially for the small-budget regimes where anchor performance varies.","section":"Section V-D"},{"comment":"The captions read 'LPBS-EIC' and should read 'LBPS-EIC.'","section":"Tables I and II"},{"comment":"The x-axis label 'Number of Defer pairs (%)' should be 'Number of Deferred Pairs (%).'","section":"Figure 6"},{"comment":"SROCC at 10% is 0.44 while PLCC is 0.80; this large gap should be explained or checked, since the two metrics are expected to be more consistent for a single ranking.","section":"Table I, PS-PC row"},{"comment":"The text says EIC 'corresponds to the KL divergence' between prior and posterior, but the equation sums two KL terms; please clarify whether EIC is the sum or an average, and state how the multivariate normal approximation of p(S|M) is parameterized.","section":"Section IV-C, Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The core issue is the oracle-information asymmetry in Tables I and II: LBPS-EIC receives aggregated ground-truth probabilities for its selected pairs while the active anchors receive sparse binary judgments. This is a load-bearing flaw in the comparison, not a cosmetic issue. If the authors fix it with a fair information-equivalent comparison and correct the Table II inconsistency, the paper could be suitable for publication. I do not see a circularity problem; the training split is appropriate, and the self-citation of PS-PC is acceptable as prior work. The topic fits the journal's scope well."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea is good, the evaluation as written overstates the case. The paper proposes an offline, uncertainty-driven sampler (LBPS-EIC) that uses MC-dropout epistemic uncertainty and aleatoric uncertainty to decide which pairs need human labels; it's a sensible, novel successor to their PS-PC. The network is trained on PieAPP, tested on held-out distortion types and on PC-IQA, and code is public. That is real evidence and the right kind of contribution.\n\nSoft spots: the headline comparison against active baselines is not apples-to-apples. In Section V-C, LBPS-EIC fills the PCM for selected pairs with the ground-truth preference probability. The active anchors, by contrast, get simulated Bernoulli judgments (PieAPP) or a single random judgment per selected trial (PC-IQA), and their PCM entries are empirical proportions from those finite draws. So LBPS-EIC gets the population value for every pair it selects, while the anchors get noisy samples. Within the trial budget, that is a substantial advantage that has nothing to do with the uncertainty criterion. The reported gains at low budgets (0.93 vs 0.8 at 10%) may mostly reflect this oracle-label effect. The fix is straightforward: for selected pairs, simulate the 15 binary votes from the ground-truth probability and use the empirical proportion as the human estimate, and report variance over runs. Without that, the main claim is not supported.\n\nAlso, Table II contradicts the text: at 50% on PC-IQA, Hybrid-MST PLCC is 0.99 vs LBPS-EIC 0.98, yet the text says LBPS-EIC has the highest PLCC at every budget. That is a factual inconsistency that needs correction.\n\nMinor: no error bars for LBPS-EIC, delta is hand-tuned (0.3, no ablation), MC-dropout iterations fixed at 200. These are minor compared to the oracle-label issue.\n\nVerdict: the contribution is worth taking seriously. A serious referee should engage, with the evaluation rework as the central request. The paper deserves a major revision, not a desk reject.","headline":"Good idea, useful framework, but the headline result against active baselines is inflated by giving LBPS-EIC oracle labels for selected pairs.","tokens_in":20023,"tokens_out":2937,"would_cite":false,"duration_ms":27376,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that pairwise image-quality tests can be made much cheaper by predicting easy preferences with a deep network and deferring only high-uncertainty pairs to humans, while keeping the resulting quality scores nearly…","keywords":["image quality assessment","pairwise comparison","subjective assessment","uncertainty estimation","Monte Carlo dropout","Bradley-Terry model","active sampling","expected information change"],"falsifier":"Compute, on a held-out set, the rank correlation between the model uncertainty $\\sigma_m^2$ of each pair and the absolute error between the model's predicted preference and the human-judged preference. If high-uncertainty pairs do not show larger prediction errors than low-uncertainty pairs, the selection criterion is not measuring what the paper claims. The same experiment could be run by replacing $\\sigma_m^2$ in the EIC update with random noise of the same magnitude; if PLCC stays unchanged, the reported advantage comes from the perturbation schedule rather than from genuine model uncertainty.","tokens_in":19015,"feed_emoji":"🖼️","tokens_out":8495,"duration_ms":70096,"temperature":0.7,"pith_summary":"Subjective image-quality tests that ask humans to compare image pairs are accurate but expensive, because the number of pairs grows quadratically with the number of images. This paper claims that most pairwise preferences can be predicted by a deep model, and that only the pairs the model is least confident about need to be judged by people. The proposed pipeline estimates each image's quality as a Gaussian distribution, converts the two distributions into a probability of preference, and ranks pairs by an expected-information-change criterion that measures how much a human judgment would move the final Bradley-Terry scores. On the PieAPP and PC-IQA benchmarks, selecting just 10% of pairs gives PLCC above 0.91–0.93, rising to 0.98–0.99 at 50%. If the claim holds, benchmarking and training quality metrics could run with a small fraction of the human comparisons currently required.","feed_headline":"Uncertainty sampling hits 0.93 accuracy at 10% of trials","feed_subtitle":"Only the hardest image pairs reach human testers; scores stay accurate with 10 percent of trials.","key_machinery":"The argument runs on four pieces: a shared-weight ResNet34 feature extractor with residual maps and bilinear pooling that outputs a mean $\\mu$ and a standard deviation $\\sigma$ for each distorted image's perceived quality; a Gaussian quality-difference model in which $\\Pr(A \\succ B) = \\Phi((\\mu_A-\\mu_B)/\\sqrt{\\sigma_A^2+\\sigma_B^2})$; Monte-Carlo dropout at inference that produces many preference estimates whose variance $\\sigma_m^2$ is the model uncertainty; and the expected information change (EIC) selector, which computes the KL divergence between the Bradley-Terry score posterior before and after hypothetically perturbing a pair's preference by $\\max(\\delta,\\sigma_m^2)$. The EIC score decides which pairs are deferred to humans.","core_discovery":"The central claim is that a learning-based pairwise sampling method, LBPS-EIC, can decide, before any human trial, which image pairs are worth showing to subjects and which can be safely replaced by the model's own preference estimates, and that the resulting Bradley-Terry quality scores stay close to those of an exhaustive test. Data uncertainty, the spread of the Gaussian quality-difference distribution, and model uncertainty, the variance of preference estimates over Monte-Carlo dropout passes, are combined into an expected information change (EIC) criterion that ranks pairs by how much a human answer would shift the posterior distribution of scores. The paper reports PLCC of 0.93 at a 10% trial budget and 0.99 at 50% on PieAPP, and 0.91 at 10% and 0.98 at 50% on PC-IQA, with LBPS-EIC giving the highest PLCC among all compared methods at every budget. Because pair selection happens offline, the subjective test becomes a fixed batch of judgments rather than an iterative procedure.","pith_inferences":["Nothing in the EIC criterion is specific to images, so the same offline uncertainty-driven selection could be tested on paired preferences in video quality, text quality, or recommender systems whenever a probabilistic predictor and a Bradley-Terry aggregation are available.","The perturbation size in the posterior update is a heuristic; a calibration step that maps $\\sigma_m^2$ to observed per-distortion error could sharpen low-budget selection, a testable extension the paper does not run.","The method could be hybridized with active sampling: use EIC to pre-select a candidate pool offline, then let an active strategy choose among those candidates once real human responses arrive.","If dropout uncertainty is miscalibrated for unseen distortion types, the reported advantage would shrink; a fallback would be to sample randomly within high data-uncertainty pairs rather than trusting $\\sigma_m^2$ alone."],"forward_implications":["If the method works as reported, a subjective test at 10% of the trial budget still yields PLCC above 0.92, so benchmarking campaigns can cut their human-labour cost by roughly an order of magnitude.","Because the pair list is fixed before the test starts, many subjects can judge pairs in parallel without the sequential dependency that active-sampling methods impose.","Combining data and model uncertainty in the EIC criterion outperforms using either uncertainty alone, which suggests that the value of the method lies in the interaction between predictor uncertainty and score aggregation.","The low and nearly constant standard deviation of the inferred Bradley-Terry scores means the method does not just rank well; it also gives stable quality estimates across budgets."],"supporting_citations":[{"why":"Supplies the Gaussian law of comparative judgment that turns mean quality difference and variance into a preference probability.","marker":"[3]"},{"why":"Supplies the Bradley-Terry aggregation model used to convert the comparison matrix into final quality scores.","marker":"[4]"},{"why":"Supplies the information-gain posterior-update machinery from active sampling that the EIC criterion is built on.","marker":"[10]"},{"why":"Supplies the hybrid active-sampling utility function and score posterior used as the basis for the EIC formulation.","marker":"[13]"},{"why":"Supplies the uncertainty-aware pairwise preference formulation for image quality assessment that motivates the data uncertainty model.","marker":"[17]"},{"why":"Supplies Monte-Carlo dropout as the estimator of model uncertainty used in sampling.","marker":"[23]"},{"why":"Supplies the ResNet34 backbone used for feature extraction in the quality-distribution network.","marker":"[25]"},{"why":"Supplies the fidelity loss used to train the preference prediction network.","marker":"[28]"},{"why":"Supplies the PieAPP dataset and its crowd-sourced pairwise preferences for training and testing.","marker":"[29]"},{"why":"Supplies the PC-IQA dataset used as a second evaluation benchmark.","marker":"[30]"}],"fun_headline_variants":["0.93 accuracy from 10% of image pair trials","AI finds hardest pairs: 90% fewer human tests","Cut subjective image tests to 10% of trials","Uncertainty sampling beats human budget by 10x","Same quality scores with 10% of the comparisons"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scheme depends on the model's self-reported uncertainty being a reliable guide to where it actually errs; if the pairs it flags as uncertain are not the ones most worth re-judging, selection degenerates toward random sampling and the reported gains shrink.","fun_headline_variants_meta":{"raw":{"variants":["0.93 accuracy from 10% of image pair trials","AI finds hardest pairs: 90% fewer human tests","Cut subjective image tests to 10% of trials","Uncertainty sampling beats human budget by 10x","Same quality scores with 10% of the comparisons"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1633,"prompt_tokens":939,"completion_tokens":694,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":614}},"tokens_in":555,"tokens_out":694,"duration_ms":6480,"temperature":1.0,"reasoning_tokens":614,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:16:13.872455+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, on a held-out set, the rank correlation between the model uncertainty $\\sigma_m^2$ of each pair and the absolute error between the model's predicted preference and the human-judged preference. If high-uncertainty pairs do not show larger prediction errors than low-uncertainty pairs, the selection criterion is not measuring what the paper claims. The same experiment could be run by replacing $\\sigma_m^2$ in the EIC update with random noise of the same magnitude; if PLCC stays unchanged, the reported advantage comes from the perturbation schedule rather than from genuine model uncertainty.","supporting_citations":[{"cited_title":"A law of comparative judgment,","cited_arxiv_id":null,"evidence_quote":"Supplies the Gaussian law of comparative judgment that turns mean quality difference and variance into a preference probability."},{"cited_title":"Active sampling for pairwise comparisons via approximate message passing and information gain maximization,","cited_arxiv_id":null,"evidence_quote":"Supplies the information-gain posterior-update machinery from active sampling that the EIC criterion is built on."},{"cited_title":"Hybrid-MST: A hybrid active sampling strategy for pairwise preference aggregation,","cited_arxiv_id":null,"evidence_quote":"Supplies the hybrid active-sampling utility function and score posterior used as the basis for the EIC formulation."},{"cited_title":"Uncertainty-aware blind image quality assessment in the laboratory and wild,","cited_arxiv_id":null,"evidence_quote":"Supplies the uncertainty-aware pairwise preference formulation for image quality assessment that motivates the data uncertainty model."},{"cited_title":"Dropout as a bayesian approximation: Representing model uncertainty in deep learning,","cited_arxiv_id":null,"evidence_quote":"Supplies Monte-Carlo dropout as the estimator of model uncertainty used in sampling."},{"cited_title":"Deep residual learning for image recognition,","cited_arxiv_id":null,"evidence_quote":"Supplies the ResNet34 backbone used for feature extraction in the quality-distribution network."},{"cited_title":"PieAPP: Perceptual image- error assessment through pairwise preference,","cited_arxiv_id":null,"evidence_quote":"Supplies the PieAPP dataset and its crowd-sourced pairwise preferences for training and testing."},{"cited_title":"Online crowdsourcing subjective image quality assessment,","cited_arxiv_id":null,"evidence_quote":"Supplies the PC-IQA dataset used as a second evaluation benchmark."}],"review_version":1}