Pith. sign in

REVIEW 3 major objections 5 minor 31 references

Uncertainty-driven Sampling for Efficient Pairwise Comparison Subjective Assessment

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that pairwise image-quality tests can be made much cheaper by predicting easy preferences with a deep network and deferring only high-uncertainty pairs to humans, while keeping the resulting quality scores nearly…

desk verdict Good idea, useful framework, but the headline result against active baselines is inflated by giving LBPS-EIC oracle labels for selected pairs. read the letter →

arxiv 2411.18372 v2 pith:UJ6PXLW6 submitted 2024-11-27 cs.MM

Shima Mohammadi , João Ascenso This is my paper · ORCID
classification cs.MM
keywords imagequalityassessmentpairwisecomparisonsubjectiveuncertaintyestimationMonteCarlodropoutBradley-Terrymodelactivesamplingexpectedinformationchange
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Subjective image-quality tests that ask humans to compare image pairs are accurate but expensive, because the number of pairs grows quadratically with the number of images. This paper claims that most pairwise preferences can be predicted by a deep model, and that only the pairs the model is least confident about need to be judged by people. The proposed pipeline estimates each image's quality as a Gaussian distribution, converts the two distributions into a probability of preference, and ranks pairs by an expected-information-change criterion that measures how much a human judgment would move the final Bradley-Terry scores. On the PieAPP and PC-IQA benchmarks, selecting just 10% of pairs gives PLCC above 0.91–0.93, rising to 0.98–0.99 at 50%. If the claim holds, benchmarking and training quality metrics could run with a small fraction of the human comparisons currently required.

What carries the argument

The argument runs on four pieces: a shared-weight ResNet34 feature extractor with residual maps and bilinear pooling that outputs a mean $\mu$ and a standard deviation $\sigma$ for each distorted image's perceived quality; a Gaussian quality-difference model in which $\Pr(A \succ B) = \Phi((\mu_A-\mu_B)/\sqrt{\sigma_A^2+\sigma_B^2})$; Monte-Carlo dropout at inference that produces many preference estimates whose variance $\sigma_m^2$ is the model uncertainty; and the expected information change (EIC) selector, which computes the KL divergence between the Bradley-Terry score posterior before and after hypothetically perturbing a pair's preference by $\max(\delta,\sigma_m^2)$. The EIC score decides which pairs are deferred to humans.

What would settle it

Compute, on a held-out set, the rank correlation between the model uncertainty $\sigma_m^2$ of each pair and the absolute error between the model's predicted preference and the human-judged preference. If high-uncertainty pairs do not show larger prediction errors than low-uncertainty pairs, the selection criterion is not measuring what the paper claims. The same experiment could be run by replacing $\sigma_m^2$ in the EIC update with random noise of the same magnitude; if PLCC stays unchanged, the reported advantage comes from the perturbation schedule rather than from genuine model uncertainty.

Watch

Extended reading notes

Core claim

The central claim is that a learning-based pairwise sampling method, LBPS-EIC, can decide, before any human trial, which image pairs are worth showing to subjects and which can be safely replaced by the model's own preference estimates, and that the resulting Bradley-Terry quality scores stay close to those of an exhaustive test. Data uncertainty, the spread of the Gaussian quality-difference distribution, and model uncertainty, the variance of preference estimates over Monte-Carlo dropout passes, are combined into an expected information change (EIC) criterion that ranks pairs by how much a human answer would shift the posterior distribution of scores. The paper reports PLCC of 0.93 at a 10% trial budget and 0.99 at 50% on PieAPP, and 0.91 at 10% and 0.98 at 50% on PC-IQA, with LBPS-EIC giving the highest PLCC among all compared methods at every budget. Because pair selection happens offline, the subjective test becomes a fixed batch of judgments rather than an iterative procedure.

Load-bearing premise

The scheme depends on the model's self-reported uncertainty being a reliable guide to where it actually errs; if the pairs it flags as uncertain are not the ones most worth re-judging, selection degenerates toward random sampling and the reported gains shrink.

Editorial extensions

If this is right

  • If the method works as reported, a subjective test at 10% of the trial budget still yields PLCC above 0.92, so benchmarking campaigns can cut their human-labour cost by roughly an order of magnitude.
  • Because the pair list is fixed before the test starts, many subjects can judge pairs in parallel without the sequential dependency that active-sampling methods impose.
  • Combining data and model uncertainty in the EIC criterion outperforms using either uncertainty alone, which suggests that the value of the method lies in the interaction between predictor uncertainty and score aggregation.
  • The low and nearly constant standard deviation of the inferred Bradley-Terry scores means the method does not just rank well; it also gives stable quality estimates across budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Nothing in the EIC criterion is specific to images, so the same offline uncertainty-driven selection could be tested on paired preferences in video quality, text quality, or recommender systems whenever a probabilistic predictor and a Bradley-Terry aggregation are available.
  • The perturbation size in the posterior update is a heuristic; a calibration step that maps $\sigma_m^2$ to observed per-distortion error could sharpen low-budget selection, a testable extension the paper does not run.
  • The method could be hybridized with active sampling: use EIC to pre-select a candidate pool offline, then let an active strategy choose among those candidates once real human responses arrive.
  • If dropout uncertainty is miscalibrated for unseen distortion types, the reported advantage would shrink; a fallback would be to sample randomly within high data-uncertainty pairs rather than trusting $\sigma_m^2$ alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an offline, uncertainty-driven sampling method for pairwise comparison subjective image quality assessment. A deep network estimates the quality distribution of each image and hence the probability of preference for each pair, together with aleatoric uncertainty (the variance of the quality difference) and epistemic uncertainty (MC-dropout variance). Three selection criteria are compared: pure data uncertainty, pure model uncertainty, and a new expected information change (EIC) criterion that combines both. The selected pairs are replaced by ground-truth preference probabilities, the remaining pairs by model predictions, and the resulting pairwise comparison matrix is aggregated with the Bradley-Terry model. Experiments on the PieAPP test set and PC-IQA dataset report that LBPS-EIC achieves the highest PLCC among compared methods, e.g., 0.93 at a 10% trial budget and 0.99 at 50% on PieAPP, and that it outperforms active sampling baselines.

Significance. The paper addresses a practically important problem: reducing the cost of pairwise comparison subjective tests while preserving the accuracy of the inferred quality scores. The proposed combination of aleatoric and epistemic uncertainty for offline pair selection is well motivated, and the EIC criterion is a reasonable adaptation of expected-information-gain active sampling. The model is trained on a disjoint split of PieAPP and evaluated on two datasets, and the code is publicly available. However, the central empirical claim of superiority over active baselines is currently supported by an unequal experimental comparison, because LBPS-EIC receives aggregated ground-truth probabilities for the pairs it selects while the anchors receive only sparse binary judgments. Until that asymmetry is fixed, the reported quantitative gains cannot be attributed to the uncertainty-driven selection criterion. With a fair comparison, the method would be a useful contribution to efficient subjective testing. I do not see a circularity problem: the preference model is evaluated on content and distortion types disjoint from its training set, and the EIC selection uses only model outputs.

major comments (3)
  1. [Section V-C, V-D, Tables I and II] The comparison between LBPS-EIC and the active anchors is not information-fair. In Section V-C, the estimated PCM for the LBPS methods is populated with model predictions, and for pairs selected for human evaluation, with the ground-truth preference probability (the aggregated human score). In Section V-D, the anchors instead receive simulated Bernoulli binary judgments on PieAPP and a randomly selected single binary judgment on PC-IQA, and their PCM entries are empirical proportions over the finite trials allocated by the sampler. Because the budget in Tables I and II is defined as a percentage of n(n-1)/2 x 15 trials, a selected pair in LBPS-EIC injects the full aggregate information of all 15 trials for that pair at once, while an anchor must spread the same budget over binary trials. The reported PLCC gains (e.g., 0.93 at 10% on PieAPP) could therefore reflect the amount of oracle information injected rather than the quality of the EIC selection criterion. I ask the authors to re-run the comparison under equal information conditions, e.g., by sampling binary judgments for the LBPS-EIC selected pairs from the same ground-truth probabilities, or by giving anchors the same aggregated probability information for their selected pairs.
  2. [Table II, Section V-E3] The text states that LBPS-EIC 'consistently exhibits the highest PLCC correlation across all trial budgets compared to other methods for both datasets,' but Table II shows Hybrid-MST with PLCC 0.99 at the 50% budget versus LBPS-EIC's 0.98 on PC-IQA. This contradicts the claimed consistency. Please correct the claim or reconcile the discrepancy.
  3. [Equation (5), Section V-E] The EIC criterion depends on the perturbation max(delta, sigma_m^2) with delta set to 0.3, and the MC-dropout inference uses a fixed 200 iterations and dropout ratio 0.2. No ablation or sensitivity analysis is provided for delta, even though the paper states only that the update 'works best.' Because Tables I and II compare LBPS-EIC against anchors, the reported advantage of EIC over the simpler LBPS-Data and LBPS-Model criteria could be sensitive to this hand-tuned value. Please add an ablation over delta and, ideally, over MC-dropout iterations, or justify the chosen value from a validation set.
minor comments (5)
  1. [Section V-D] The paper says the standard deviation over the 25 runs is not significant and is not reported; please report the standard deviations or provide a statistical test, especially for the small-budget regimes where anchor performance varies.
  2. [Tables I and II] The captions read 'LPBS-EIC' and should read 'LBPS-EIC.'
  3. [Figure 6] The x-axis label 'Number of Defer pairs (%)' should be 'Number of Deferred Pairs (%).'
  4. [Table I, PS-PC row] SROCC at 10% is 0.44 while PLCC is 0.80; this large gap should be explained or checked, since the two metrics are expected to be more consistent for a single ranking.
  5. [Section IV-C, Eq. (6)] The text says EIC 'corresponds to the KL divergence' between prior and posterior, but the equation sums two KL terms; please clarify whether EIC is the sum or an average, and state how the multivariate normal approximation of p(S|M) is parameterized.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central derivation is self-contained; the anchor-comparison asymmetry is a validity concern, not a circular reduction.

full rationale

The claimed derivation chain is not circular. The preference model is trained on the disjoint PieAPP training split and then applied to the PieAPP test set and the independent PC-IQA dataset, so the predicted preferences used to populate the unselected entries of each PCM are not derived from the test labels. The MC-dropout model uncertainty sigma_m in Eq. (5) and the EIC criterion in Eq. (6) are computed from the model's own stochastic forward passes and from the Bradley-Terry posterior over the model-initialized PCM; they do not use ground-truth preferences for the pair under consideration. The use of ground-truth probabilities for pairs selected for human evaluation is the intended measurement operation of a subjective test, not a fitted parameter renamed as a prediction. The paper does contain minor self-citations (PS-PC, ref. [15], and the evaluation methodology of ref. [31]), but these are contextual comparisons and do not carry the load of the central claim. The strongest validity concern is the unequal anchor protocol: Section V-D states that for PieAPP 'the observed judgments were simulated based on the probability of preference available for each pair of images' while for LBPS-EIC Section V-C states that 'some preferences in ˆmi may have been obtained from ground truth scores if the pair was selected for human evaluation.' This gives LBPS-EIC oracle-valued entries for selected pairs while active anchors receive finite Bernoulli samples, which may inflate the reported PLCC gap. That is an experimental fairness/correctness issue, not a circular derivation, because the final scores are not equivalent by construction to the selection criterion's inputs. Additionally, Table II at 50% on PC-IQA shows Hybrid-MST PLCC 0.99 versus LBPS-EIC 0.98, which conflicts with the text's claim that LBPS-EIC 'consistently exhibits the highest PLCC correlation across all trial budgets,' but this is an internal consistency issue, not circularity. No self-citation chain or definitional identity forces the reported result.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim depends on standard psychophysical and Bayesian approximation assumptions, plus two heuristic choices: the Gaussian posterior approximation for scores and the perturbation rule in Eq. (5) with hand-tuned delta. No new entities are introduced, but the hand-tuned delta and the normality shortcut add fragility.

free parameters (3)
  • delta in EIC update (Eq. 5) = 0.3
    Hand-selected perturbation amount for the expected information change criterion. The paper notes the update 'was found to work best' without ablation or sensitivity analysis, and the pair selection results may depend on it.
  • MC-dropout inference iterations = 200
    Number of stochastic forward passes to estimate model uncertainty. No sensitivity analysis is reported, and the variance estimate depends on this choice.
  • dropout ratio = 0.2
    Used both during training and inference for MC-dropout. It controls the strength of the epistemic uncertainty estimate and is not tuned in the paper.
assumptions (5)
  • domain assumption The perceptual quality of an image is Gaussian distributed, and the qualities of the two images in a pair are independent.
    Section III-A, Eq. (1). This is Thurstone's model from [3] and underlies the preference probability formula (2).
  • domain assumption MC-dropout with Bernoulli masks approximates the Bayesian posterior over network weights.
    Section III-B, Eq. (3) from [23], [24]. The model uncertainty sigma_m and mean prediction mu_m rely on this approximation.
  • ad hoc to paper The posterior distribution of Bradley-Terry scores p(S|M) is approximately multivariate normal.
    Section IV-C, Eq. (6). The EIC criterion uses KL divergence between multivariate normals, but no justification is given for the normality approximation.
  • ad hoc to paper The perturbation of preference by +/- max(delta, sigma_m) in Eq. (5) spans the range of plausible preference changes for a pair.
    Section IV-C. The update rule is heuristic and stated to work best; no derivation or validation is provided.
  • domain assumption The PieAPP and PC-IQA test sets are representative of the target population for subjective image quality assessment.
    Section V-B. The model is trained on PieAPP training contents and evaluated on disjoint test contents and PC-IQA; the assumption is that these cover realistic distortions and preferences.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncertainty-driven Sampling for Efficient Pairwise Comparison Subjective Assessment." pith.science (2026). https://pith.science/paper/UJ6PXLW6

@misc{pith2026241118372,
  author       = {Pith},
  title        = {Pith review of: Uncertainty-driven Sampling for Efficient Pairwise Comparison Subjective Assessment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UJ6PXLW6}},
  note         = {Machine review of arXiv:2411.18372}
}
read the original abstract

Assessing image quality is crucial in image processing tasks such as compression, super-resolution, and denoising. While subjective assessments involving human evaluators provide the most accurate quality scores, they are impractical for large-scale or continuous evaluations due to their high cost and time requirements. Pairwise comparison subjective assessment tests, which rank image pairs instead of assigning scores, offer more reliability and accuracy but require numerous comparisons, leading to high costs. Although objective quality metrics are more efficient, they lack the precision of subjective tests, which are essential for benchmarking and training learning-based quality metrics. This paper proposes an uncertainty-based sampling method to optimize the pairwise comparison subjective assessment process. By utilizing deep learning models to estimate human preferences and identify pairs that need human labeling, the approach reduces the number of required comparisons while maintaining high accuracy. The key contributions include modeling uncertainty for accurate preference predictions and for pairwise sampling. The experimental results demonstrate superior performance of the proposed approach compared to traditional active sampling methods. Software is publicly available at: shimamohammadi/LBPS-EIC

Figures

Figures reproduced from arXiv: 2411.18372 by the authors.

Figure 1
Figure 1. provides a visual representation for quality dif￾ference distribution, where image A exhibits higher quality than image B. Notably, the positive area under the quality difference distribution Q(A)−Q(B) exceeds the negative area, indicating higher preference for selecting A over B. 2 0 2 4 Density A B PDF of Q(A)-Q(B) [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Overview of the pairwise sampling procedure. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Proposed architecture of the deep learning framework ( [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Predicted preference versus ground truth preference using a scatter plot [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Model uncertainty versus predicted preferences. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Evaluation of the pairwise selection method [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Performance evaluation reporting standard deviation over the entire [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Qualitative analysis for some selected pairs of a reference content. Red box: The first pair [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Qualitative analysis for some selected pairs of a reference content. Red box: The first pair [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 27 canonical work pages

  1. [1]

    Methods for the subjective assessment of video quality, audio quality and audiovisual quality of internet video and distribution quality television in any environment,

    R. I.-T. P.913, “Methods for the subjective assessment of video quality, audio quality and audiovisual quality of internet video and distribution quality television in any environment,” 2016

    work page 2016

  2. [2]

    Recommendation 500-19: Methodology for the subjective assessment of the quality of television pictures,

    ITU-R Question 102-3/6, “Recommendation 500-19: Methodology for the subjective assessment of the quality of television pictures,” ITU-R Recommendation BT, 2019. 13

    work page 2019

  3. [3]

    A law of comparative judgment,

    L. Thurstone, “A law of comparative judgment,” Psychology Review , vol. 34, pp. 273–86, April 1994

    work page 1994

  4. [4]

    Rank analysis of incomplete block designs: I. the method of paired comparisons,

    R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. the method of paired comparisons,” Biometrika, vol. 39, no. 3/4, pp. 324–345, 1952

    1952

  5. [5]

    Comparing subjective video quality testing methodologies,

    M. H. Pinson and S. Wolf, “Comparing subjective video quality testing methodologies,” in Visual Communications and Image Processing , vol. 5150, 2003, pp. 573–582

    work page 2003

  6. [6]

    Performance comparisons of subjective quality assessment methods for mobile video,

    T. Tominaga, T. Hayashi, J. Okamoto, and A. Takahashi, “Performance comparisons of subjective quality assessment methods for mobile video,” in International Workshop on Quality of Multimedia Experience , Trond- heim, Norway, June 2010, pp. 82–87

    work page 2010

  7. [7]

    Comparison of four subjective methods for image quality assessment,

    R. K. Mantiuk, A. Tomaszewska, and R. Mantiuk, “Comparison of four subjective methods for image quality assessment,” in Computer graphics forum, vol. 31, no. 8, 2012, pp. 2478–2491

    work page 2012

  8. [8]

    HodgeRank on random graphs for subjective video quality assessment,

    Q. Xu, Q. Huang, T. Jiang, B. Yan, W. Lin, and Y . Yao, “HodgeRank on random graphs for subjective video quality assessment,” IEEE Transactions on Multimedia , vol. 14, no. 3, pp. 844–857, 2012

    work page 2012

Show all 31 references
  1. [9]

    Image database TID2013: Peculiarities, results and perspectives,

    N. Ponomarenko, L. Jin, O. Ieremeiev, V . Lukin, K. Egiazarian, J. Astola, B. V ozel, K. Chehdi, M. Carli, F. Battisti et al. , “Image database TID2013: Peculiarities, results and perspectives,” Signal processing: Image communication, vol. 30, pp. 57–77, 2015

    2015

  2. [10]

    Active sampling for pairwise comparisons via approximate message passing and information gain maximization,

    A. Mikhailiuk, C. Wilmot, M. Perez-Ortiz, D. Yue, and R. K. Mantiuk, “Active sampling for pairwise comparisons via approximate message passing and information gain maximization,” in International Confer- ence on Pattern Recognition , Milan, Italy, January 2021

    2021

  3. [11]

    Hodgerank with information maximization for crowdsourced pairwise ranking aggrega- tion,

    Q. Xu, J. Xiong, X. Chen, Q. Huang, and Y . Yao, “Hodgerank with information maximization for crowdsourced pairwise ranking aggrega- tion,” in AAAI Conference on Artificial Intelligence , vol. 32, Louisiana, USA, 2018

    2018

  4. [12]

    Pairwise ranking aggregation in a crowdsourced setting,

    X. Chen, P. N. Bennett, K. Collins-Thompson, and E. Horvitz, “Pairwise ranking aggregation in a crowdsourced setting,” in ACM international conference on Web search and data mining, Rome, Italy, February 2013, pp. 193–202

    2013

  5. [13]

    Hybrid-MST: A hybrid active sampling strategy for pairwise preference aggregation,

    J. Li, R. Mantiuk, J. Wang, S. Ling, and P. Le Callet, “Hybrid-MST: A hybrid active sampling strategy for pairwise preference aggregation,” in Advances in Neural Information Processing Systems , vol. 31, Montr ´eal, Canada, 2018

    2018

  6. [14]

    Active sampling exploiting reliable informativeness for subjective image quality assessment based on pair- wise comparison,

    Z. Fan, T. Jiang, and T. Huang, “Active sampling exploiting reliable informativeness for subjective image quality assessment based on pair- wise comparison,” IEEE Transactions on Multimedia , vol. 19, no. 12, pp. 2720–2735, 2017

    2017

  7. [15]

    Predictive sampling for efficient pairwise subjective image quality assessment,

    S. Mohammadi and J. a. Ascenso, “Predictive sampling for efficient pairwise subjective image quality assessment,” in Proceedings of the 31st ACM International Conference on Multimedia , ser. MM ’23, New York, NY , USA, October 2023, p. 6676–6684

    2023

  8. [16]

    Improving relevant subjective testing for validation: Comparing machine learning algorithms for finding similarities in vqa datasets using objective measures,

    A. Aldahdooh, E. Masala, G. Van Wallendael, P. Lambert, and M. Barkowsky, “Improving relevant subjective testing for validation: Comparing machine learning algorithms for finding similarities in vqa datasets using objective measures,” Signal Processing: Image Communication, vo...

    2019

  9. [17]

    Uncertainty-aware blind image quality assessment in the laboratory and wild,

    W. Zhang, K. Ma, G. Zhai, and X. Yang, “Uncertainty-aware blind image quality assessment in the laboratory and wild,” IEEE Transactions on Image Processing, vol. 30, pp. 3474–3486, March 2021

    2021

  10. [18]

    Image quality assessment: Integrating model- centric and data-centric approaches,

    P. Cao, D. Li, and K. Ma, “Image quality assessment: Integrating model- centric and data-centric approaches,” in Conference on Parsimony and Learning, January 2024, pp. 529–541

    2024

  11. [19]

    dipIQ: Blind image quality assessment by learning-to-rank discriminable image pairs,

    K. Ma, W. Liu, T. Liu, Z. Wang, and D. Tao, “dipIQ: Blind image quality assessment by learning-to-rank discriminable image pairs,” IEEE Transactions on Imgage Processing , vol. 26, no. 8, p. 3951–3964, Auguest 2017

    2017

  12. [20]

    Exploiting unlabeled data in cnns by self-supervised learning to rank,

    X. Liu, J. v. d. Weijer, and A. D. Bagdanov, “Exploiting unlabeled data in cnns by self-supervised learning to rank,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 8, pp. 1862–1878, 2019

    2019

  13. [21]

    A review of uncertainty quan- tification in deep learning: Techniques, applications and challenges,

    M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya, V . Makarenkov, and S. Nahavandi, “A review of uncertainty quan- tification in deep learning: Techniques, applications and challenges,” Information Fusio...

    2021

  14. [22]

    How to analyze paired comparison data,

    K. Tsukida and M. Gupta, “How to analyze paired comparison data,” UWEE Technical Report 206, 2011 , January 2011

    2011

  15. [23]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning,

    Y . Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in Proceedings of The 33rd International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, M. F. Balcan and K. Q. Weinberger, Eds., ...

    2016

  16. [24]

    A survey of uncertainty in deep neural networks,

    J. Gawlikowski, C. Tassi, M. Ali, J. Lee, M. Humt, J. Feng, A. Kruspe, R. Triebel, P. Jung, R. Roscher, M. Shahzad, W. Yang, R. Bamler, and X. Zhu, “A survey of uncertainty in deep neural networks,” Artificial Intelligence Review, vol. 56, pp. 1–77, July 2023

    2023

  17. [25]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV , USA, June 2016, pp. 770–778

    2016

  18. [26]

    Bilinear CNN models for fine-grained visual recognition,

    T.-Y . Lin, A. RoyChowdhury, and S. Maji, “Bilinear CNN models for fine-grained visual recognition,” in IEEE International Conference on Computer Vision (ICCV) , 2015, pp. 1449–1457

    2015

  19. [27]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition , 2009, pp. 248–255

    2009

  20. [28]

    Frank: a ranking method with fidelity loss,

    M.-F. Tsai, T.-Y . Liu, T. Qin, H.-H. Chen, and W.-Y . Ma, “Frank: a ranking method with fidelity loss,” in 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ser. SIGIR ’07. New York, NY , USA: Association for Computing Machin...

    2007

  21. [29]

    PieAPP: Perceptual image- error assessment through pairwise preference,

    E. Prashnani, H. Cai, Y . Mostofi, and P. Sen, “PieAPP: Perceptual image- error assessment through pairwise preference,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , UT, USA, June 2018

    2018

  22. [30]

    Online crowdsourcing subjective image quality assessment,

    Q. Xu, Q. Huang, and Y . Yao, “Online crowdsourcing subjective image quality assessment,” in ACM International Conference on Multimedia , ser. MM ’12, October 2012, p. 359–368

    2012

  23. [31]

    Evaluation of sampling algorithms for a pairwise subjective assessment methodology,

    S. Mohammadi and J. Ascenso, “Evaluation of sampling algorithms for a pairwise subjective assessment methodology,” in IEEE International Symposium on Multimedia (ISM) , December 2022, pp. 288–292. Shima Mohammadi (Graduate Student Member, IEEE) received the M.Sc. degree in Ele...

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.