Pith. sign in

REVIEW 4 major objections 7 minor 42 references

Uncertainty Estimation by Human Perception versus Neural Models

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Neural-network uncertainty estimates show almost no correlation with human-perceived uncertainty.

desk verdict The cross-dataset comparison is genuinely new, but the coarse human-uncertainty target (4-6 labels per image) caps the correlations, so the 'no correlation' conclusion is not yet established. read the letter →

arxiv 2506.15850 v2 pith:SWTY73GO submitted 2025-06-18 cs.LG cs.AI

Pedro Mendes , Paolo Romano , David Garlan This is my paper
classification cs.LGcs.AI
keywords uncertaintyestimationhumanperceptualpredictiveentropymodelcalibrationsoftlabelshuman-AIalignmentimageclassificationPearsoncorrelation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish a negative result: the uncertainty scores neural networks attach to images do not reflect the uncertainty humans report when classifying those same images, and this matters because human-AI systems route uncertain predictions to human reviewers. Across three vision benchmarks with human annotations, the authors compute predictive entropy for both human label distributions and model outputs and measure the Pearson correlation between them. For six uncertainty-estimation methods, correlations peak at 0.24 on CIFAR10-H and 0.35 on ImageNet-16H and are effectively zero on CIFAR-N, leading the authors to conclude that model-derived uncertainty estimates are uncorrelated with human perceptual uncertainty. The paper also reports that training with human soft labels raises the correlations to 0.34 and 0.47 on the two larger-annotation benchmarks, suggesting human judgments carry information that standard losses leave unused.

What carries the argument

The central object is predictive entropy, $\text{PE}(p)=-\sum_i p_i \log p_i$, applied symmetrically to the averaged human label distribution and the model's predicted class distribution for the same image. The human-side entropy is the paper's operationalization of human perceptual uncertainty, and Pearson's correlation between the two entropy series is the measuring stick. The same metric on both sides is meant to ensure that any divergence reflects the distributions themselves rather than the estimator. Task-complexity subgroups are defined by whether the human and model predictions agree with the true class, and correlations are recomputed within those subgroups.

What would settle it

Recompute the correlations using per-annotator human uncertainty, defined as the mean predictive entropy of individual annotators for each image, or as the variance of the human label distribution, and check whether the Pearson coefficients rise above 0.5 on CIFAR10-H and ImageNet-16H. If they do, the paper's negative conclusion is an artifact of averaging labels before computing entropy rather than a property of human and model uncertainty.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that predictive entropy from neural networks and predictive entropy from human annotations essentially do not track each other. Using averaged human soft-label entropy as the reference for human-perceived uncertainty, ResNet18 on CIFAR10-H/CIFAR-N and ResNet50 on ImageNet-16H all give Pearson correlations of at most 0.24, 0.005, and 0.35 respectively across CE with MC Dropout, EUAT, CALS, deep ensembles, isotonic regression, and DEUP. The p-values are below 0.05 on CIFAR10-H and ImageNet-16H but not on CIFAR-N, so the authors read the relationship as statistically significant but weak at best and absent on CIFAR-N. Splitting by task complexity does not reveal any consistent pattern. Mixing a subset of human soft labels into the cross-entropy loss raises the correlation to 0.34 on CIFAR10-H and 0.47 on ImageNet-16H, which the authors interpret as evidence that human annotations carry signal, although not enough to close the gap.

Load-bearing premise

The comparison rests on treating the predictive entropy of averaged human labels as the ground truth for human-perceived uncertainty; if averaging annotators before computing entropy does not faithfully represent how uncertain humans are, the reported correlations compare two entropy estimates rather than human and model uncertainty.

Editorial extensions

If this is right

  • Uncertainty-based triage systems will frequently flag images that humans find clear while missing images that humans find ambiguous, because model uncertainty is nearly independent of human uncertainty.
  • Calibration measured against correctness is insufficient for human-compatible trust: a model can be statistically well calibrated yet order its uncertainties in a way that conflicts with human judgment.
  • Human soft-label training moves model uncertainty toward human uncertainty on two benchmarks, so human annotations are a usable signal even if current losses under-use them.
  • The near-zero correlations on CIFAR-N suggest that low annotation counts per image can wash out any measurable human-model alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Averaging annotator labels before computing entropy collapses the full pattern of disagreement into a single mean distribution; computing human uncertainty as the mean of per-annotator entropies, or as the dispersion of the label distribution, could reveal more alignment than the paper reports.
  • Annotation depth likely explains part of the dataset difference: CIFAR10-H has fifty judgments per image while CIFAR-N has four to six, so its human-entropy estimates are far noisier; weighting by annotation count or requiring minimum judgments per image is a natural robustness check.
  • If the soft-label result replicates, a direct extension is to train a dedicated uncertainty head on held-out human disagreement while keeping the classifier loss fixed, and then measure both correlation and calibration separately, since the paper reports only correlation changes for the soft-label experiments.
  • The abstract's claim that soft-label training improves calibration without compromising accuracy is not backed by reported calibration or accuracy numbers in the full text; verifying that claim would require those measurements.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper compares human-perceived uncertainty, derived from human soft labels on three image benchmarks (CIFAR10-H, CIFAR-N, ImageNet-16H), with model uncertainty estimates from several baselines (CE, EUAT, ensembles, CALS, isotonic regression, DEUP). Using Pearson correlation as the main metric, it reports weak or near-zero correlations overall, weak correlations stratified by task complexity, and improved correlations after training with human soft labels. The abstract further claims that soft-label training improves calibration without compromising accuracy. The central conclusion is that model-estimated uncertainty is essentially uncorrelated with human-perceived uncertainty, with implications for human-AI trust.

Significance. If established, the negative correlation result would be a valuable contribution to uncertainty estimation and human-AI interaction, since it challenges the common assumption that calibrated model confidence reflects human-perceived uncertainty. The study benefits from using multiple public benchmarks, several uncertainty baselines, and ten training seeds. The main correlational analysis is self-contained: human and model entropies are measured independently, so the null result is not forced by construction. However, the statistical treatment has gaps that currently undermine the headline claims, particularly the discretization of the human uncertainty target and the absence of any calibration or accuracy metric for the soft-label training claim.

major comments (4)
  1. [Section 3.1, Tables 2 and 3] For CIFAR-N and ImageNet-16H, human uncertainty is computed as the entropy of averaged soft labels from only four to six annotations per image, so this quantity takes a small number of discrete values. Pearson correlation between this coarse discrete variable and a nearly continuous model entropy is therefore capped well below 1, and the cap is tightest for CIFAR-N, where the paper reports values of 0.003–0.005 across all baselines. The manuscript does not report the number of unique human-entropy levels, their distribution, or a Monte Carlo/analytical ceiling for Pearson r under the observed annotation counts. The near-zero CIFAR-N correlations may thus be an attenuation artifact rather than evidence of no alignment, and the same mechanism may depress the 0.24/0.35 values elsewhere. Please add a discretization ceiling analysis, report correlations on subsets with sufficient label counts, or use a rank-based measure with an explicit attenuation correction.
  2. [Abstract, Section 5, Table 4] The abstract's claim that incorporating human soft labels 'can improve calibration without compromising accuracy' is not supported anywhere in the full text: Table 4 reports only Pearson correlations, and no accuracy or calibration metric (e.g., ECE, reliability diagram, top-1 accuracy) is reported for either the baseline or the soft-label-trained model. This claim is load-bearing for the paper's positive contribution, so please report accuracy and calibration metrics with standard deviations over the ten seeds, for both the CE and CE-with-soft-labels conditions, and state whether any observed correlation improvement comes at a measurable accuracy cost.
  3. [Section 3.1, Tables 2–4] The paper states that every method was trained with ten different seeds, yet every result table reports a single Pearson coefficient with no variance, confidence interval, or per-seed breakdown. This makes it impossible to judge whether the differences between baselines, or the improvements in Table 4, are statistically meaningful, especially given the small effect sizes. Please report mean and standard deviation (or confidence intervals) across seeds for all reported correlations, and perform significance tests that account for multiple comparisons across baselines and datasets.
  4. [Sections 4 and 5] The conclusion that 'there is no correlation between human perception uncertainty and model-derived uncertainty estimates' overstates the results, because two of the three datasets show statistically significant though weak correlations (up to 0.24 and 0.35), and the third dataset is subject to the discretization problem described above. Please soften the conclusion to 'weak or mostly absent correlation', explicitly acknowledge the attenuation limitation, and discuss what can be concluded from the significant weak correlations versus the CIFAR-N null result.
minor comments (7)
  1. [Section 3.1] Dataset names are used inconsistently (e.g., 'Cifar10-H' vs 'CIFAR10-H', 'CifarN' vs 'CIFAR-N'); please adopt the original dataset names throughout.
  2. [Table 3] No p-values or significance markers are reported for the subgroup correlations, so it is unclear which coefficients are distinguishable from zero; please add significance information or explicitly mark non-significant entries.
  3. [Figure 1] The pink triangle markers for the mean are difficult to see in grayscale and are not described in a legend; please use more visible markers and a colorblind-safe legend.
  4. [Section 4] The text states that p-values are below the 0.05 threshold for two datasets but does not report the actual p-values; please provide exact p-values in the table or text.
  5. [Section 3.1] It is unclear whether the subset of human annotations used for soft-label training comes from the training split or the test split; if test-human labels are used in training, the Table 4 correlations would be biased upward, so please clarify the data split and the size of the subset.
  6. [Section 3.2] DEUP [16] is described as a post-processing calibration method, but DEUP is a direct uncertainty prediction method rather than a post-hoc calibration technique; please correct this categorization.
  7. [Table 1, Section 3.2] The 'difficult' task-complexity category mixes images where the model is correct and the human is wrong with images where both are wrong, which means the stratification conditions on model correctness; this may distort the subgroup correlations, and the paper does not discuss this potential selection effect.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central correlation study measures model and human uncertainty independently, and the self-citations are only baselines, not load-bearing.

full rationale

The paper's central claim—that model-estimated uncertainty is weakly or not correlated with human-perceived uncertainty—is the outcome of a direct empirical comparison, not a construction. Human uncertainty is computed as the predictive entropy of averaged human soft labels (CIFAR10-H, CIFAR-N, ImageNet-16H), while model uncertainty is computed as predictive entropy from independently trained models using CE, MC dropout, ensembles, isotonic regression, DEUP, EUAT, and CALS. The two quantities are measured on the same test images but are not defined in terms of each other, so a null correlation cannot be forced by definition. The paper's self-citations to EUAT [22] and CLUE [23] are used only as baseline uncertainty-estimation methods alongside several external baselines (CE [42], deep ensembles [17], isotonic regression [41], DEUP [16], CALS [20]); they do not provide the load-bearing justification for the main negative result. The soft-label training experiment (Table 4) does train models with human-derived soft labels and then evaluates correlation against human-derived uncertainty, so the improvement in correlation is not surprising, but the paper does not present it as a first-principles prediction; it is an explicit intervention that re-evaluates the updated model against the same type of human annotations. No equation shows that the evaluated correlation reduces to the training objective by construction, and the held-out test-split design described in Section 3.1 is consistent with an independent evaluation. The abstract's claim that soft-label training improves calibration without compromising accuracy is unsupported by any reported calibration or accuracy metric, but missing evidence is a correctness concern, not circularity. Overall, no circular step can be identified from the paper's own text.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The core comparison rests on treating annotator disagreement or confidence as ground-truth human uncertainty and on Pearson correlation as the alignment metric, both of which are reasonable but contestable modeling choices. No new entities are introduced, and the only load-bearing free parameter is the unreported soft-label subset size behind the positive result in Table 4.

free parameters (1)
  • soft-label training subset size
    Section 3.2 states that 'a subset of human annotations was incorporated into the model's training' without giving size, selection, or mixing ratio; the improved correlations in Table 4 depend on this unreported choice.
assumptions (3)
  • domain assumption Human perceptual uncertainty is adequately operationalized by the predictive entropy of the averaged human soft-label distribution.
    Section 3.1: 'By applying the same uncertainty estimation method for both human and model assessments, a direct comparison of the results becomes possible.' If annotator disagreement or confidence does not capture human-perceived uncertainty, the correlation analysis does not measure the stated construct.
  • domain assumption Human epistemic uncertainty is negligible on these everyday-object benchmarks, so human uncertainty is treated as aleatoric.
    Section 3.1: 'epistemic uncertainty ... of human predictions can be assumed to be low. In this context, human uncertainty likely arises from noise, ambiguity, or inherent randomness.' This assumption justifies comparing total human uncertainty to total model uncertainty without separating components.
  • domain assumption Pearson correlation is an appropriate and sufficient measure of alignment between human and model uncertainty.
    Section 3.1: 'Pearson's correlation coefficient is computed for each baseline.' Nonlinear but monotone relationships, or agreement on only the most uncertain examples, would be missed, yet the paper draws global conclusions from this linear metric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncertainty Estimation by Human Perception versus Neural Models." pith.science (2026). https://pith.science/paper/SWTY73GO

@misc{pith2026250615850,
  author       = {Pith},
  title        = {Pith review of: Uncertainty Estimation by Human Perception versus Neural Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SWTY73GO}},
  note         = {Machine review of arXiv:2506.15850}
}
read the original abstract

Modern neural networks (NNs) often achieve high predictive accuracy but are poorly calibrated, producing overconfident predictions even when wrong. This miscalibration poses serious challenges in applications where reliable uncertainty estimates are critical. In this work, we investigate how human perceptual uncertainty compares to uncertainty estimated by NNs. Using three vision benchmarks annotated with both human disagreement and crowdsourced confidence, we assess the correlation between model-predicted uncertainty and human-perceived uncertainty. Our results show that current methods only weakly align with human intuition, with correlations varying significantly across tasks and uncertainty metrics. Notably, we find that incorporating human-derived soft labels into the training process can improve calibration without compromising accuracy. These findings reveal a persistent gap between model and human uncertainty and highlight the potential of leveraging human insights to guide the development of more trustworthy AI systems.

Figures

Figures reproduced from arXiv: 2506.15850 by the authors.

Figure 1
Figure 1. Normalized uncertainty distribution using different baselines, models, and [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 41 canonical work pages

  1. [1]

    Information Fusion76, 243–297 (2021)

    Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U.R., Makarenkov, V., Nahavandi, S.:Areviewofuncertaintyquantificationindeeplearning:Techniques,applications and challenges. Information Fusion76, 243–297 (2021)

    work page 2021

  2. [2]

    In: Proceedings of the 34th International Conference on Neural Informa- tion Processing Systems

    Antorán,J.,Allingham,J.U.,Hernández-Lobato,J.M.:Depthuncertaintyinneural networks. In: Proceedings of the 34th International Conference on Neural Informa- tion Processing Systems. NIPS’20, Curran Associates Inc., Red Hook, NY, USA (2020)

    work page 2020

  3. [3]

    In: Proceedings of the 32nd International Conference on Interna- tional Conference on Machine Learning - Volume 37 (2015)

    Blundell, C., Cornebise, J., Kavukcuoglu, K., Wierstra, D.: Weight uncertainty in neural networks. In: Proceedings of the 32nd International Conference on Interna- tional Conference on Machine Learning - Volume 37 (2015)

    work page 2015

  4. [4]

    Single Shot MC Dropout Approximation

    Brach, K., Sick, B., Dürr, O.: Single shot mc dropout approximation. ArXiv abs/2007.03293(2020), https://api.semanticscholar.org/CorpusID:220381176

    work page Pith review arXiv 2020

  5. [5]

    In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A

    Einbinder, B.S., Romano, Y., Sesia, M., Zhou, Y.: Training uncertainty-aware clas- sifiers with conformalized deep learning. In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (eds.) Advances in Neural Information Processing Systems. vol. 35 (2022)

    work page 2022

  6. [6]

    IEEE Transactions on Radiation and Plasma Medical Sciences 5(2021)

    Fan, F.L., Xiong, J., Li, M., Wang, G.: On interpretability of artificial neural networks: A survey. IEEE Transactions on Radiation and Plasma Medical Sciences 5(2021)

    work page 2021

  7. [7]

    In: Proceedings of The 33rd International Conference on Machine Learning

    Gal, Y., Ghahramani, Z.: Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In: Proceedings of The 33rd International Conference on Machine Learning. vol. 48. PMLR (2016)

    work page 2016

  8. [8]

    In: Proceedings of the 34th International Conference on Machine Learning

    Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neu- ral networks. In: Proceedings of the 34th International Conference on Machine Learning. vol. 70. PMLR (2017)

    work page 2017

Show all 42 references
  1. [9]

    In: International Conference on Learning Representations (2021)

    Gupta, K., Rahimi, A., Ajanthan, T., Mensink, T., Sminchisescu, C., Hartley, R.: Calibration of neural networks using splines. In: International Conference on Learning Representations (2021)

    2021

  2. [10]

    In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    2016

  3. [11]

    AI Magazine8(3) (1987)

    Hink, R.F., Woods, D.L.: How humans process uncertain knowledge: An introduc- tion for knowledge engineers. AI Magazine8(3) (1987)

    1987

  4. [12]

    Journal of Machine Learning Research14(40) (2013)

    Hoffman, M.D., Blei, D.M., Wang, C., Paisley, J.: Stochastic variational inference. Journal of Machine Learning Research14(40) (2013)

    2013

  5. [13]

    In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W

    Karandikar, A., Cain, N., Tran, D., Lakshminarayanan, B., Shlens, J., Mozer, M.C., Roelofs, B.: Soft calibration objectives for neural networks. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems. vo...

    2021

  6. [14]

    In: Advances in Neural Information Processing Systems

    Krishnan, R., Tickoo, O.: Improving model calibration with accuracy versus un- certainty optimization. In: Advances in Neural Information Processing Systems. vol. 33 (2020)

    2020

  7. [15]

    In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics

    Kull, M., Filho, T.S., Flach, P.: Beta calibration: a well-founded and easily imple- mented improvement on logistic calibration for binary classifiers. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. vol. 54 (2017) 12 Pedro Mende...

    2017

  8. [16]

    Transactions on Machine Learning Research (2023), https://openreview.net/forum?id=eGLdVRvvfQ

    Lahlou, S., Jain, M., Nekoei, H., Butoi, V.I., Bertin, P., Rector- Brooks, J., Korablyov, M., Bengio, Y.: DEUP: Direct epistemic uncer- tainty prediction. Transactions on Machine Learning Research (2023), https://openreview.net/forum?id=eGLdVRvvfQ

    2023

  9. [17]

    In: Proceedings of the 31st Interna- tional Conference on Neural Information Processing Systems (2017)

    Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. In: Proceedings of the 31st Interna- tional Conference on Neural Information Processing Systems (2017)

    2017

  10. [18]

    Department of Computer Science University of Massachusetts (2013)

    Learned-Miller, E.G.: Entropy and mutual information. Department of Computer Science University of Massachusetts (2013)

    2013

  11. [19]

    In: Computer Vision and Pattern Recognition Conference (2022)

    Liu, B., Ayed, I.B., Galdran, A., Dolz, J.: The devil is in the margin: Margin- based label smoothing for network calibration. In: Computer Vision and Pattern Recognition Conference (2022)

    2022

  12. [20]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Liu, B., Rony, J., Galdran, A., Dolz, J., Ben Ayed, I.: Class adaptive network calibration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16070–16079 (2023)

    2023

  13. [21]

    Neural Computation4(3), 448–472 (1992)

    MacKay, D.J.C.: A Practical Bayesian Framework for Backpropagation Networks. Neural Computation4(3), 448–472 (1992)

    1992

  14. [22]

    In: Proceedings of the 27th European Conference on Artificial Intelligence (2024)

    Mendes, P., Romano, P., Garlan, D.: Error-driven uncertainty aware training. In: Proceedings of the 27th European Conference on Artificial Intelligence (2024)

    2024

  15. [23]

    arXiv preprint arXiv:2505.22803 (2025)

    Mendes, P., Romano, P., Garlan, D.: Clue: Neural networks calibration via learning uncertainty-error alignment. arXiv preprint arXiv:2505.22803 (2025)

    2025 arXiv

  16. [24]

    In: Advances in Neural Information Processing Systems

    Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P., Dokania, P.: Cali- brating deep neural networks using focal loss. In: Advances in Neural Information Processing Systems. vol. 33 (2020)

    2020

  17. [25]

    In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (2015)

    Naeini,M.P.,Cooper,G.F.,Hauskrecht,M.:Obtainingwellcalibratedprobabilities using bayesian binning. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (2015)

    2015

  18. [26]

    Springer-Verlag (1996)

    Neal, R.M.: Bayesian Learning for Neural Networks. Springer-Verlag (1996)

    1996

  19. [27]

    In: Advances in Neural Information Processing Systems

    Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lak- shminarayanan, B., Snoek, J.: Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In: Advances in Neural Information Processing Systems. vol. 32 (2019)

    2019

  20. [28]

    2019 IEEE/CVF International Conference on Computer Vision (2019)

    Peterson, J.C., Battleday, R.M., Griffiths, T.L., Russakovsky, O.: Human uncer- tainty makes classification more robust. 2019 IEEE/CVF International Conference on Computer Vision (2019)

    2019

  21. [29]

    In: Advances in Large Margin Classifiers (1999)

    Platt, J.C.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in Large Margin Classifiers (1999)

    1999

  22. [30]

    Nature Neuroscience19(3) (2016)

    Pouget, A., Drugowitsch, J., Kepecs, A.: Confidence and certainty: distinct prob- abilistic quantities for different goals. Nature Neuroscience19(3) (2016)

    2016

  23. [31]

    In: Artstein, R., Boleda, G., Keller, F., Schulte im Walde, S

    Reidsma, D., op den Akker, R.: Exploiting ‘subjective’ annotations. In: Artstein, R., Boleda, G., Keller, F., Schulte im Walde, S. (eds.) Coling 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics (Aug 2008)

    2008

  24. [32]

    International Journal of Computer Vision (IJCV)115(3), 211–252 (2015)

    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV)115(3), 211–252 (2015). https://d...

    2015 doi

  25. [33]

    In: International Design Engineering Tech- nical Conferences and Computers and Information in Engineering Conference

    Segalman, D.J., Brake, M.R., Bergman, L.A., Vakakis, A.F., Willner, K.: Epistemic and aleatoric uncertainty in modeling. In: International Design Engineering Tech- nical Conferences and Computers and Information in Engineering Conference. vol. Volume 8: 22nd Reliability, Stres...

    2013

  26. [34]

    ArXivabs/2105.11828(2021)

    Seuss, D.: Bridging the gap between explainable ai and uncertainty quantification to enhance trustability. ArXivabs/2105.11828(2021)

    2021 arXiv

  27. [35]

    ArXivabs/2110.03260(2023)

    Shamsi, A., Asgharnezhad, H., Tajally, A., Nahavandi, S., Leung, H.: An uncertainty-aware loss function for training neural networks with calibrated pre- dictions. ArXivabs/2110.03260(2023)

    2023 arXiv

  28. [36]

    The Bell System Tech- nical Journal27(1948)

    Shannon, C.E.: A mathematical theory of communication. The Bell System Tech- nical Journal27(1948)

    1948

  29. [37]

    Proceedings of the National Academy of Sciences119(11) (2022)

    Steyvers, M., Tejeda, H., Kerrigan, G., Smyth, P.: Bayesian modeling of human–ai complementarity. Proceedings of the National Academy of Sciences119(11) (2022)

    2022

  30. [38]

    Science185(4157) (1974)

    Tversky, A., Kahneman, D.: Judgment under uncertainty: Heuristics and biases. Science185(4157) (1974)

    1974

  31. [39]

    ACM Comput

    Wang, H., Yeung, D.Y.: A survey on bayesian deep learning. ACM Comput. Surv. 53(5) (2020)

    2020

  32. [40]

    In: International Conference on Learning Representations (2022)

    Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., Liu, Y.: Learning with noisy labels re- visited: A study using real-world human annotations. In: International Conference on Learning Representations (2022)

    2022

  33. [41]

    In: Proceedings of the Eighteenth International Conference on Machine Learning

    Zadrozny, B., Elkan, C.: Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In: Proceedings of the Eighteenth International Conference on Machine Learning. ICML ’01 (2001)

    2001

  34. [42]

    In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K.,Cesa-Bianchi,N.,Garnett,R.(eds.)AdvancesinNeuralInformationProcessing Systems

    Zhang, Z., Sabuncu, M.: Generalized cross entropy loss for training deep neural networks with noisy labels. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K.,Cesa-Bianchi,N.,Garnett,R.(eds.)AdvancesinNeuralInformationProcessing Systems. vol. 31 (2018)

    2018

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.