Human uncertainty makes classification more robust

Joshua C. Peterson; Olga Russakovsky; Ruairidh M. Battleday; Thomas L. Griffiths

REVIEW 3 major objections 3 minor 1 cited by

Human uncertainty makes classification more robust

T0 review · 3 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training classifiers on full human label distributions improves out-of-distribution generalization and attack resistance.

desk verdict A solid empirical paper introducing CIFAR10H and showing human soft labels improve OOD generalization, but the adversarial robustness claim is overstated relative to the PGD results. read the letter →

arxiv 1908.07086 v1 pith:34UNMSIC submitted 2019-08-19 cs.CV

Joshua C. Peterson , Ruairidh M. Battleday , Thomas L. Griffiths , Olga Russakovsky This is my paper · ORCID

classification cs.CV

keywords humanuncertaintysoftlabelsCIFAR10Hdistributionalshiftadversarialrobustnessgeneralizationlabeldistributionssecond-bestaccuracy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces CIFAR10H, a dataset that augments the CIFAR10 test set with full human label distributions: roughly 50 crowdsourced judgments per image, 511,400 in total. The authors' central claim is that training classifiers on these soft labels, instead of one-hot ground-truth labels, teaches them the similarity structure that human perception uses, and that this transfers as better generalization under distributional shift and stronger resistance to adversarial attacks. Across eight CNN architectures, soft-label training improved accuracy on every out-of-distribution test set considered, with the gains growing as the test data moved further from the training distribution. If the claim stands, human label uncertainty is a practical and comparatively cheap training signal for robustness, one that also gives a new benchmark for whether a model's errors are human-like.

What carries the argument

The load-bearing object is the measured human label distribution $p_{\mathrm{hum}}(y|x)$ for each image, collected as a vector of ten category probabilities from the CIFAR10H judgments. The argument is carried by replacing the one-hot label in the empirical risk with this distribution, so that for a network output $p_{\theta}(y|x)$ the training loss is the cross-entropy $-\frac{1}{m}\sum_{j=1}^{m}\sum_{c} p_{\mathrm{hum}}(y_j=c|x_j)\,\log p_{\theta}(y_j=c|x_j)$. This is exactly the expected negative log-likelihood under the human distribution, used as a proxy for the true conditional label distribution $p(y|x)$. The same human distribution also defines the evaluation target, so CIFAR10H doubles as a benchmark for whether a model's uncertainty is human-like.

What would settle it

Train the same eight architectures on CIFAR10H soft labels, attack them with PGD at an $\ell_\infty$ bound of 8 over more iterations than the paper reports, and compare cross-entropy trajectories with hard-label controls; if the soft-label advantage disappears or reverses, the robustness claim is falsified. A second test would be to collect human soft labels for a different base dataset and check whether the distributional-shift accuracy gains replicate; if they do not, the effect is specific to CIFAR10H rather than to human uncertainty as a training signal.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that substituting the full human label distribution $p_{\mathrm{hum}}(y|x)$ for a single modal label in the training objective changes what a network learns: its predictions come to resemble human patterns of uncertainty rather than merely matching consensus labels. Trained this way, all eight architectures improved accuracy relative to hard-label controls on CIFAR10, CIFAR10.1 v4 and v6, CINIC10, and ImageNet-Far, and the margin grew with distributional shift, from about one percentage point on CIFAR10 to about two on ImageNet-Far. Cross-entropy to ground truth fell by roughly a third on CIFAR10 and by 38% on ImageNet-Far, and second-best accuracy rose by about five points on average. Under FGSM attacks, cross-entropy was roughly halved compared with hard-label fine-tuning, and accuracy was higher for five of eight architectures; under PGD, loss rose more slowly with each iteration. The paper also reports that human soft labels outperform alternative ways of producing soft targets, including class-level penalties, an ensemble knowledge-distillation baseline, mixup, and labels sampled from the human distribution.

Load-bearing premise

The load-bearing premise is that the way humans distribute probability over the ten classes for an image is a good estimate of the true conditional label distribution $p(y|x)$; if human perceptual uncertainty points to the wrong categories, or is irrelevant to the classification task, the reported gains would be specific to mimicking human choices rather than general robustness.

Editorial extensions

If this is right

Soft-label training on CIFAR10H improves accuracy and lowers cross-entropy on all tested datasets, with larger gains as the test distribution moves further from CIFAR10.
The same training makes model errors more human-like: models are less confident when wrong, have better second-best accuracy, and fit human label distributions more closely.
Human soft targets beat other soft-label methods, including class-level penalties, ensemble knowledge distillation, mixup, and sampled hard labels, both at matching human uncertainty and at out-of-distribution generalization.
Fine-tuning on human labels roughly halves cross-entropy under FGSM attacks and slows loss growth under PGD, without any explicit adversarial defense.
The benefit arises from the human information itself, not from the mere fact of training with distributed labels; category-level and network-generated soft labels do not reproduce it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: if the mechanism generalizes, the concentration of human uncertainty in roughly 30% of images means targeted collection of high-uncertainty labels could capture most of the robustness benefit at a fraction of the labeling cost.
Inference: since the largest gains appear under distributional shift, human soft labels collected on a source domain could serve as a domain-adaptation signal before target-domain labels exist.
Inference: evaluating models with human-label cross-entropy or second-best accuracy could become a standard complement to top-1 accuracy, revealing robustness differences that accuracy alone hides.
Inference: combining human soft labels with explicit adversarial training or with larger-scale distillation is an untested extension suggested by the paper's mechanism, and the two sources of robustness might compound.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The real news here is the dataset: CIFAR10H, full human label distributions for all 10,000 CIFAR10 test images, built from half a million judgments. That is a genuinely useful resource, and the paper deserves credit for releasing it. The training story is also credible: across eight architectures, training on these soft labels improves accuracy and crossentropy on CIFAR10.1 and on their constructed ImageNet-Far shift, and the gains grow with shift. The comparison against mixup, class-level soft labels, and sampled hard labels is the right kind of control, and the result that image-level human uncertainty beats the alternatives is convincing. I buy the central generalization claim.

Where I part company is the adversarial robustness headline. The abstract says the method 'confers robustness to adversarial attacks,' but the paper's own PGD numbers show accuracy collapsing to 0% and 1% under both label schemes. What actually improves is crossentropy during the attack, which is a calibration/confidence effect, not resistance to misclassification. The FGSM table is also mixed: accuracy improves for five of eight architectures and degrades for two, with no error bars or significance tests. So the robustness claim is overstated. It should be restricted to 'lower crossentropy under FGSM and slower loss growth under PGD,' not 'robustness.' This matters because the abstract and contribution bullets will be read as claiming defensive robustness, and the evidence does not support that.

Other soft spots are minor. There are no error bars in the main tables. The dataset link is missing from this version. And the underlying assumption that human perceptual uncertainty approximates the natural p(y|x) is a modeling choice that is not independently validated. But the empirical gains do not hinge on that assumption being literally true; the method works whether or not humans are the 'gold standard.'.

The paper is worth a serious referee. The central generalization result is solid and the dataset will be reused regardless of the robustness quibble. I would send it to review, and I would push the authors to fix the robustness language, add variance estimates, and release the data before publication.

Referee Report

3 major / 3 minor

Summary. The paper introduces CIFAR10H, a new dataset of full human label distributions for the 10,000-image CIFAR10 test set, collected from 511,400 crowdsourced judgments. The authors train eight CNN architectures on these soft labels and compare them against hard-label controls and several soft-label baselines (mixup, class-level soft labels, and sampled hard labels). They report that training on human soft labels improves accuracy and crossentropy on increasingly out-of-distribution test sets (CIFAR10.1, CINIC10, ImageNet-Far), improves second-best accuracy, and yields better fits to human uncertainty. The paper also presents FGSM and PGD adversarial experiments and claims that human-label training confers adversarial robustness.

Significance. If the generalization results hold, CIFAR10H is a valuable community resource, and the demonstration that image-level human uncertainty can improve out-of-distribution generalization is a meaningful empirical contribution. The experimental design is solid in several respects: eight architectures are trained with 10-fold cross-validation, multiple external datasets are used, and competitive baselines such as mixup, class-level soft labels, and sampled hard labels are included. The paper would be strengthened by error bars and statistical tests, and the adversarial robustness claim is overstated relative to the evidence. Nonetheless, the core dataset and the generalization findings are useful and likely of interest to the computer vision community.

major comments (3)

[Section 7, Table 2, Figure 4] The abstract and the fourth contribution bullet state that training with CIFAR10H 'confers robustness to adversarial attacks,' but the evidence in Section 7 supports a narrower claim. Under PGD, the paper reports that accuracy was driven to 0% for standard-label networks and 1% for human-label networks; the plotted advantage is only that crossentropy grows more slowly (Figure 4). In Table 2, under FGSM, accuracy improves for five of eight architectures, degrades for PyramidNet and ResNext, and is tied for Shake-Shake, and no error bars or significance tests are given. Because soft-label models are less confident on errors (Figure 3a), part of the crossentropy gap after attack may reflect this baseline confidence shift rather than increased robustness to misclassification. Please restrict the robustness claim to the measured crossentropy/loss behavior under these attacks, or add accuracy-based results under stronger attacks with proper statistics.
[Section 5.2, Figure 2] The central generalization claim that accuracy improves on all generalization datasets and for every individual CNN is presented without variance information. The reported improvements are small (about 1 percentage point on CIFAR10 and 2 percentage points on ImageNet-Far), and the statement that the pattern replicated across folds is asserted but not shown. Please provide error bars or confidence intervals across the 10 folds for the main results, and ideally a paired statistical test across architectures, so that the reliability of the observed gains can be assessed.
[Section 3 and Section 6] The theoretical motivation in Section 3 rests on the assumption that the human label distribution p_hum(y|x) is a good estimator of the true conditional label distribution p(y|x). This assumption is not independently verified, and the benchmark for 'fit to humans' in Section 6 uses the same CIFAR10H labels, so high fit is partly by construction. The comparisons against mixup and class-level soft labels are informative, but an explicit test against a generic soft-label regularizer with matched label entropy (e.g., label smoothing) would help establish that the benefits come specifically from human-like uncertainty rather than from any form of softened targets.

minor comments (3)

[Section 4.2] There are small presentation errors: 'evalulate' should be 'evaluate' in Section 4.1, and the number of participants removed by attention checks is given as ' 14 total' with a stray leading space; please clarify the count.
[Table 1 caption] The caption lists columns from left to right but the column headers use abbreviations 'c10H', 'c10', 'v4', and 'v6' that are not defined in the table or its caption. Please define these abbreviations and state explicitly that lower crossentropy is better.
[Section 7] The text says that l-infinity bounds of 4 to 8 in increments of 1 were explored but that all results are reported with a bound of 4 for brevity. Please show at least a summary of the other bounds or justify why they were omitted.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central claims rest on external held-out and out-of-distribution evaluations rather than on fitted inputs or self-citations.

full rationale

The paper's central claims are empirical comparisons, not derivations. Models are trained on 9,000-image folds of CIFAR10H and evaluated on held-out folds and, more importantly, on external and shifted benchmarks (CIFAR10 train set, CIFAR10.1, CINIC10, ImageNet-Far) and under FGSM/PGD attacks. The one relation that can look tautological—training on human soft labels improves fit to held-out human soft labels in Table 1—is a standard cross-validated benchmark and is not the load-bearing generalization claim; the out-of-distribution accuracy, crossentropy, SBA, and adversarial results are measured against ground-truth labels or external datasets. Section 3's use of p_hum(y|x) as a proxy for p(y|x) is an explicit modeling assumption, not a definition that forces the reported improvements. No load-bearing self-citations or imported uniqueness theorems are used; the citation to Griffiths's earlier work (e.g., [23]) appears only in related-work background. The PGD results (accuracy driven to 0% for hard-label and only 1% for soft-label networks) do weaken the abstract's unqualified adversarial-robustness claim, since the measured advantage is chiefly in crossentropy/calibration, but that is an evidentiary or overstatement concern, not circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical. No numerical parameters are fitted to make a derivation work; the learning-rate choice is a hyperparameter shared by both conditions. The load-bearing input is the unproven assumption that human label distributions are a good proxy for the natural conditional label distribution, plus standard cross-entropy math.

assumptions (3)

domain assumption Human label distributions p_hum(y|x) approximate the true conditional label distribution p(y|x).
Invoked in Section 3 to justify minimizing cross-entropy to human labels as the optimal objective; if false, the claimed generalization and robustness benefits are human-specific rather than general.
standard math The expectation identity in Eq. (3) (cross-entropy with human distribution) follows from standard conditional-likelihood decomposition.
Used in Section 3 to move from objective (2) to objective (3); it is a standard identity under the proxy assumption.
domain assumption Crowdsourced labels from 2,571 Amazon Mechanical Turk workers, filtered by attention checks, provide reliable per-image label distributions.
Section 4.2; the dataset quality depends on this. The paper reports removing 14 participants with attention accuracy below 75%, which mitigates but does not eliminate quality concerns.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Human uncertainty makes classification more robust." pith.science (2026). https://pith.science/paper/34UNMSIC

@misc{pith2026190807086,
  author       = {Pith},
  title        = {Pith review of: Human uncertainty makes classification more robust},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/34UNMSIC}},
  note         = {Machine review of arXiv:1908.07086}
}

read the original abstract

The classification performance of deep neural networks has begun to asymptote at near-perfect levels. However, their ability to generalize outside the training set and their robustness to adversarial attacks have not. In this paper, we make progress on this problem by training with full label distributions that reflect human perceptual uncertainty. We first present a new benchmark dataset which we call CIFAR10H, containing a full distribution of human labels for each image of the CIFAR10 test set. We then show that, while contemporary classifiers fail to exhibit human-like uncertainty on their own, explicit training on our dataset closes this gap, supports improved generalization to increasingly out-of-training-distribution test datasets, and confers robustness to adversarial attacks.

Figures

Figures reproduced from arXiv: 1908.07086 by the authors.

**Figure 1.** CIFAR10 images for which humans and our best traditionally-trained CNN (Shake-Shake [11]) agree in their top guess, but systematically differ over other choices. (see [1]). Indeed, consider the examples shown in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** Generalization results. Left: accuracy against ground-truth labels, for increasingly out-of-training-sample distri [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** (A) Mean confidence for correctly/incorrectly [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Crossentropy as a function of PGD iteration. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Detecting LLM-Generated Short Answers and Effects on Learner Performance
cs.HC 2025-06 conditional novelty 5.0 of 10

A fine-tuned GPT-4o detects human-annotated LLM-generated short answers at 80% accuracy, outperforming GPTZero, and flagged LLM use is associated with higher posttest MCQ scores.

Reference graph

Works this paper leans on

54 extracted references · 42 canonical work pages · cited by 1 Pith paper

[1]

The moral machine experiment

Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-Franc ¸ois Bon- nefon, and Iyad Rahwan. The moral machine experiment. Nature, 563(7729):59, 2018

work page 2018
[2]

Wild patterns: Ten years af- ter the rise of adversarial machine learning

Battista Biggio and Fabio Roli. Wild patterns: Ten years af- ter the rise of adversarial machine learning. Pattern Recog- nition, 84:317–331, 2018

work page 2018
[3]

Jonathan Bragg, Mausam, and Daniel S. Weld. Crowd- sourcing multi-label classiﬁcation for taxonomy creation. In Conference on Human Computation and Crowdsourcing (HCOMP), 2013

work page 2013
[4]

Visual recognition with humans in the loop

Steve Branson, Catherine Wah, Florian Schroff, Boris Babenko, Peter Welinder, Pietro Perona, and Serge Belongie. Visual recognition with humans in the loop. In European Conference on Computer Vision (ECCV), 2010

work page 2010
[5]

Amazon’s mechanical turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1):3–5, 2011

Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. Amazon’s mechanical turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1):3–5, 2011

work page 2011
[6]

L. B. Chilton, G. Little, D. Edge, D. S. Weld, and J. A. Lan- day. Cascade: Crowdsourcing taxonomy creation. InConfer- ence on Human Factors in Computing Systems (CHI), 2013

work page 2013
[7]

Crowley, Antreas Antoniou, and Amos J

Luke Nicholas Darlow, Elliot J. Crowley, Antreas Antoniou, and Amos J. Storkey. CINIC-10 is not imagenet or CIFAR-

work page
[8]

Large-scale object classiﬁcation using la- bel relation graphs

Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classiﬁcation using la- bel relation graphs. In European Conference on Computer Vision (ECCV), 2014

work page 2014

Show all 54 references

[9]

Fergus, H

R. Fergus, H. Bernal, Y . Weiss, and A. Torralba. Semantic label sharing for learning with many categories. InEuropean Conference on Computer Vision (ECCV), 2010

2010
[10]

arXiv preprint arXiv:1810.03505, 2018

2018 arXiv
[11]

Frome, G.S

A. Frome, G.S. Corrado, J. Shlens, S. Bengio, J. Dean, and T. Mikolov. Devise: A deep visual-semantic embedding model. In Advances in Neural Information Processing Sys- tems (NeurIPS), 2013

2013
[12]

Shake-shake regularization

Xavier Gastaldi. Shake-shake regularization. arXiv preprint arXiv:1705.07485, 2017

2017 arXiv
[13]

Robust loss functions under label noise for deep neural networks

Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. InCon- ference on Artiﬁcial Intelligence (AAAI), 2017

2017
[14]

Explaining and harnessing adversarial examples

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014

2014 arXiv
[15]

Grifﬁn and P

G. Grifﬁn and P. Perona. Learning and using taxonomies for fast visual categorization. In Computer Vision and Pattern Recognition (CVPR), 2008

2008
[16]

Deep pyra- midal residual networks

Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyra- midal residual networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017

2017
[17]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR), 2016

2016
[18]

Identity mappings in deep residual networks

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision (ECCV), 2016

2016
[19]

Diamos, Heewoo Jun, Hassan Kianinejad, Md

Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically.arXiv preprint arXiv:1712.00409, 2017

2017 arXiv
[20]

Hinton, Oriol Vinyals, and Jeffrey Dean

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Dis- tilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

2015 arXiv
[21]

Densely connected convolutional net- works

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kil- ian Q Weinberger. Densely connected convolutional net- works. In Computer Vision and Pattern Recognition (CVPR), 2017

2017
[22]

Gpipe: Efﬁcient training of giant neural networks us- ing pipeline parallelism

Yanping Huang, Yonglong Cheng, Dehao Chen, Hy- oukJoong Lee, Jiquan Ngiam, Quoc V Le, and Zhifeng Chen. Gpipe: Efﬁcient training of giant neural networks us- ing pipeline parallelism. arXiv preprint arXiv:1811.06965, 2018

2018 arXiv
[23]

Hwang, F

S.J. Hwang, F. Sha, and K. Grauman. Sharing features be- tween objects and their attributes. In Computer Vision and Pattern Recognition (CVPR), 2011

2011
[24]

Jia, J.T

Y . Jia, J.T. Abbott, J. Austerweil, T. Grifﬁths, and T. Dar- rell. Visual concept learning: Combining machine vision and bayesian generalization on concept hierarchies. In Advances in Neural Information Processing Systems (NeurIPS), 2013

2013
[25]

Kingma and Jimmy Ba

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2014

2014
[26]

Crowdsourcing in Computer Vision

Adriana Kovashka, Olga Russakovsky, Li Fei-Fei, and Kristen Grauman. Crowdsourcing in Computer Vision. Foundation and Trends in Computer Graphics and Vision , 10(3):177–243, 2016

2016
[27]

Shamma, Michael S

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. ...

2017
[28]

Krishna, Kenji Hata, Stephanie Chen, Joshua Kravitz, David A

Ranjay A. Krishna, Kenji Hata, Stephanie Chen, Joshua Kravitz, David A. Shamma, Li Fei-Fei, and Michael S. Bern- stein. Embracing error to enable rapid crowdsourcing. In Conference on Human Factors in Computing Systems (CHI), 2016

2016
[29]

Learning multiple layers of features from tiny images

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009

2009
[30]

Ad- versarial examples in the physical world

Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Ad- versarial examples in the physical world. In International Conference on Learning Representations (ICLR), 2016

2016
[31]

Adver- sarial machine learning at scale

Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adver- sarial machine learning at scale. In International Conference on Learning Representations (ICLR), 2016

2016
[32]

Women, ﬁre, and dangerous things

George Lakoff. Women, ﬁre, and dangerous things. Univer- sity of Chicago press, 2008

2008
[33]

C.H. Lampert. Maximum margin multi-label structured pre- diction. In Advances in Neural Information Processing Sys- tems (NeurIPS), 2011

2011
[34]

Microsoft coco: Common objects in context

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014

2014
[35]

Marszalek and C

M. Marszalek and C. Schmid. Semantic hierarchies for vi- sual object recognition. In Computer Vision and Pattern Recognition (CVPR), 2007

2007
[36]

Combining crowd and expert labels using decision theoretic active learning

An Thanh Nguyen, Byron C Wallace, and Matthew Lease. Combining crowd and expert labels using decision theoretic active learning. In Conference on Human Computation and Crowdsourcing (HCOMP), 2015

2015
[37]

Learning classiﬁcation models with soft-label information

Quang Nguyen, Hamed Valizadegan, and Milos Hauskrecht. Learning classiﬁcation models with soft-label information. Journal of the American Medical Informatics Association , 21(3):501–508, 2014

2014
[38]

Distillation as a defense to adver- sarial perturbations against deep neural networks

Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adver- sarial perturbations against deep neural networks. In IEEE Symposium on Security and Privacy (SP) , pages 582–597. IEEE, 2016

2016
[39]

Automatic differentiation in PyTorch

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Al- ban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NeurIPS Autodiff Workshop, 2017

2017
[40]

Do cifar-10 classiﬁers generalize to cifar- 10? arXiv preprint arXiv:1806.00451, 2018

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classiﬁers generalize to cifar- 10? arXiv preprint arXiv:1806.00451, 2018

2018 arXiv
[41]

Belongie, and Nir Shavit

David Rolnick, Andreas Veit, Serge J. Belongie, and Nir Shavit. Deep learning is robust to massive label noise. arXiv preprint arXiv:1705.10694, 2017

2017 arXiv
[42]

Imagenet large scale visual recognition challenge

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252, 2015

2015
[43]

Very deep con- volutional networks for large-scale image recognition

Karen Simonyan and Andrew Zisserman. Very deep con- volutional networks for large-scale image recognition. In In- ternational Conference on Learning Representations (ICLR), 2014

2014
[44]

Rethinking the inception ar- chitecture for computer vision

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception ar- chitecture for computer vision. In Computer Vision and Pat- tern Recognition (CVPR), 2016

2016
[45]

In- triguing properties of neural networks

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. In- triguing properties of neural networks. In International Con- ference on Learning Representations (ICLR), 2013

2013
[46]

Unbiased look at dataset bias

Antonio Torralba, Alexei A Efros, et al. Unbiased look at dataset bias. In Computer Vision and Pattern Recognition (CVPR), 2011

2011
[47]

80 million tiny images: A large data set for nonparametric object and scene recognition

Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pat- tern Analysis and Machine Intelligence , 30(11):1958–1970, 2008

1958
[48]

Sharing visual features for multiclass and multiview object detection

Antonio Torralba, Kevin P Murphy, and William T Freeman. Sharing visual features for multiclass and multiview object detection. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 29(5):854–869, 2007

2007
[49]

Toward robustness against label noise in train- ing deep discriminative neural networks

Arash Vahdat. Toward robustness against label noise in train- ing deep discriminative neural networks. In Advances in Neural Information Processing Systems (NeurIPS). 2017

2017
[50]

Aggregated residual transformations for deep neural networks

Saining Xie, Ross B Girshick, Piotr Doll´ar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Computer Vision and Pattern Recogni- tion (CVPR), 2017

2017
[51]

Wide residual net- works

Sergey Zagoruyko and Nikos Komodakis. Wide residual net- works. In British Machine Vision Conference (BMVC), 2016

2016
[52]

mixup: Beyond empirical risk minimiza- tion

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion. In International Conference on Learning Representa- tions (ICLR), 2017

2017
[53]

Places: A 10 million image database for scene recognition

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 2017

2017
[54]

Zweig and D

A. Zweig and D. Weinshall. Exploiting object hierarchy: combining models from different category levels. In Inter- national Conference on Computer Vision (ICCV), 2007

2007

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

The moral machine experiment

Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-Franc ¸ois Bon- nefon, and Iyad Rahwan. The moral machine experiment. Nature, 563(7729):59, 2018

work page 2018

[2] [2]

Wild patterns: Ten years af- ter the rise of adversarial machine learning

Battista Biggio and Fabio Roli. Wild patterns: Ten years af- ter the rise of adversarial machine learning. Pattern Recog- nition, 84:317–331, 2018

work page 2018

[3] [3]

Jonathan Bragg, Mausam, and Daniel S. Weld. Crowd- sourcing multi-label classiﬁcation for taxonomy creation. In Conference on Human Computation and Crowdsourcing (HCOMP), 2013

work page 2013

[4] [4]

Visual recognition with humans in the loop

Steve Branson, Catherine Wah, Florian Schroff, Boris Babenko, Peter Welinder, Pietro Perona, and Serge Belongie. Visual recognition with humans in the loop. In European Conference on Computer Vision (ECCV), 2010

work page 2010

[5] [5]

Amazon’s mechanical turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1):3–5, 2011

Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. Amazon’s mechanical turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1):3–5, 2011

work page 2011

[6] [6]

L. B. Chilton, G. Little, D. Edge, D. S. Weld, and J. A. Lan- day. Cascade: Crowdsourcing taxonomy creation. InConfer- ence on Human Factors in Computing Systems (CHI), 2013

work page 2013

[7] [7]

Crowley, Antreas Antoniou, and Amos J

Luke Nicholas Darlow, Elliot J. Crowley, Antreas Antoniou, and Amos J. Storkey. CINIC-10 is not imagenet or CIFAR-

work page

[8] [8]

Large-scale object classiﬁcation using la- bel relation graphs

Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classiﬁcation using la- bel relation graphs. In European Conference on Computer Vision (ECCV), 2014

work page 2014