{"id":"3e299a58-85e1-4b3e-add7-a7ca01e8e768","arxiv_id":"1908.07086","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Training image classifiers on human label distributions rather than one-hot labels improves out-of-distribution generalization and adversarial robustness, as demonstrated with the new CIFAR10H dataset.","lead":"The authors introduce CIFAR10H, a dataset of full human label distributions for the CIFAR10 test set, and train neural networks on these distributions instead of one-hot labels. They report that this training improves out-of-distribution generalization and reduces vulnerability to adversarial attacks, with no defensive training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract claims adversarial robustness, but the paper's own PGD results show accuracy collapsing to 0%/1% under both label schemes; the measurable advantage is lower crossentropy, so the robustness claim is overstated unless restricted to FGSM loss/calibration.","rationale":"The reader's conditional verdict is appropriate, but the most load-bearing concern is not the Section 3 proxy assumption about p_hum approximating p(y|x). That assumption motivates the method but is hard to test directly. A sharper, internally verifiable problem is that the paper's adversarial-robustness claim exceeds its own evidence: PGD drives accuracy to 0%/1% in both conditions, and FGSM accuracy gains are mixed, unreplicated with uncertainty, and may reflect confidence calibration rather than attack resistance. The generalization results may still support a conditional accept, but the abstract should be qualified. I therefore keep the verdict unchanged while flagging this as a required revision.","tokens_in":14221,"tokens_out":9760,"duration_ms":109105,"concrete_test":"From the released models or a rerun, produce per-fold FGSM and PGD accuracy tables (not only loss) with 95% confidence intervals over the 10 folds, and paired tests comparing hard- and soft-label fine-tuning. If PGD accuracy is near 0% for both conditions and FGSM accuracy gains are not significant at the fold level, the adversarial part of the central claim must be narrowed to FGSM loss/calibration, and the 'five of eight' claim in Table 2 should carry explicit uncertainty estimates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract and contribution bullet) that human-label training 'confers robustness to adversarial attacks' is not supported by the paper's own stronger-attack results. Section 7 reports that under PGD, 'accuracy was driven to 0% for each network when trained on standard labels, and 1% for each network with human labels'; the plotted advantage (Figure 4) is only that crossentropy rises more slowly. That is a calibration/confidence effect, not resistance to misclassification. Under FGSM (Table 2), accuracy improves for only 5 of 8 architectures, degrades for PyramidNet and ResNext, and ties for Shake-Shake, with no error bars or significance tests. Since soft-label-trained models are generally less confident on errors (Figure 3a), lower post-attack crossentropy may partly reflect this baseline confidence shift rather than true robustness. The abstract's unqualified 'confers robustness to adversarial attacks' is therefore an overstatement of a mixed, loss-based result. This is an internal-evidence concern, independent of whether p_hum is the correct target distribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CIFAR10H, a new dataset of full human label distributions for the 10,000-image CIFAR10 test set, collected from 511,400 crowdsourced judgments. The authors train eight CNN architectures on these soft labels and compare them against hard-label controls and several soft-label baselines (mixup, class-level soft labels, and sampled hard labels). They report that training on human soft labels improves accuracy and crossentropy on increasingly out-of-distribution test sets (CIFAR10.1, CINIC10, ImageNet-Far), improves second-best accuracy, and yields better fits to human uncertainty. The paper also presents FGSM and PGD adversarial experiments and claims that human-label training confers adversarial robustness.","tokens_in":14363,"tokens_out":4606,"duration_ms":43297,"significance":"If the generalization results hold, CIFAR10H is a valuable community resource, and the demonstration that image-level human uncertainty can improve out-of-distribution generalization is a meaningful empirical contribution. The experimental design is solid in several respects: eight architectures are trained with 10-fold cross-validation, multiple external datasets are used, and competitive baselines such as mixup, class-level soft labels, and sampled hard labels are included. The paper would be strengthened by error bars and statistical tests, and the adversarial robustness claim is overstated relative to the evidence. Nonetheless, the core dataset and the generalization findings are useful and likely of interest to the computer vision community.","major_comments":[{"comment":"The abstract and the fourth contribution bullet state that training with CIFAR10H 'confers robustness to adversarial attacks,' but the evidence in Section 7 supports a narrower claim. Under PGD, the paper reports that accuracy was driven to 0% for standard-label networks and 1% for human-label networks; the plotted advantage is only that crossentropy grows more slowly (Figure 4). In Table 2, under FGSM, accuracy improves for five of eight architectures, degrades for PyramidNet and ResNext, and is tied for Shake-Shake, and no error bars or significance tests are given. Because soft-label models are less confident on errors (Figure 3a), part of the crossentropy gap after attack may reflect this baseline confidence shift rather than increased robustness to misclassification. Please restrict the robustness claim to the measured crossentropy/loss behavior under these attacks, or add accuracy-based results under stronger attacks with proper statistics.","section":"Section 7, Table 2, Figure 4"},{"comment":"The central generalization claim that accuracy improves on all generalization datasets and for every individual CNN is presented without variance information. The reported improvements are small (about 1 percentage point on CIFAR10 and 2 percentage points on ImageNet-Far), and the statement that the pattern replicated across folds is asserted but not shown. Please provide error bars or confidence intervals across the 10 folds for the main results, and ideally a paired statistical test across architectures, so that the reliability of the observed gains can be assessed.","section":"Section 5.2, Figure 2"},{"comment":"The theoretical motivation in Section 3 rests on the assumption that the human label distribution p_hum(y|x) is a good estimator of the true conditional label distribution p(y|x). This assumption is not independently verified, and the benchmark for 'fit to humans' in Section 6 uses the same CIFAR10H labels, so high fit is partly by construction. The comparisons against mixup and class-level soft labels are informative, but an explicit test against a generic soft-label regularizer with matched label entropy (e.g., label smoothing) would help establish that the benefits come specifically from human-like uncertainty rather than from any form of softened targets.","section":"Section 3 and Section 6"}],"minor_comments":[{"comment":"There are small presentation errors: 'evalulate' should be 'evaluate' in Section 4.1, and the number of participants removed by attention checks is given as ' 14 total' with a stray leading space; please clarify the count.","section":"Section 4.2"},{"comment":"The caption lists columns from left to right but the column headers use abbreviations 'c10H', 'c10', 'v4', and 'v6' that are not defined in the table or its caption. Please define these abbreviations and state explicitly that lower crossentropy is better.","section":"Table 1 caption"},{"comment":"The text says that l-infinity bounds of 4 to 8 in increments of 1 were explored but that all results are reported with a bound of 4 for brevity. Please show at least a summary of the other bounds or justify why they were omitted.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The dataset and generalization comparisons are likely to be a useful contribution to the community. The main issue is that the adversarial robustness claim needs to be substantially scaled back or re-evaluated with accuracy-based metrics under stronger attacks; once that is addressed, the paper could be publishable. I would also encourage the editor to request error bars and significance tests for the headline generalization results, since the reported gains are small."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real news here is the dataset: CIFAR10H, full human label distributions for all 10,000 CIFAR10 test images, built from half a million judgments. That is a genuinely useful resource, and the paper deserves credit for releasing it. The training story is also credible: across eight architectures, training on these soft labels improves accuracy and crossentropy on CIFAR10.1 and on their constructed ImageNet-Far shift, and the gains grow with shift. The comparison against mixup, class-level soft labels, and sampled hard labels is the right kind of control, and the result that image-level human uncertainty beats the alternatives is convincing. I buy the central generalization claim.\n\nWhere I part company is the adversarial robustness headline. The abstract says the method 'confers robustness to adversarial attacks,' but the paper's own PGD numbers show accuracy collapsing to 0% and 1% under both label schemes. What actually improves is crossentropy during the attack, which is a calibration/confidence effect, not resistance to misclassification. The FGSM table is also mixed: accuracy improves for five of eight architectures and degrades for two, with no error bars or significance tests. So the robustness claim is overstated. It should be restricted to 'lower crossentropy under FGSM and slower loss growth under PGD,' not 'robustness.' This matters because the abstract and contribution bullets will be read as claiming defensive robustness, and the evidence does not support that.\n\nOther soft spots are minor. There are no error bars in the main tables. The dataset link is missing from this version. And the underlying assumption that human perceptual uncertainty approximates the natural p(y|x) is a modeling choice that is not independently validated. But the empirical gains do not hinge on that assumption being literally true; the method works whether or not humans are the 'gold standard.'.\n\nThe paper is worth a serious referee. The central generalization result is solid and the dataset will be reused regardless of the robustness quibble. I would send it to review, and I would push the authors to fix the robustness language, add variance estimates, and release the data before publication.","headline":"A solid empirical paper introducing CIFAR10H and showing human soft labels improve OOD generalization, but the adversarial robustness claim is overstated relative to the PGD results.","tokens_in":14945,"tokens_out":546,"would_cite":true,"duration_ms":6853,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training classifiers on full human label distributions improves out-of-distribution generalization and attack resistance.","keywords":["human uncertainty","soft labels","CIFAR10H","distributional shift","adversarial robustness","generalization","label distributions","second-best accuracy"],"falsifier":"Train the same eight architectures on CIFAR10H soft labels, attack them with PGD at an $\\ell_\\infty$ bound of 8 over more iterations than the paper reports, and compare cross-entropy trajectories with hard-label controls; if the soft-label advantage disappears or reverses, the robustness claim is falsified. A second test would be to collect human soft labels for a different base dataset and check whether the distributional-shift accuracy gains replicate; if they do not, the effect is specific to CIFAR10H rather than to human uncertainty as a training signal.","tokens_in":13976,"feed_emoji":"🎯","tokens_out":10169,"duration_ms":82943,"temperature":0.7,"pith_summary":"This paper introduces CIFAR10H, a dataset that augments the CIFAR10 test set with full human label distributions: roughly 50 crowdsourced judgments per image, 511,400 in total. The authors' central claim is that training classifiers on these soft labels, instead of one-hot ground-truth labels, teaches them the similarity structure that human perception uses, and that this transfers as better generalization under distributional shift and stronger resistance to adversarial attacks. Across eight CNN architectures, soft-label training improved accuracy on every out-of-distribution test set considered, with the gains growing as the test data moved further from the training distribution. If the claim stands, human label uncertainty is a practical and comparatively cheap training signal for robustness, one that also gives a new benchmark for whether a model's errors are human-like.","feed_headline":"Human uncertainty improves model generalization and attack resistance","feed_subtitle":"CIFAR10H's 500k human judgments help eight CNNs generalize further and resist adversarial attacks.","key_machinery":"The load-bearing object is the measured human label distribution $p_{\\mathrm{hum}}(y|x)$ for each image, collected as a vector of ten category probabilities from the CIFAR10H judgments. The argument is carried by replacing the one-hot label in the empirical risk with this distribution, so that for a network output $p_{\\theta}(y|x)$ the training loss is the cross-entropy $-\\frac{1}{m}\\sum_{j=1}^{m}\\sum_{c} p_{\\mathrm{hum}}(y_j=c|x_j)\\,\\log p_{\\theta}(y_j=c|x_j)$. This is exactly the expected negative log-likelihood under the human distribution, used as a proxy for the true conditional label distribution $p(y|x)$. The same human distribution also defines the evaluation target, so CIFAR10H doubles as a benchmark for whether a model's uncertainty is human-like.","core_discovery":"The central discovery, stated on the paper's own terms, is that substituting the full human label distribution $p_{\\mathrm{hum}}(y|x)$ for a single modal label in the training objective changes what a network learns: its predictions come to resemble human patterns of uncertainty rather than merely matching consensus labels. Trained this way, all eight architectures improved accuracy relative to hard-label controls on CIFAR10, CIFAR10.1 v4 and v6, CINIC10, and ImageNet-Far, and the margin grew with distributional shift, from about one percentage point on CIFAR10 to about two on ImageNet-Far. Cross-entropy to ground truth fell by roughly a third on CIFAR10 and by 38% on ImageNet-Far, and second-best accuracy rose by about five points on average. Under FGSM attacks, cross-entropy was roughly halved compared with hard-label fine-tuning, and accuracy was higher for five of eight architectures; under PGD, loss rose more slowly with each iteration. The paper also reports that human soft labels outperform alternative ways of producing soft targets, including class-level penalties, an ensemble knowledge-distillation baseline, mixup, and labels sampled from the human distribution.","pith_inferences":["Inference: if the mechanism generalizes, the concentration of human uncertainty in roughly 30% of images means targeted collection of high-uncertainty labels could capture most of the robustness benefit at a fraction of the labeling cost.","Inference: since the largest gains appear under distributional shift, human soft labels collected on a source domain could serve as a domain-adaptation signal before target-domain labels exist.","Inference: evaluating models with human-label cross-entropy or second-best accuracy could become a standard complement to top-1 accuracy, revealing robustness differences that accuracy alone hides.","Inference: combining human soft labels with explicit adversarial training or with larger-scale distillation is an untested extension suggested by the paper's mechanism, and the two sources of robustness might compound."],"forward_implications":["Soft-label training on CIFAR10H improves accuracy and lowers cross-entropy on all tested datasets, with larger gains as the test distribution moves further from CIFAR10.","The same training makes model errors more human-like: models are less confident when wrong, have better second-best accuracy, and fit human label distributions more closely.","Human soft targets beat other soft-label methods, including class-level penalties, ensemble knowledge distillation, mixup, and sampled hard labels, both at matching human uncertainty and at out-of-distribution generalization.","Fine-tuning on human labels roughly halves cross-entropy under FGSM attacks and slows loss growth under PGD, without any explicit adversarial defense.","The benefit arises from the human information itself, not from the mere fact of training with distributed labels; category-level and network-generated soft labels do not reproduce it."],"supporting_citations":[{"why":"Supplies CIFAR10, the base dataset whose test set CIFAR10H re-labels with full human label distributions.","marker":"[28]"},{"why":"Provides the CIFAR10.1 v4 and v6 datasets used to test generalization under near-distribution shift.","marker":"[39]"},{"why":"Defines the FGSM attack used to measure adversarial robustness of hard-label and soft-label trained models.","marker":"[29]"},{"why":"Defines the PGD attack whose loss trajectories are plotted to compare robustness across training conditions.","marker":"[30]"},{"why":"Supplies the knowledge-distillation baseline and the rationale that soft targets carry class-similarity structure.","marker":"[19]"},{"why":"Supplies the mixup soft-label baseline that human soft labels are compared against.","marker":"[51]"},{"why":"Motivates the expectation that distillation-style training with soft targets can confer adversarial robustness.","marker":"[37]"},{"why":"Supplies the CINIC10 out-of-sample generalization dataset used in the distribution-shift evaluation.","marker":"[7]"},{"why":"Supplies the Shake-Shake architecture, the strongest of the eight models tested.","marker":"[11]"}],"fun_headline_variants":["Human labels beat hard labels on shifted and attacked data","Using human uncertainty gives models stronger adversarial defense","Training on full label distributions boosts robustness to attacks","Human uncertainty improves accuracy on shifted test sets","Modelling human perceptual uncertainty improves generalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the way humans distribute probability over the ten classes for an image is a good estimate of the true conditional label distribution $p(y|x)$; if human perceptual uncertainty points to the wrong categories, or is irrelevant to the classification task, the reported gains would be specific to mimicking human choices rather than general robustness.","fun_headline_variants_meta":{"raw":{"variants":["Human labels beat hard labels on shifted and attacked data","Using human uncertainty gives models stronger adversarial defense","Training on full label distributions boosts robustness to attacks","Human uncertainty improves accuracy on shifted test sets","Modelling human perceptual uncertainty improves generalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000441,"raw_usage":{"total_tokens":2210,"prompt_tokens":894,"completion_tokens":1316,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":1248}},"tokens_in":510,"tokens_out":1316,"duration_ms":10221,"temperature":1.0,"reasoning_tokens":1248,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:26:38.121609+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same eight architectures on CIFAR10H soft labels, attack them with PGD at an $\\ell_\\infty$ bound of 8 over more iterations than the paper reports, and compare cross-entropy trajectories with hard-label controls; if the soft-label advantage disappears or reverses, the robustness claim is falsified. A second test would be to collect human soft labels for a different base dataset and check whether the distributional-shift accuracy gains replicate; if they do not, the effect is specific to CIFAR10H rather than to human uncertainty as a training signal.","supporting_citations":[{"cited_title":"Krishna, Kenji Hata, Stephanie Chen, Joshua Kravitz, David A","cited_arxiv_id":null,"evidence_quote":"Supplies CIFAR10, the base dataset whose test set CIFAR10H re-labels with full human label distributions."},{"cited_title":"Automatic differentiation in PyTorch","cited_arxiv_id":null,"evidence_quote":"Provides the CIFAR10.1 v4 and v6 datasets used to test generalization under near-distribution shift."},{"cited_title":"Ad- versarial examples in the physical world","cited_arxiv_id":null,"evidence_quote":"Defines the PGD attack whose loss trajectories are plotted to compare robustness across training conditions."},{"cited_title":"Wide residual net- works","cited_arxiv_id":null,"evidence_quote":"Supplies the mixup soft-label baseline that human soft labels are compared against."},{"cited_title":"Learning classification models with soft-label information","cited_arxiv_id":null,"evidence_quote":"Motivates the expectation that distillation-style training with soft targets can confer adversarial robustness."},{"cited_title":"Crowley, Antreas Antoniou, and Amos J","cited_arxiv_id":null,"evidence_quote":"Supplies the CINIC10 out-of-sample generalization dataset used in the distribution-shift evaluation."},{"cited_title":"Frome, G.S","cited_arxiv_id":null,"evidence_quote":"Supplies the Shake-Shake architecture, the strongest of the eight models tested."}],"review_version":1}