Pith. sign in

REVIEW 4 major objections 5 minor 14 references

Distributions In, Distributions Out: The Case for Soft-Label Training

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Soft-label training, which uses full annotation distributions instead of majority-vote labels, matches or exceeds hard-label accuracy while producing models whose uncertainty closely tracks human disagreement.

desk verdict A clear, honest empirical paper on soft-label training; the headline KL number is off and the noise concern is real, but the core argument and most measurements hold up. read the letter →

arxiv 2511.14117 v2 pith:LR3WB3LU submitted 2025-11-18 cs.LG cs.AI

Agamdeep Singh , Ashish Tiwari , Hosein Hasanbeig , Priyanshu Gupta This is my paper · ORCID
classification cs.LGcs.AI
keywords soft-labeltrainingannotationdistributionsepistemicuncertaintymajorityvoteKLdivergenceentropycorrelationhumanambiguousclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that when annotators genuinely disagree about a sample, the distribution of their labels is the ground truth, not a noisy estimate to be collapsed into a single majority vote. Training on those full distributions, rather than on one-hot hard labels, preserves the ambiguity that humans actually experience. Across natural language inference, politeness classification, and image categorization, soft-label training matches or improves accuracy while reducing KL divergence to human annotations by 32% and strengthening the correlation between model and annotator entropy by 61%. If correct, this reframes annotation distributions from signals to be aggregated away into faithful targets that models should learn to reproduce. The practical stake is that AI systems deployed in human contexts should be uncertain in exactly the places where people are uncertain.

What carries the argument

The central mechanism is soft-label training with cross-entropy loss against the full human annotation distribution, compared against hard-label training with categorical cross-entropy on the majority-vote label. Two evaluation metrics carry the argument: KL divergence between model predictions and annotation distributions, which measures how closely the model reproduces human uncertainty, and the Pearson correlation between annotation entropy and model prediction entropy, which measures whether the model is uncertain on the same samples humans are. To isolate the effect of label type, both conditions use frozen pretrained embeddings (OpenAI Text Embeddings 3 Large for text, DINOv2 Small for

What would settle it

Construct a dataset with known true class-conditional distributions, then simulate annotator disagreement that is pure input-independent noise rather than genuine ambiguity; if soft-label training on these noisy distributions does not reduce KL divergence to the true distributions or does not maintain accuracy relative to hard-label training, the benefit would be specific to genuine disagreement rather than distributional targets generally.

Watch

Extended reading notes

Core claim

The central claim is that the annotation distribution itself is the correct learning target for data with genuine epistemic ambiguity. The authors demonstrate this by training identical models on either soft labels (the full distribution over annotator votes) or hard labels (the collapsed majority vote) across three datasets: ChaosNLI, POPQUORN, and a curated CIFAR-10H subset. Soft-label training achieves significantly lower KL divergence to human annotations on all three datasets (p < 0.001, average 32% improvement), produces predictions whose per-sample entropy correlates 61% more strongly with annotator entropy, and maintains or improves standard accuracy—most notably raising ChaosNLI acc

Load-bearing premise

The argument rests on persistent annotator disagreement being genuine epistemic ambiguity rather than annotator noise, task artifact, or demographic labeling bias, and on the observed annotation distributions being reliable enough to train against even with as few as 6.7 annotators per sample.

Editorial extensions

If this is right

  • Accuracy-only evaluation penalizes honest uncertainty: a model that confidently predicts the majority class on a 60-40 split may score higher accuracy but worse epistemic alignment, so evaluation should include distributional metrics on ambiguous tasks.
  • Investing in multiple annotations per sample yields measurable returns in model uncertainty alignment, with the effect strengthening as the number of annotators grows, from 6.7 per sample in POPQUORN to 100 in ChaosNLI.
  • Soft-label training resists overfitting: validation loss continues to improve for more epochs with distributional targets, suggesting a regularizing effect that could benefit small-data and transfer regimes.
  • On tasks with inherent subjectivity, annotation distributions should be treated as ground truth rather than as noisy measurements, which repositions multi-annotator data collection as a way to capture epistemic uncertainty rather than merely to estimate a single correct label.
  • The benefit generalizes across domains: the observed improvements hold in both NLP (natural language inference, politeness) and vision (image categorization), indicating a domain-independent effect of training on distributional targets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: A natural testable extension would be to synthetic datasets where true conditional distributions are known and annotation disagreement is generated independently of the input; the claim predicts soft-label training should recover the true distribution better than hard-label training, while pure labeling noise should not confer this advantage.
  • Inference: The entropy-correlation result suggests that soft-label models may also yield better downstream calibration and selective prediction, since a model that knows where it is uncertain can defer or abstain more reliably—though the paper does not test these downstream tasks directly.
  • Inference: The paper's framing implies a caution for demographic labeling bias: if annotator disagreement partly reflects systematic demographic differences rather than irreducible ambiguity, then treating the observed distribution as ground truth could entrench those biases, so the normative claim likely depends on disagreement being genuinely epistemic.
  • Inference: The few-annotator regime, such as POPQUORN's 6.7 annotations per sample, introduces sampling noise into the observed distributions; modeling that noise explicitly could either strengthen the argument (if noise is accounted for) or bound the benefit in low-annotation settings, a direction the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that when annotators disagree on genuinely ambiguous tasks, the full annotation distribution is the correct learning target and should be used as soft labels, rather than being collapsed to a majority-vote hard label. It compares soft-label training (cross-entropy to the empirical annotation distribution) with hard-label training on three datasets (ChaosNLI, POPQUORN, CIFAR-10H-Hard), using a frozen-embedding MLP head. It reports that soft-label training reduces test KL divergence to the annotation distribution, increases the correlation between model and annotator entropy, maintains or improves accuracy, and slows overfitting. The main evidence is in Tables 3 and 4; the paper frames these results as supporting a normative claim about epistemic alignment rather than as a purely practical trick.

Significance. If the empirical claims were robust, this would be a useful conceptual contribution: it would reframe soft-label training as the epistemically aligned objective for subjectively labeled data and would provide a concrete evaluation criterion (KL to annotation distributions). The paper's strengths are its use of three heterogeneous datasets, significance testing over 10 seeds, and clear reporting of per-dataset results. However, the central quantitative claim is not reproducible from the tables, and the key interpretative premise—that observed empirical distributions are ground truth rather than noisy estimates—is not actually tested by the experiments. The paper is therefore not ready for acceptance in its current form, but the issues are addressable with additional analysis and revised claims.

major comments (4)
  1. [Abstract; §4.2, Table 4] The headline '32% average KL reduction' is not supported by Table 4. The relative reductions are (0.367–0.319)/0.367 ≈ 13.1% for ChaosNLI, (0.596–0.406)/0.596 ≈ 31.9% for CIFAR-10H-Hard, and (0.430–0.361)/0.430 ≈ 16.0% for POPQUORN. The unweighted mean is about 20%, and pooling by hard-label KL gives about 22%; only CIFAR-10H-Hard reaches 32%. The abstract and §4.2 should be corrected, and per-dataset relative reductions should be reported instead of a single unsupported average.
  2. [§1.1, §2, Eq. (1)] The central claim that the annotation distribution is the ground truth, not a noisy estimate, is not tested. The targets used for both training and evaluation are per-sample empirical distributions p_hat. For POPQUORN, Table 1 gives 6.7 annotators per sample, so p_hat is a high-variance estimate of any population-level distribution. Furthermore, soft-label training minimizes cross-entropy to p_hat, which is D_KL(p_hat||q)+H(p_hat), so reporting lower test KL to the same p_hat is partly a restatement of the objective. The noise concern is supported by Table 4: entropy-correlation gains are largest on ChaosNLI (100 annotators, +119%), intermediate on CIFAR-10H-Hard (50, +40%), and smallest on POPQUORN (6.7, +24%)—exactly the pattern expected if the benefit partly reflects the reliability of the estimated targets. To support the normative claim, the authors should model annotation sampling
  3. [§3.3, Table 2] The text states that 'all other training procedures, model architectures, and optimization settings are identical across both conditions,' but Table 2 reports different learning rates, batch sizes, epochs, weight decays, and schedulers for soft and hard models on every dataset. This means the accuracy/KL comparisons and, in particular, the overfitting claim based on Figure 4 are not controlled for optimization configuration. The authors should provide a matched-hyperparameter ablation (or report results over the full grid) to show that the qualitative conclusions are not an artifact of selecting different configurations per condition.
  4. [§4.3, §5] The paper itself notes that the effect 'strengthens with more annotations' (ChaosNLI +119%) and is smallest on POPQUORN (+24%). This observation is treated as supporting evidence, but it directly undercuts the Section 1.1 claim that the observed distribution is the ground truth rather than a noisy estimate: if the distribution were the target concept, model quality should not depend so strongly on how many annotators were averaged. The authors need to address this confound explicitly, for example by showing that the entropy-correlation gain persists when the number of annotations per sample is held fixed.
minor comments (5)
  1. [Table 1] The entries '1000.59' and '500.24' appear to be missing a separator between the annotation count and the mean entropy (should be '100 0.59' and '50 0.24'). Please fix the formatting.
  2. [§4.3, Table 4] The text describes the entropy correlation as 'Pearson correlation coefficient ... averaged across all test samples,' but a Pearson correlation is typically computed over a set of points. Clarify whether Table 4 reports a single correlation over the whole test set, and if so how (or whether) it is averaged over seeds.
  3. [§3.2, §3.3] The use of OpenAI's proprietary Text Embeddings 3 Large as a frozen embedding model makes the exact features irreproducible. Releasing the extracted feature cache or using an open embedding model would improve reproducibility, which is especially important given the small datasets.
  4. [Figure 4] The validation-loss curves compare raw cross-entropy between soft and hard targets, but soft-label targets contain an additional entropy term H(p) that differs per dataset. The curves would be more informative if plotted as KL divergence to a common reference, or if the constant shift were acknowledged.
  5. [§4.2, Table 3] For CIFAR-10H-Hard and POPQUORN, 'statistically indistinguishable' is inferred from failure to reject at α=0.05; an equivalence test or confidence interval on the accuracy difference would be a more appropriate way to support the 'no degradation' claim.

Circularity Check

1 steps flagged · score 6.0 of 10

Paper's primary metric (KL to annotation distribution) is the soft-label training loss itself; the epistemic-alignment evidence is partly by construction, though accuracy parity and entropy correlation remain independent.

  1. self definitional [Section 2 (Eq. 1) and Section 3.3 (Training Procedure)]
    ""If annotation distributions represent ground truth, success should be measured by distributional similarity rather than accuracy on collapsed labels. We utilize Kullback-Leibler Divergence (KLD) ... A lower KLD indicates better epistemic alignment—the model's label distribution matches human label distribution." "For soft-label training, we use a cross-entropy loss where the full human annotation distribution serves as the target distribution.""

    Cross-entropy to soft target p is H(p,q) = D_KL(p||q) + H(p); H(p) is constant w.r.t. the model, so the soft-label training loss is the evaluation metric in Eq. (1) up to a constant. Reporting that soft-label models achieve lower KL to the annotation distribution is therefore a restatement of the objective they were optimized for, not an independent confirmation of the normative claim that the annotation distribution is the correct target. The hard-label baseline minimizes KL to one-hot targets, so its higher KL to the soft distribution is also built into the comparison. The independent accuracy-parity and entropy-correlation results do not remove this partial circularity in the headline 'epistemic alignment' evidence.

full rationale

The central empirical evidence for 'epistemic alignment' is lower KL divergence to human annotation distributions, but that metric is exactly (up to an additive constant) the soft-label training loss. Thus the headline 32% KL improvement is partly the training objective re-reported on test data, not an independent test of whether annotation distributions are ground truth. The hard-label baseline is trained to minimize KL to one-hot targets, so the gap is largely by construction. Accuracy parity is an independent and meaningful result; entropy correlation is not directly optimized and provides some independent support, though it is a natural consequence of KL matching. No load-bearing self-citations or imported uniqueness theorems are present. Separately, the finite-annotation issue (e.g., POPQUORN 6.7 annotators/sample) is a validity concern, not circularity: if empirical distributions are noisy estimates, minimizing KL to them may fit sampling noise. Also, Table 4's stated '32%' average KL reduction is not reproducible from the per-dataset reductions (13.1%, 31.9%, 16.0%; mean ~20%), a consistency issue outside the circularity assessment. Overall: partial circularity in the primary claim, with independent components, so score 6.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper's central claim is empirical and positional, not derived from first principles. It rests on the philosophical assumption that disagreement equals epistemic ambiguity, on finite annotation distributions being treated as ground truth, and on a frozen-feature setup that may not generalize. No new theoretical entities are introduced.

free parameters (2)
  • Per-condition hyperparameters (learning rate, batch size, epochs, weight decay, scheduler) = Table 2
    Selected by grid search on validation loss separately for soft and hard conditions; the comparison is between two tuned configurations rather than identical training settings.
  • CIFAR-10H-Hard sampling cap (samples per entropy bin) = 200
    Arbitrary cap used to build the 1,103-sample subset; affects absolute numbers though not the soft-vs-hard comparison.
assumptions (4)
  • domain assumption Persistent annotator disagreement reflects genuine epistemic ambiguity rather than annotator noise or bias
    Section 1.1; the whole 'distributions are the ground truth' argument depends on this interpretive premise, which the experiments do not test against noise models.
  • domain assumption Observed annotation distributions are reliable targets (adequate annotator counts)
    Table 1: POPQUORN has ~6.7 annotations/sample; with few annotators the empirical distribution is a noisy estimate, yet it is treated as ground truth.
  • domain assumption Frozen embeddings plus a small MLP head isolates the effect of label type
    Section 3.2; conclusions may not transfer to end-to-end training where the representation also adapts to soft targets.
  • domain assumption CIFAR-10H-Hard subset is a valid evaluation set
    Section 3.1; stratified entropy-bin sampling with a 200/bin cap creates a non-standard, non-released subset whose accuracy/KL values are not comparable to the full dataset.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distributions In, Distributions Out: The Case for Soft-Label Training." pith.science (2026). https://pith.science/paper/LR3WB3LU

@misc{pith2026251114117,
  author       = {Pith},
  title        = {Pith review of: Distributions In, Distributions Out: The Case for Soft-Label Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LR3WB3LU}},
  note         = {Machine review of arXiv:2511.14117}
}
read the original abstract

Supervised classifiers output a distribution over classes but are typically trained against a single label obtained by collapsing multiple annotators into a majority vote. On tasks where annotator disagreement reflects genuine ambiguity -- natural language inference, politeness, visually ambiguous categorization -- this collapse discards information and forces models to express uniform confidence on inputs where humans systematically disagree. We compare soft-label training, which uses the full annotation distribution as the target, against hard-label training across three datasets spanning vision and NLP (ChaosNLI, POPQUORN, CIFAR-10H). Soft-label training matches or exceeds hard-label accuracy on every dataset, reduces KL divergence to the annotator distribution by 32% on average (p < 10^-4), and produces predictions whose per-sample entropy correlates 61% more strongly with annotator entropy -- models trained on distributions are uncertain precisely where humans are. We argue these benefits follow from a basic observation: when annotators legitimately disagree, the annotation distribution is the correct learning target, not a noisy estimate of it.

Figures

Figures reproduced from arXiv: 2511.14117 by the authors.

Figure 1
Figure 1. Annotator entropy varies across samples. From near-consensus (a) to substantial disagree [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Both models predict the same majority class, yet their uncertainty profiles diverge sharply [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Entropy distributions across datasets. Entropy is normalized for number of classes. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Validation loss for soft-label (blue) and hard-label (orange) models. Loss magnitudes [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: On high-entropy samples, soft-label predictions (orange) mirror the spread of human [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 1 canonical work pages

  1. [1]

    Yixin Nie, Xiang Zhou, and Mohit Bansal. What can we learn from collective human opinions on natural language inference data? In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors,Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131–9143, Online, November 2020. As- sociation for Computational ...

    doi:10.18653/v1/2020.emnlp-main.734 2020

  2. [2]

    When do annotator demographics matter? measuring the in- fluence of annotator demographics with the POPQUORN dataset

    Jiaxin Pei and David Jurgens. When do annotator demographics matter? measuring the in- fluence of annotator demographics with the POPQUORN dataset. In Jakob Prange and An- nemarie Friedrich, editors,Proceedings of the 17th Linguistic Annotation Workshop (LAW- XVII), pages 252–265, Toronto, Canada, July 2023. Association for Computational Linguistics. doi:...

    doi:10.18653/v1/2023.law-1.25 2023

  3. [3]

    Battleday, Joshua C

    Ruairidh M. Battleday, Joshua C. Peterson, and Thomas L. Griffiths. Capturing human categorization of natural images by combining deep networks and cognitive models.Na- ture Communications, 11(1):5418, October 2020. doi: 10.1038/s41467-020-18946-z. URL https://doi.org/10.1038/s41467-020-18946-z

    doi:10.1038/s41467-020-18946-z 2020

  4. [4]

    Deep learning from crowds

    Filipe Rodrigues and Francisco Pereira. Deep learning from crowds. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  5. [5]

    Domain-weighted majority voting for crowdsourcing.IEEE Transactions on Neural Networks and Learning Systems, 30 (1):163–174, 2019

    Dapeng Tao, Jun Cheng, Zhengtao Yu, Kun Yue, and Lizhen Wang. Domain-weighted majority voting for crowdsourcing.IEEE Transactions on Neural Networks and Learning Systems, 30 (1):163–174, 2019. doi: 10.1109/TNNLS.2018.2836969

    arXiv 2019

  6. [6]

    Learning from multiple annotators with varying expertise.Machine learning, 95(3):291–327, 2014

    Yan Yan, R ´omer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy. Learning from multiple annotators with varying expertise.Machine learning, 95(3):291–327, 2014

    2014

  7. [7]

    Learning from multi-annotator data: A noise-aware classification framework.ACM Trans

    Xueying Zhan, Yaowei Wang, Yanghui Rao, and Qing Li. Learning from multi-annotator data: A noise-aware classification framework.ACM Trans. Inf. Syst., 37(2), February 2019. ISSN 1046-8188. doi: 10.1145/3309543. URLhttps://doi.org/10.1145/3309543

    work page doi:10.1145/3309543 2019

  8. [8]

    Distilling the knowledge in a neural network,

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network,

Show all 14 references
  1. [9]

    Label distribution learning, 2016

    Xin Geng. Label distribution learning, 2016. URLhttps://arxiv.org/abs/1408. 6027

    2016

  2. [10]

    Learn- ing from biased soft labels

    Hua Yuan, Yu Shi, Ning Xu, Xu Yang, Xin Geng, and Yong Rui. Learn- ing from biased soft labels. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 59566–59584. Curran Associates, I...

    2023

  3. [11]

    Learning with confidence: Training better classifiers from soft labels.Machine Learning, 114(238), 2025

    Sjoerd de Vries and Dirk Thierens. Learning with confidence: Training better classifiers from soft labels.Machine Learning, 114(238), 2025

    2025

  4. [12]

    Don’t waste a single annotation: improving single-label classifiers through soft labels

    Ben Wu, Yue Li, Yida Mu, Carolina Scarton, Kalina Bontcheva, and Xingyi Song. Don’t waste a single annotation: improving single-label classifiers through soft labels. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Findings of the Association for Computational Linguistic...

    2023 doi

  5. [13]

    A theoretical analysis of soft-label vs hard-label training in neural networks

    Saptarshi Mandal, Xiaojun Lin, and Rayadurgam Srikant. A theoretical analysis of soft-label vs hard-label training in neural networks. InProceedings of the 7th Annual Learning for Dynamics & Control Conference, PMLR, number 283, pages 1078–1089, 2025. 7 A Appendix Figure 5 pic...

    2025

  6. [2015]

    URLhttps://arxiv.org/abs/1503.02531

    arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.