Pith. sign in

REVIEW 3 major objections 6 minor 93 references

Calibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinement

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SiDyP recovers true labels from LLM-annotated data by retrieving candidate labels from text-embedding neighborhoods and refining them with a simplex diffusion model, raising fine-tuned BERT accuracy by an average of 7.21% in zero-shot and…

desk verdict A practically useful denoising method for LLM-generated noisy labels, with a plausible mechanism, but the headline gains are weakened by oracle noise-rate setting and unreported baseline tuning budgets. read the letter →

arxiv 2505.19675 v2 pith:FTUMV32X submitted 2025-05-26 cs.CL cs.AI

Liqin Ye , Agam Shah , Chao Zhang , Sudheer Chava This is my paper
classification cs.CLcs.AI
keywords largelanguagemodelsnoisylabelsdiffusionmodelBERTtextclassificationweaksupervisionlabelcandidateretrievalsimplex
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that label noise produced by LLM annotators is qualitatively different from synthetic noise—it is context-dependent and class-correlated—and that existing denoising methods built for synthetic or crowd noise leave accuracy on the table. It proposes SiDyP, which first retrieves a short list of plausible true labels for each sample from the labels of its text-embedding neighbors, then trains a simplex diffusion model to refine those candidates into a final label, iteratively updating the candidates as the diffusion model learns. Across four text classification tasks and five LLM annotators, SiDyP improves the accuracy of a BERT classifier fine-tuned on LLM-generated noisy labels by an average of 7.21% in the zero-shot setting and 7.30% in the few-shot setting, and it also outperforms baselines on synthetic and real-world noise. If this holds, LLM-produced training data becomes a more dependable substitute for human annotation, since a post-hoc diffusion step can repair much of the labeling error.

What carries the argument

The machinery is (1) a candidate-retrieval step that splits the training set into certain and uncertain samples based on the KNN label distribution in BERT embedding space, keeping labels with probability above $\lambda$ as certain and the top-two candidates above $\gamma$ as a reduced uncertain list; and (2) a simplex diffusion model (TESS-style) that maps one-hot labels to a $k$-logit simplex, forward-diffuses with Gaussian noise, and learns a reverse denoising conditioned on the noisy-label simplex and input embedding, with candidate distillation updating the candidate weights during training. Together, these two components turn noisy LLM labels into a calibrated posterior over true labels.

What would settle it

Run SiDyP on a classification task whose BERT embeddings do not separate classes (for example, a fine-grained sentiment or stance task), with an LLM annotator, and compare against a plain fine-tuned BERT classifier; if SiDyP fails to improve, or the candidate lists in Algorithm 1 contain the true label on fewer than half of the noisy samples, the embeddings-based prior that feeds the diffusion model is the point of failure.

Watch

Extended reading notes

Core claim

SiDyP treats the true-label posterior as $p(y|x)=\sum_{\tilde{y}} p(\tilde{y}|x)\,p(y|\tilde{y},x)$, where the noisy-label prior is estimated by fine-tuning a pre-trained language classifier (BERT) on the noisy dataset and the correction posterior is approximated by a simplex diffusion model conditioned on the noisy label and input. The key move is to replace the fixed noisy label with a dynamic list of candidate true labels obtained from the neighborhood label distribution in embedding space, separating certain from uncertain samples via thresholds, and then to distill the candidates during diffusion training by multinomial sampling weighted by the diffusion model's own predictions. The paper reports that this calibrated correction improves BERT fine-tuned on both zero-shot and few-shot LLM-generated noisy labels by an average of 7.21% and 7.30%, and that it outperforms five noisy-label learning baselines on all tested LLMs and tasks.

Load-bearing premise

Texts belonging to the same class cluster in BERT embedding space, so the nearest-neighbor label distribution over the estimated-clean subset recovers the true label distribution for noisy samples.

Editorial extensions

If this is right

  • If SiDyP's gains hold, LLM-generated training labels become usable for fine-tuning classifiers without costly human re-annotation, because the diffusion step repairs the majority of the labeling error.
  • Since SiDyP also improves on synthetic and real-world noise, the candidate-retrieval-plus-diffusion recipe may offer a unified denoiser for label noise beyond the LLM setting.
  • The 7.21% and 7.30% average gains imply that previous noisy-label baselines leave significant accuracy on the table specifically for LLM noise, so evaluations of LLM-based annotation should include a denoising stage.
  • SiDyP's gains are largest on the noisiest task (SemEval), suggesting that iterative refinement matters more as label quality degrades.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The assumption that same-class texts cluster in embedding space is testable per task; on tasks with weak semantic separation, the candidate retrieval could feed the diffusion model a wrong prior and amplify errors, so SiDyP's gains may be bounded by embedding quality.
  • Inference: Because SiDyP uses BERT embeddings and training dynamics, it inherits whatever biases the PLC encodes; one extension would be to swap in a different backbone or embedding source and measure whether the 7% gain persists.
  • Inference: The candidate distillation loop resembles a self-training scheme with soft labels; a cleaner theoretical account of when the multinomial sampling converges could motivate applying the same trick to other generative label models.
  • Inference: The paper's ablation shows the Gaussian diffusion falls behind simplex diffusion; an untested hypothesis is that the simplex constraint is what matters, not the diffusion schedule, which could be checked by ablating the noise schedule alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SiDyP, a two-stage framework for fine-tuning a BERT classifier on datasets whose labels are generated by LLMs and therefore noisy. In Stage I a PLC is fine-tuned on the noisy labels; training dynamics are used to split the data into estimated-clean and estimated-noisy subsets. In Stage II, a KNN classifier over the estimated-clean subset produces a weighted list of candidate true labels for each sample, and a simplex diffusion model conditioned on the noisy label and input embedding iteratively refines these candidate weights. The authors report average improvements of 7.21% (zero-shot) and 7.30% (few-shot) over the fine-tuned PLC baseline across four NLP tasks and five LLMs, and claim to outperform five existing noisy-label baselines in all settings.

Significance. If the reported gains are real and the comparison is fair, SiDyP would be a practically useful method for reducing the impact of LLM annotation errors, which is an increasingly relevant problem as LLM-generated datasets proliferate. The paper provides a broad empirical evaluation across multiple tasks, several LLMs, three noise families, and an ablation of the main components, and it releases code. However, the central claim is empirical, so the validity of the comparison and the absence of oracle information in the hyperparameters are decisive. The paper does not supply machine-checked proofs or parameter-free derivations; its contribution is algorithmic and experimental. The significance is therefore conditional on the tuning and the degree to which the self-refining candidate mechanism is genuinely estimating a posterior rather than reinforcing its own prior.

major comments (3)
  1. [Appendix D, Tables 6-8] The hyperparameter σ is set to the ground-truth noise rate. The values labeled "Noise Ratio (Original)" in Table 7 are actually clean-label accuracies (e.g., NumClaim zero-shot 91.69% clean; SemEval zero-shot 50.96% clean), and the σ values in Table 6 (0.1 for NumClaim, 0.5 for SemEval, 0.3 for TREC) equal one minus these clean ratios up to rounding. Since Algorithm 1 uses σ to split the training set into clean and noisy parts, and that split determines the KNN candidates and all downstream refinement, setting σ from the true clean labels is an oracle choice. The statement in Appendix D that "we use the same estimated error rate in all other baselines" does not address the per-dataset grid search over λ, γ, α, β, K, train/inference timesteps, and learning rate, for which no equivalent tuning budget is reported for the five baselines. The reported 7.21%/7.30% average gains may therefore reflect a tuning advantage rather than the SiDyP mechanism. Please report results with σ estimated from the noisy data alone, or provide an ablation over σ values that deliberately miss the true noise rate, together with a comparable hyperparameter budget for the baselines.
  2. [Section 3.2, Algorithm 2] The candidate distillation loop is self-referential. In lines 8-11 of Algorithm 2, the generative model's own predictions increase the weight of any candidate that the model already predicts, and in line 12 the model's training label for an uncertain sample is sampled from those same updated weights. Since the candidate lists themselves come from a KNN classifier trained on a PLC fine-tuned on the noisy labels, no component of the pipeline provides an independent estimate of p(y|y~,x). The self-reinforcing update can amplify KNN errors. The ablation in Table 5 compares FP+Sim-Diff with DP+Sim-Diff, but FP uses a single fixed label rather than a fixed candidate distribution, so it does not isolate the effect of self-updating weights. Please add an ablation in which the candidate weights are frozen after Algorithm 1, or one in which the diffusion model's predictions are not used to update the weights, to demonstrate that the iterative refinement is the actual source of the gain.
  3. [Section 3.1, Figure 3] The load-bearing assumption that texts belonging to the same class cluster in BERT embedding space receives only an aggregate test in Figure 3. The figure reports candidate accuracy over all datasets combined and over threshold ranges, but the paper does not report per-class candidate accuracy or sensitivity to K. For SemEval, where the nine relation classes overlap semantically and the noise ratio is near 50%, poor class separation would make the KNN candidate distribution positively misleading, and the diffusion model trained on those candidates would encode the wrong prior. Please report candidate accuracy and final classifier accuracy as a function of K and of embedding choice (e.g., different PLC stages), or at least provide per-dataset/per-class breakdowns, to show that the method does not rely on a favorable embedding geometry.
minor comments (6)
  1. [Table 7] The column header "Noise Ratio (Original)" is misleading; the reported values (e.g., 91.69 for NumClaim zero-shot) are clean-label accuracies. Rename the column to "Clean Ratio" or "Accuracy" to avoid confusion.
  2. [Equation (5)] Equation (5) has malformed notation: "L = L_t,q(...)" appears to be a typo or missing a function definition. Use standard cross-entropy notation and define all terms.
  3. [Algorithm 1] The input M_train (the noisy marker from training dynamics) is not defined in Section 3; it is only described in Appendix C. Define it where Algorithm 1 is first introduced.
  4. [Abstract and Section 5.3] The abstract's claim of "7.21% and 7.30% average increase" should specify the baseline (the fine-tuned PLC, presumably) and should be reconciled with the varying percentage gains reported in the text (e.g., 2.05%, 3.7%, 4.47%).
  5. [References] References [8] and [9] are the same arXiv paper (Label-Retrieval-Augmented Diffusion Models); please cite it once.
  6. [Figure 3] The caption text "The arrows (to the upper-bottom corner)" is confusing; clarify which arrows point to the fixed-prior and dynamic-prior accuracy.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central claim is an empirical comparison evaluated on clean held-out test sets.

full rationale

The paper's central claim is an empirical accuracy comparison on clean held-out test sets (Tables 1, 3, 4), so the success metric is external to the method's construction. The decomposition p(y|x) = sum_y~ p(y~|x)p(y|y~,x) is a standard identity, and neither the KNN candidate retrieval (Algorithm 1) nor the simplex diffusion training (Algorithm 2) defines the final test accuracy in terms of its own inputs. The candidate-distillation loop in Algorithm 2 is a self-training/expectation-maximization-style mechanism, not a logical reduction: the model's predictions influence candidate weights, but the final evaluation is against independent clean labels. Citations to DyGen [68], a same-group prior work, are used for standard components (training dynamics, co-regularization) and are not the sole support for the headline result. One caveat outside circularity: sigma in Table 6 closely matches the complement of the Table 7 'Noise Ratio' values (which are actually clean-label accuracies), suggesting sigma was set with access to clean labels; this is a potential evaluation-fairness or data-leakage issue, not a circular derivation.

Assumptions & free parameters 11 free parameters · 5 assumptions · 0 invented entities

The method rests on several tuned hyperparameters and on domain assumptions borrowed from prior work (DyGen, neighbor consistency). No new physical or conceptual entities are introduced. The main unstated cost is that sigma is taken from the true noise ratio, which is not available in a genuine noisy-label setting.

free parameters (11)
  • sigma (estimated error rate) = 0.05 to 0.5 per dataset (Table 6)
    Used in Algorithm 1 to split clean/noisy via training dynamics; set to the true noise ratio computed from ground-truth labels, which is normally unavailable.
  • lambda (certainty threshold) = 0.7 to 1.0 grid
    Grid-searched per dataset to decide whether a KNN candidate is treated as deterministic.
  • gamma (dominance threshold) = 0.4 to 0.8 grid
    Grid-searched to decide if the top two candidates dominate the candidate list.
  • alpha (warmup epochs) = 1 to 6 grid
    Grid-searched; number of epochs the diffusion model trains only on deterministic candidates before distillation.
  • beta (evaluation iterations) = 2 to 8 grid
    Grid-searched; how many times the model evaluates uncertain samples to update candidate weights.
  • K (KNN neighbors) = 10 to 30
    Grid-searched number of neighbors for candidate retrieval.
  • train timesteps = 200 to 800 per dataset
    Grid-searched diffusion training steps.
  • inference timesteps = 10 to 100
    Grid-searched diffusion sampling steps.
  • learning rate (diffusion) = 1e-4 to 1e-3
    Grid-searched.
  • k (logit simplex magnitude) = not stated
    Hyperparameter in Eq. 1 mapping one-hot labels to +/-k logit simplex; value not reported in the paper.
  • number of branches m = 3
    Number of co-regularized model branches in Stage I and II; chosen without sensitivity analysis.
assumptions (5)
  • standard math Conditional probability decomposition p(y|x)=sum_{y~} p(y~|x) p(y|y~,x)
    Used in Section 2 to factor the true label posterior; standard law of total probability.
  • domain assumption Training dynamics (distance to assigned label) separate clean from noisy samples
    Adopted from DyGen [68]; assumed to hold for BERT fine-tuned on LLM-noisy labels; basis for splitting into D_clean and D_noisy in Algorithm 1.
  • domain assumption Same-class texts cluster in embedding space so neighbor label distribution reveals true labels
    Stated in Section 3; supports KNN candidate retrieval. If embeddings are poor for the task, the candidate lists are unreliable.
  • domain assumption A continuous diffusion model conditioned on noisy label and input can approximate p(y|y~,x)
    The paper adopts simplex diffusion to model the posterior without identifiability or consistency guarantees; empirical validation only.
  • domain assumption Co-regularization across branches improves robustness
    Appendix C; KL-consensus between M branches assumed to reduce error propagation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Calibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinement." pith.science (2026). https://pith.science/paper/FTUMV32X

@misc{pith2026250519675,
  author       = {Pith},
  title        = {Pith review of: Calibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FTUMV32X}},
  note         = {Machine review of arXiv:2505.19675}
}
read the original abstract

The traditional process of creating labeled datasets is labor-intensive and expensive. Recent breakthroughs in open-source large language models (LLMs) have opened up a new avenue in generating labeled datasets automatically for various natural language processing (NLP) tasks, providing an alternative to such an expensive annotation process. However, the reliability of such auto-generated labels remains a significant concern due to inherent inaccuracies. When learning from noisy labels, the model's generalization is likely to be harmed as it is prone to overfit to those label noises. While previous studies in learning from noisy labels mainly focus on synthetic noise and real-world noise, LLM-generated label noise receives less attention. In this paper, we propose SiDyP: Simplex Label Diffusion with Dynamic Prior to calibrate the classifier's prediction, thus enhancing its robustness towards LLM-generated noisy labels. SiDyP retrieves potential true label candidates by neighborhood label distribution in text embedding space and iteratively refines noisy candidates using a simplex diffusion model. Our framework can increase the performance of the BERT classifier fine-tuned on both zero-shot and few-shot LLM-generated noisy label datasets by an average of 7.21% and 7.30% respectively. We demonstrate the effectiveness of SiDyP by conducting extensive benchmarking for different LLMs over a variety of NLP tasks. Our code is available on Github.

Figures

Figures reproduced from arXiv: 2505.19675 by the authors.

Figure 1
Figure 1. The SiDyP framework, containing (1) pre-trained classifier fine-tuning; (2) dynamic label candidates retrieval and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Confusion Matrix of LLM-generated label noise, synthetic noise, and real-world noise on SemEval dataset. We prompt [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. The percentage of uncertain labels being corrected [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Confusion Matrix of LLM-generated label noise, synthetic noise, real-world noise on SemEval dataset. We include [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

93 extracted references · 38 canonical work pages

  1. [1]

    O’Connor, and Kevin McGuin- ness

    Eric Arazo, Diego Ortego, Paul Albert, Noel E. O’Connor, and Kevin McGuin- ness. 2019. Unsupervised Label Noise Modeling and Loss Correction. arXiv:1904.11238 [cs.CV] https://arxiv.org/abs/1904.11238

    arXiv 2019

  2. [2]

    Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien

    Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. 2017. A Closer Look at Memorization in Deep Networks. arXiv:1706.05394 [stat.ML] https://arxiv.org/abs/1706.05394

    arXiv 2017

  3. [3]

    HeeSun Bae, Seungjae Shin, Byeonghu Na, JoonHo Jang, Kyungwoo Song, and Il- Chul Moon. 2022. From noisy prediction to true label: Noisy prediction calibration via generative model. In International Conference on Machine Learning . PMLR, 1277–1297

    2022

  4. [4]

    Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama

  5. [5]

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101 – Min- ing Discriminative Components with Random Forests. In Computer Vision – ECCV 2014, David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, 446–461

    2014

  6. [6]

    Brown et al

    Tom B. Brown et al . 2020. Language Models are Few-Shot Learners. arXiv:2005.14165 [cs.CL] https://arxiv.org/abs/2005.14165

    arXiv 2020

  7. [7]

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. 2023. Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. arXiv:2312.09390 [cs.CL] https: //arxiv.org/abs/2312.09390

    arXiv 2023

  8. [9]

    Jian Chen, Ruiyi Zhang, Tong Yu, Rohan Sharma, Zhiqiang Xu, Tong Sun, and Changyou Chen. 2023. Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labels. arXiv:2305.19518 [cs.LG]

    arXiv 2023

Show all 93 references
  1. [10]

    Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu

  2. [11]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs.CL] https://arxiv.org/abs/1810.04805

    2019 arXiv

  3. [12]

    arXiv:2010.02347 [cs.LG] https://arxiv.org/abs/2010.02347

    Learning with Instance-Dependent Label Noise: A Sample Sieve Approach. arXiv:2010.02347 [cs.LG] https://arxiv.org/abs/2010.02347

    2010 arXiv

  4. [13]

    Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences 120, 30 (July 2023). doi:10.1073/pnas.2305016120

    2023 doi

  5. [14]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  6. [15]

    Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. 2018. Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels. arXiv:1804.06872 [cs.LG]

    2018 arXiv

  7. [16]

    Goh, Charles Siegel, Abhinav Vishnu, and Nathan O

    Garrett B. Goh, Charles Siegel, Abhinav Vishnu, and Nathan O. Hodas. 2018. Using Rule-Based Labels for Weak Supervised Learning: A ChemNet for Transferable Chemical Property Prediction. arXiv:1712.02734 [stat.ML] https://arxiv.org/abs/ 1712.02734

    2018 arXiv

  8. [17]

    Xizewen Han, Huangjie Zheng, and Mingyuan Zhou. 2022. CARD: Classification and Regression Diffusion Models. arXiv:2206.07275 [stat.ML]

    2022 arXiv

  9. [18]

    Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. 2018. Co-teaching: Robust training of deep neural net- works with extremely noisy labels. Advances in neural information processing systems 31 (2018)

    2018

  10. [19]

    Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2019. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. arXiv preprint arXi...

    2019 arXiv

  11. [20]

    Hastings, Sherri Weitl-Harms, Joseph Doty, Zachary J

    John D. Hastings, Sherri Weitl-Harms, Joseph Doty, Zachary J. Myers, and War- ren Thompson. 2024. Utilizing Large Language Models to Synthesize Product Desirability Datasets. In 2024 IEEE International Conference on Big Data (BigData) . IEEE, 5352–5360. doi:10.1109/bigdata6232...

    2024

  12. [21]

    Jiang et al

    Albert Q. Jiang et al. 2024. Mixtral of Experts. arXiv:2401.04088 [cs.LG]

    2024 arXiv

  13. [22]

    Ahmet Iscen, Jack Valmadre, Anurag Arnab, and Cordelia Schmid. 2022. Learning with Neighbor Consistency for Noisy Labels. arXiv:2202.02200 [cs.CV]

    2022 arXiv

  14. [23]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Opti- mization. arXiv:1412.6980 [cs.LG]

    2017 arXiv

  15. [24]

    Weonyoung Joo, Wonsung Lee, Sungrae Park, and Il-Chul Moon. 2019. Dirichlet Variational Autoencoder. arXiv:1901.02739 [cs.LG] https://arxiv.org/abs/1901. 02739

    2019 arXiv

  16. [25]

    Xintong Li, Jinya Jiang, Ria Dharmani, Jayanth Srinivasa, Gaowen Liu, and Jingbo Shang. 2024. Open-world Multi-label Text Classification with Extremely Weak Supervision. arXiv:2407.05609 [cs.CL] https://arxiv.org/abs/2407.05609

    2024 arXiv

  17. [26]

    Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Proceedings of the Twelfth International Conference on Machine Learning . 331–339

    1995

  18. [27]

    Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. 2023. Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations. arXiv:2310.07849 [cs.CL] https://arxiv.org/abs/2310.07849

    2023 arXiv

  19. [28]

    Xin Li and Dan Roth. 2002. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics

    2002

  20. [29]

    Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. 2023. Relative representations enable zero-shot latent space communication. arXiv:2209.15430 [cs.LG] https://arxiv.org/abs/2209. 15430

    2023 arXiv

  21. [30]

    Peters, and Arman Cohan

    Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2024. TESS: Text-to-Text Self- Conditioned Simplex Diffusion. arXiv:2305.08379 [cs.CL]

    2024 arXiv

  22. [31]

    Vitor Oliveira, Gabriel Nogueira, Thiago Faleiros, and Ricardo Marcacini. 2024. Combining prompt-based language models and weak supervision for labeling named entity recognition on legal documents. Artificial Intelligence and Law (2024), 1–21

    2024

  23. [32]

    Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. 2019. SELF: Learning to Filter Noisy Labels with Self-Ensembling. arXiv:1910.01842 [cs.CV] https: //arxiv.org/abs/1910.01842

    2019 arXiv

  24. [33]

    Diego Ortego, Eric Arazo, Paul Albert, Noel E O’Connor, and Kevin McGuinness

  25. [34]

    OpenAI et al. 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https: //arxiv.org/abs/2303.08774

    2024 arXiv

  26. [35]

    Giorgio Patrini, Alessandro Rozza, Aditya Menon, Richard Nock, and Lizhen Qu

  27. [36]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Multi-objective interpolation training for robustness to label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6606–6615. KDD ’25, August 3–7, 2025, Toronto, ON, Canada Liqin Ye, Agam Shah, Chao Zhang, and Sudheer Chava

    2025

  28. [37]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fan...

    2019 arXiv

  29. [38]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi

  30. [39]

    Agam Shah and Sudheer Chava. 2023. Zero is Not Hero Yet: Benchmarking Zero-Shot Performance of LLMs for Financial Tasks. arXiv:2305.16633 [cs.CL] https://arxiv.org/abs/2305.16633

    2023 arXiv

  31. [40]

    Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a General-Purpose Natural Language Processing Task Solver? arXiv:2302.06476 [cs.CL] https://arxiv.org/abs/2302.06476

    2023 arXiv

  32. [41]

    Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases, Vol. 11 (3). NIH Public Access, 269

    2017

  33. [42]

    Weiss, Niru Maheswaranathan, and Surya Ganguli

    Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli

  34. [43]

    Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large Language Models for Data Annotation: A Survey. arXiv:2402.13446 [cs.CL] https://arxiv.org/abs/2402.13446

    2024 arXiv

  35. [44]

    Ke Wang, Jiahui Zhu, Minjie Ren, Zeming Liu, Shiwei Li, Zongye Zhang, Chenkai Zhang, Xiaoyu Wu, Qiqi Zhan, Qingjie Liu, and Yunhong Wang. 2024. A Survey on Data Synthesis and Augmentation for Large Language Models. arXiv:2410.12896 [cs.CL] https://arxiv.org/abs/2410.12896

    2024 arXiv

  36. [45]

    Agam Shah, Arnav Hiray, Pratvi Shah, Arkaprabha Banerjee, Anushka Singh, Dheeraj Eidnani, Bhaskar Chaudhury, and Sudheer Chava. 2024. Numerical Claim Detection in Finance: A New Financial Dataset, Weak-Supervision Model, and Market Analysis. arXiv preprint arXiv:2402.11728 (2024)

    2024 arXiv

  37. [46]

    Team Snorkel. [n. d.]. Using few-shot learning language models as weak super- vision — snorkel.ai. https://snorkel.ai/blog/few-shot-learning-large-language- models/. [Accessed 07-02-2025]

    2025

  38. [47]

    Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. 2020. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 13726–13735

    2020

  39. [48]

    Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu

  40. [49]

    Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A Broad- Coverage Challenge Corpus for Sentence Understanding through Inference. arXiv:1704.05426 [cs.CL] https://arxiv.org/abs/1704.05426

    2018 arXiv

  41. [50]

    Thomas Wolf et al. 2020. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. arXiv:1910.03771 [cs.CL] https://arxiv.org/abs/1910.03771

    2020 arXiv

  42. [51]

    Lei Wang, Yi Hu, Jiabang He, Xing Xu, Ning Liu, Hui Liu, and Heng Tao Shen. 2023. T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Lan- guage Model Signals for Science Question Answering. arXiv:2305.03453 [cs.CL] https://arxiv.org/abs/2305.03453

    2023 arXiv

  43. [52]

    Song Wang, Zhen Tan, Ruocheng Guo, and Jundong Li. 2023. Noise- Robust Fine-Tuning of Pretrained Language Models via External Guidance. arXiv:2311.01108 [cs.CL] https://arxiv.org/abs/2311.01108

    2023 arXiv

  44. [53]

    Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. 2015. Learning from massive noisy labeled data for image classification. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 2691–2699. doi:10.1109/ CVPR.2015.7298885

    2015

  45. [54]

    Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang

  46. [55]

    Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. 2021. Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learning. arXiv:2006.07805 [cs.LG]

    2021 arXiv

  47. [56]

    Peilin Yu and Stephen Bach. 2023. Alfred: A System for Prompted Weak Su- pervision. In Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 3: System Demonstrations) , Danushka Bollegala, Ruihong Huang, and Alan Ritter (Eds.). Assoc...

    2023 doi

  48. [57]

    Tsang, and Masashi Sugiyama

    Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W. Tsang, and Masashi Sugiyama. 2019. How does Disagreement Help Generalization against Label Corruption? arXiv:1901.04215 [cs.LG] https://arxiv.org/abs/1901.04215

    2019 arXiv

  49. [59]

    Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. 2020. Part-dependent Label Noise: Towards Instance-dependent Label Noise. arXiv:2006.07836 [cs.LG]

    2020 arXiv

  50. [60]

    Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner. 2021. Wrench: A comprehensive benchmark for weak supervi- sion. arXiv preprint arXiv:2109.11377 (2021)

    2021 arXiv

  51. [61]

    Rongzhi Zhang, Yue Yu, Pranav Shetty, Le Song, and Chao Zhang. 2022. PRBoost: Prompt-Based Rule Discovery and Boosting for Interactive Weakly-Supervised Learning. arXiv preprint arXiv:2203.09735 (2022)

    2022 arXiv

  52. [62]

    arXiv:2109.02986 [stat.ML] https://arxiv.org/abs/2109.02986

    Instance-dependent Label-noise Learning under a Structural Causal Model. arXiv:2109.02986 [stat.ML] https://arxiv.org/abs/2109.02986

    arXiv

  53. [63]

    Yivan Zhang, Gang Niu, and Masashi Sugiyama. 2021. Learning Noise Tran- sition Matrix from Only Noisy Labels via Total Variation Regularization. arXiv:2102.02414 [stat.ML]

    2021 arXiv

  54. [64]

    Zhilu Zhang and Mert R. Sabuncu. 2018. Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels. arXiv:1805.07836 [cs.LG] https://arxiv.org/abs/1805.07836

    2018 arXiv

  55. [65]

    Zhi-Hua Zhou. 2018. A brief introduction to weakly supervised learning.National science review 5, 1 (2018), 44–53

    2018

  56. [66]

    Yue Yu, Jiaming Shen, Tianqi Liu, Zhen Qin, Jing Nathan Yan, Jialu Liu, Chao Zhang, and Michael Bendersky. 2023. Explanation-aware soft ensemble empowers large language model in-context learning. arXiv preprint arXiv:2311.07099 (2023)

    2023 arXiv

  57. [67]

    Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang. 2020. Fine-tuning pre-trained language model with weak supervision: A contrastive- regularized self-training approach. arXiv preprint arXiv:2010.07835 (2020)

    2020 arXiv

  58. [68]

    in-claim

    Yuchen Zhuang, Yue Yu, Lingkai Kong, Xiang Chen, and Chao Zhang. 2023. Dy- Gen: Learning from Noisy Labels via Dynamics-Enhanced Generative Modeling. arXiv preprint arXiv:2305.19395 (2023). A Dataset and Task Detail • Numerical Claim Detection (NumClaim) : This involves extrac...

    2023 arXiv

  59. [74]

    Dawei Zhu, Xiaoyu Shen, Marius Mosbach, Andreas Stephan, and Dietrich Klakow. 2023. Weaker Than You Think: A Critical Look at Weakly Supervised Learning. arXiv:2305.17442 [cs.CL] https://arxiv.org/abs/2305.17442

    2023 arXiv

  60. [75]

    Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. 2023. Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks. arXiv:2304.10145 [cs.AI] https://arxiv.org/abs/2304.10145

    2023 arXiv

  61. [77]

    ’alt.atheism’: Discussions related to atheism

  62. [78]

    ’comp.graphics’: Topics about computer graphics, in- cluding software and hardware

  63. [79]

    ’comp.os.ms-windows.misc’: Discussions about the Mi- crosoft Windows operating system

  64. [80]

    ’comp.sys.ibm.pc.hardware’: Topics related to IBM PC hardware

  65. [81]

    ’comp.sys.mac.hardware’: Discussions about Mac hard- ware

  66. [82]

    ’comp.windows.x’: Topics about the X Window System

  67. [83]

    ’misc.forsale’: Posts related to buying and selling items

  68. [84]

    ’rec.autos’: Discussions about automobiles

  69. [85]

    ’rec.motorcycles’: Topics related to motorcycles

  70. [86]

    ’rec.sport.baseball’: Discussions about baseball

  71. [87]

    ’rec.sport.hockey’: Discussions about hockey

  72. [88]

    ’sci.crypt’: Topics about cryptography and encryption

  73. [89]

    ’sci.electronics’: Discussions about electronic systems and devices

  74. [90]

    ’sci.med’: Topics related to medical science and health- care

  75. [91]

    ’sci.space’: Discussions about space and astronomy

  76. [92]

    ’soc.religion.christian’: Topics about Christianity and related discussions

  77. [93]

    ’talk.politics.guns’: Discussions about gun politics and related debates

  78. [94]

    ’talk.politics.mideast’: Topics about politics in the Mid- dle East

  79. [95]

    ’talk.politics.misc’: General political discussions not covered by other categories

  80. [96]

    For the provided text below, determine the most appropri- ate category based on the descriptions above

    ’talk.religion.misc’: Discussions about miscellaneous religious topics. For the provided text below, determine the most appropri- ate category based on the descriptions above. Respond by selecting the label (e.g., alt.atheism, comp.graphics, etc.) that best matches the topic o...

    2025

  81. [2015]

    arXiv:1503.03585 [cs.LG]

    Deep Unsupervised Learning using Nonequilibrium Thermodynamics. arXiv:1503.03585 [cs.LG]

    arXiv

  82. [2017]

    arXiv:1609.03683 [stat.ML]

    Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach. arXiv:1609.03683 [stat.ML]

    arXiv

  83. [2019]

    arXiv:1907.10641 [cs.CL] https://arxiv.org/abs/1907.10641

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale. arXiv:1907.10641 [cs.CL] https://arxiv.org/abs/1907.10641

    1907 arXiv

  84. [2021]

    arXiv:2001.03772 [cs.LG]

    Confidence Scores Make Instance-dependent Label-noise Learning Possible. arXiv:2001.03772 [cs.LG]

    2001 arXiv

  85. [2022]

    arXiv:2110.12088 [cs.LG] https://arxiv.org/abs/2110.12088

    Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations. arXiv:2110.12088 [cs.LG] https://arxiv.org/abs/2110.12088

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.