Pith. sign in

REVIEW 3 major objections 5 minor 50 references

Stronger Than You Think: Benchmarking Weak Supervision on Realistic Tasks

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On realistic high-cardinality, imbalanced tasks, weak supervision keeps an advantage over supervised learning until clean labels exceed 1,000, and on one task never loses within 200 labels.

desk verdict A genuinely harder WS benchmark that deserves review, but the headline claim of 1,000+ label crossovers is inflated by an unfair supervised baseline. read the letter →

arxiv 2501.07727 v2 pith:XWUIWAJY submitted 2025-01-13 cs.LG

Tianyi Zhang , Linrong Cai , Jeffrey Li , Nicholas Roberts , Neel Guha , Jinoh Lee , Frederic Sala This is my paper · ORCID
classification cs.LG
keywords weaksupervisionlabelingfunctionslabelmodelsbenchmarktextclassificationcrossoverpointclassimbalancemultilingualtransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BOXWRENCH is a new benchmark for weak supervision (WS), built from text-classification tasks that have many classes, imbalanced label distributions, and genuine domain expertise requirements. The paper's central claim is that on these realistic tasks, supervised learning needs more than 1,000 clean labels to match the better of weak supervision or continuously fine-tuned weak supervision, whereas earlier benchmark studies suggested 50 labels were enough. Two datasets exhibit crossover points beyond 1,000 clean labels, one never crosses within its 200-label validation budget, and carefully rewritten labeling functions raise a fourth dataset's crossover to about 1,600. The same labeling functions written for English are reusable in other languages through translation, yielding crossovers above 1,000 without writing new functions. If the claim holds, weak supervision is a substantially more practical tool for label-hungry applications than recent critiques concluded.

What carries the argument

The load-bearing measure is the crossover point: the amount of clean validation labels at which a supervised-only model's test performance matches or overtakes the better of the weakly supervised or continuously fine-tuned models (where the latter is a model first trained on weak labels and then further fine-tuned on the clean validation labels). The argument is carried by a controlled pipeline in which labeling functions -- small heuristic rules that vote a label or abstain -- are written by a documented development-set procedure, aggregated by simple label models such as majority vote, and used to fine-tune pretrained language models; validation labels are used for early stopping and for selecting the final labeling functions. Because the same pipeline is run on existing simple benchmarks and on the new harder tasks, the crossover differences are attributed to task realism and labeling-function quality rather than to implementation choices.

What would settle it

Re-run the crossover analysis with a single unified label budget that charges the weak-supervision pipeline for the development labels used to write labeling functions and the validation labels used for early stopping and selection; if supervised training then catches up to weak supervision with fewer than 1,000 labels on Amazon31 or Banking77, the paper's headline claim fails.

Watch

Extended reading notes

Core claim

The paper's discovery is that the perceived weakness of weak supervision is an artifact of overly simple benchmark tasks and low-quality labeling functions, not a property of the method. On the new BOXWRENCH tasks, the number of clean labels at which supervised-only training overtakes weak supervision -- the crossover point -- is 1,000 or more on Amazon31 and Banking77, and on Claude9 no crossover occurs within the available 200 validation labels, with continuous fine-tuning ending about 5% higher in macro-F1. Improving the ChemProt labeling functions by adding entity-distance features and pruning low-accuracy rules raises the F1 crossover from roughly 800 to roughly 1,600. In the multilingual setting, English labeling functions applied after machine translation reach crossovers above 1,000 on Chinese and Japanese variants, and the gain persists even when only a fraction of the target-language unlabeled data is available.

Load-bearing premise

The comparison counts only the clean labels used to train the supervised end model, while the weak-supervision pipeline also spends clean labels on designing its labeling functions and on validation-based early stopping and selection; if those labels are charged to weak supervision, the reported crossover points shrink.

Editorial extensions

If this is right

  • On Amazon31 and Banking77, supervised-only training does not catch up to weak supervision until the clean-label budget passes 1,000 examples.
  • On Claude9, no crossover occurs within the 200 available validation labels, and the continuously fine-tuned weak-supervision model stays about 5% higher in macro-F1.
  • Improving the ChemProt labeling functions raises the F1 crossover from roughly 800 to roughly 1,600 clean labels, showing that labeling-function quality directly controls how long weak supervision stays ahead.
  • Reusing English labeling functions via translation yields crossovers above 1,000 on the Chinese and Japanese variants of the voice-assistant task, with no new labeling functions written in the target languages.
  • The simplest aggregation strategies (majority vote and an expectation-maximization-based label model) matched or beat more complex label models across the new datasets, so the conclusion does not depend on a sophisticated aggregation trick.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: an apples-to-apples label budget would charge the weak-supervision pipeline for its development set (250 examples on Amazon31, 24 on Claude9) and for validation labels used in early stopping and labeling-function selection; those charges would pull the crossover points down, and the paper does not quantify how far.
  • Editorial inference: the translation-based labeling-function reuse recipe should be testable on non-parallel multilingual corpora and on languages without dedicated pretrained encoders; if it holds, weak supervision becomes a cheap route to low-resource text classification.
  • Editorial inference: the paper's appendix finds that LLM-generated labeling functions have higher coverage but lower precision than the hand-written ones; a natural next step is a pipeline that combines LLM suggestion with the paper's careful validation and pruning procedure.
  • Editorial inference: the crossover-point framing gives the field a single number to compare datasets and labeling-function sets, which could make future weak-supervision benchmarks directly comparable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces BOXWRENCH, a weak supervision benchmark consisting of five text classification settings (Banking77, ChemProt, Claude9, MASSIVE18/60, and Amazon31) chosen to be more realistic than earlier WS suites in class cardinality, class imbalance, domain expertise, and cross-lingual LF reuse. Against Zhu et al.'s "Weaker Than You Think" finding, the authors measure crossover points where fine-tuning on clean validation labels overtakes weak supervision (WS) or continuous fine-tuning (CFT). They report crossovers beyond 1,000 clean labels on Amazon31 and Banking77, no crossover on Claude9, an increase in the ChemProt F1 crossover to about 1,600 after improved LFs, and crossovers above 1,000 on multilingual MASSIVE18 variants. The paper releases code, labeling functions, and (for most tasks) datasets.

Significance. If the crossover results hold under a fair comparison, BOXWRENCH would be a genuinely useful benchmark: it moves WS evaluation toward high-cardinality, imbalanced, domain-specific tasks; it publishes a documented LF-development procedure; it reproduces the opposing Zhu et al. results on existing benchmarks; and it demonstrates an interesting cross-lingual LF-reuse mechanism with DeepL. The public code release is a real strength and makes the control experiments requested below feasible. I would not yet endorse the headline "stronger than you think," because the two label-efficiency asymmetries discussed below directly affect the magnitude and even the existence of the reported crossover points.

major comments (3)
  1. [Section 3.4, Figures 3-5] The supervised baseline is trained for a fixed 6,000 steps with no early stopping, while weakly supervised and CFT models use early stopping on the validation set. On clean sets of 50-1,000 examples, a fixed 6,000-step RoBERTa fine-tune is very likely to be in the overfitting regime, so the supervised curves in Figures 3-5 may be systematically depressed. Table 2 is consistent with this concern: ChemProt supervised F1_macro is essentially flat (0.405-0.412) as the validation size grows from 100 to 1,607. This asymmetry is load-bearing because the paper's headline is that crossovers exceed 1,000 labels and that Claude9 shows no crossover with 200 labels. Following Zhu et al. [49] does not make the fixed schedule fair: this paper argues against that work's conclusion and should not inherit a protocol that handicaps the supervised side. Please rerun supervised and CFT with matched model selection (e.g., early stopping on the same validation labels for all methods), or provide evidence that 6,000 fixed steps is not overfitting for these datasets.
  2. [Section 3.3 and Sections 3.2/4.1] The label-efficiency comparison undercounts WS's clean-label consumption. Section 3.3 describes a 250-example LF development set for Amazon31 and a 24-example development set for Claude9, and states that final LFs are evaluated on the original validation set; the same validation labels are also used for WS early stopping and hyperparameter selection (Section 3.4). The crossover analysis in Section 4.1 compares supervised training on Dval against WS/CFT, counting only Dval labels as the "label cost." The clean development labels and the clean labels used to select LFs are not charged to WS. A fair comparison should either add these labels to the supervised budget (e.g., supervised with nval+250 labels for Amazon31 or nval+24 for Claude9) or explicitly report a sensitivity analysis showing that the >1,000 crossovers are robust under this accounting. Without this, the reported crossover points conflate WS's algorithmic value with the extra clean labels consumed during LF development and selection.
  3. [Section 4.1 and Figures 3-5] Crossover points are reported as point estimates without uncertainty intervals. The curves in Figures 3-5 have no error bars, and Table 2 reports means and standard deviations only at discrete validation fractions, not at the crossover. Because fine-tuning is stochastic and several of the clean sets are tiny (12, 25, 50, 100, and 200 labels for Claude9), the claims "beyond 1,000," "no crossover," and "around 1,600" need confidence intervals or bootstrap distributions over seeds. As written, the reader cannot tell whether, for example, the Amazon31 crossover is statistically distinct from 800 or whether Claude9's "no crossover" would survive perturbing the supervised curve by one standard error. Please report per-point standard errors and, if feasible, uncertainty on the interpolated crossover locations.
minor comments (5)
  1. [Table 2 caption] The dataset name "ChemPort" in the Table 2 caption is a typo for "ChemProt."
  2. [Appendix B and Section 3.3] Amazon31 is no longer publicly available and is explicitly not released with BOXWRENCH; because Amazon31 is one of the two datasets supporting the >1,000 crossover claim, the main text should mark it as non-reproducible and, if possible, release the processed version or make it available on request.
  3. [Section 3.1] Please specify how crossover points are computed from the validation-size grid (e.g., linear interpolation, first crossing, or visual inspection). The current text says only that the curves "intersect."
  4. [Section 5, limitation (3)] The paper notes in Limitations that WS hyperparameters were not thoroughly tuned, following Zhu et al.; this choice is conservative for WS and should be stated alongside the main crossover results rather than only in the appendix/limitations, especially because the fixed 6,000-step rule for supervised training is a separate and non-conservative choice.
  5. [Abstract and Section 4] The abstract's "1000+" claim is too broad: MASSIVE60's crossover is reported as exceeding only 500, and some ChemProt accuracy improvements are described as "smaller but consistent." A more precise summary of where the 1,000-label claim does and does not hold would help readers.

Circularity Check

1 steps flagged · score 3.0 of 10

Partial circularity via asymmetric label accounting: WS's clean-label development and selection costs are not charged to its side of the crossover comparison, inflating the reported 1000+ label crossover points.

  1. fitted input called prediction [Section 3.3 (LF Design Pipeline), Section 3.4 (Supervised learning scenario), Section 4.1 (crossover claims)]
    "We use 250 examples as a development set for Amazon31 ... For Claude9, the development set had 24 examples ... To evaluate the final LFs, we calculate their accuracy scores on the original validation set. ... Supervised: We use clean labels from the validation set directly for fine-tuning an end model."

    The crossover analysis charges clean labels only to the supervised arm: the x-axis is the size of Dval used to train fθSUP. The WS arm consumes clean labels before the comparison: LFs are built from clean development sets (250 for Amazon31, 24 for Claude9), final LFs are selected using validation accuracy, and WS/CFT training is early-stopped on Dval. None of these clean-label uses are charged to WS. Thus the headline result—'supervised learning requires substantial amounts (1000+) of labeled examples to match WS'—is not a clean-label cost-parity measurement; it compares a supervised model trained only on Dval with a WS pipeline already fitted to clean labels.

full rationale

This is an empirical benchmark paper, not a mathematical derivation, so most circularity patterns (self-definitional equations, uniqueness theorems, ansatz-via-citation) do not apply. The identified issue is a cost-accounting asymmetry: the WS arm uses clean labels for LF development, LF selection, and early stopping, while the supervised baseline is charged for every clean label it trains on; the fixed-6,000-step supervised protocol may additionally depress the supervised curve. These choices can inflate the reported crossover points, but they do not make the crossover equal to an input by construction: the WS result still depends on real LF quality, task difficulty, and model behavior, and the released code makes a symmetric re-run feasible. The paper's citations of prior work are not load-bearing self-citations: [37] is contextual, and [49] is the protocol being challenged and replicated in Appendix E. No uniqueness claim or ansatz is smuggled in. Score 3 reflects a partial circularity in the label-budget framing of the central claim, not a reduction of the result to its inputs.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central empirical result depends on choices made by the authors rather than on a formal derivation: the size of the LF development sets, the fixed supervised training steps, the validation-based LF selection, and the use of majority vote as the default label model. These are honest, disclosed choices, but they are the load-bearing assumptions that a replication study would need to vary.

free parameters (4)
  • LF development set size (Amazon31) = 250
    Chosen by hand for LF crafting; affects LF quality and thus the reported crossover point.
  • LF development set size (Claude9) = 24
    Manually inspected labeled examples used to write LFs for Claude9; affects LF quality and crossover results.
  • Supervised fine-tuning steps = 6000
    Fixed training steps for the supervised baseline following Zhu et al.; WS models use early stopping instead, which may understate supervised performance.
  • Validation proportions for crossover grid = 6.25%, 12.5%, 25%, 50%, 100%
    Evaluation grid chosen for plotting crossover curves; not a model parameter but determines how crossover points are estimated.
assumptions (3)
  • domain assumption LF accuracy on the validation set is an acceptable criterion for selecting final LFs
    The authors evaluate LFs on the validation set (Section 3.3), using labels that also define the supervised baseline; this assumes the selection does not unfairly advantage WS.
  • domain assumption Majority vote is a representative label model for assessing WS usefulness
    Majority vote is the default label model; ablations show other LMs do not consistently beat it, but this limits the generality of the WS effectiveness claim.
  • domain assumption DeepL translation preserves LF signal across languages
    MASSIVE cross-lingual experiments translate non-English text to English before applying English LFs (Figure 2); no analysis of translation errors is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stronger Than You Think: Benchmarking Weak Supervision on Realistic Tasks." pith.science (2026). https://pith.science/paper/XWUIWAJY

@misc{pith2026250107727,
  author       = {Pith},
  title        = {Pith review of: Stronger Than You Think: Benchmarking Weak Supervision on Realistic Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XWUIWAJY}},
  note         = {Machine review of arXiv:2501.07727}
}
read the original abstract

Weak supervision (WS) is a popular approach for label-efficient learning, leveraging diverse sources of noisy but inexpensive weak labels to automatically annotate training data. Despite its wide usage, WS and its practical value are challenging to benchmark due to the many knobs in its setup, including: data sources, labeling functions (LFs), aggregation techniques (called label models), and end model pipelines. Existing evaluation suites tend to be limited, focusing on particular components or specialized use cases. Moreover, they often involve simplistic benchmark tasks or de-facto LF sets that are suboptimally written, producing insights that may not generalize to real-world settings. We address these limitations by introducing a new benchmark, BOXWRENCH, designed to more accurately reflect real-world usages of WS. This benchmark features tasks with (1) higher class cardinality and imbalance, (2) notable domain expertise requirements, and (3) opportunities to re-use LFs across parallel multilingual corpora. For all tasks, LFs are written using a careful procedure aimed at mimicking real-world settings. In contrast to existing WS benchmarks, we show that supervised learning requires substantial amounts (1000+) of labeled examples to match WS in many settings.

Figures

Figures reproduced from arXiv: 2501.07727 by the authors.

Figure 1
Figure 1. We compare end models trained with three different pipelines: (1) [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Using an off-the-shelf translator (DeepL) to reuse English LFs for the multilingual variants of MASSIVE18. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Crossover points on three of our datasets: Amazon31, Banking77, and Claude9. For both Amazon31 and [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (10 more)
Figure 3
Figure 3. Figure 3: Crossover points on six existing WS benchmarks: AGNews, Semeval, Yelp, ChemProt, Trec, IMDB. The [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]
Figure 5
Figure 5. Figure 5: Crossover points on ChemProt with our new LFs. The new LFs for ChemProt demonstrate a higher [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Crossover points on the multilingual MASSIVE18 dataset. The comparison of the green and yellow solid [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: This plot illustrates the crossover points when comparing ChemProt performance with our old and new [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 9
Figure 9. Figure 9: Using English text as an augmented data for [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Crossoverpoints for additional NER task. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Crossoverpoints for additional NER task. [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Crossover points using ARS2 and Cosine as end models. [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: SciBERT Results [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Using LLM as baseline 22 [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 43 canonical work pages

  1. [49]

    activat”, “increas

    Dawei Zhu, Xiaoyu Shen, Marius Mosbach, Andreas Stephan, and Dietrich Klakow. Weaker than you think: A critical look at weakly supervised learning, 2023. 13 A Broader Social Impact of WS Benchmarking Our benchmark aims to provide a platform to evaluate WS methods on more realistic datasets. Methods with successful performance are more likely to be useful ...

    work page 2023

  2. [1]

    URL https://huggingface.co/datasets/defunct-datasets/ amazon_us_reviews/tree/main

    Amazon-us-review. URL https://huggingface.co/datasets/defunct-datasets/ amazon_us_reviews/tree/main

    work page

  3. [2]

    Antunes and S

    R. Antunes and S. Matos. Extraction of chemical-protein interactions from the literature using neural networks and narrow instance representation. Database : the journal of biological databases and curation, 2019:baz095,

    work page 2019

  4. [3]

    Snorkel drybell: A case study in deploying weak supervision at industrial scale

    Stephen H Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alex Ratner, Braden Hancock, Houman Alborzi, et al. Snorkel drybell: A case study in deploying weak supervision at industrial scale. In Proceedings of the 2019 International Conference on Management of Data , pages 362–375, 2019

    work page 2019

  5. [4]

    SciBERT: A pretrained language model for scientific text

    Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. InEMNLP, pages 3615–3620, 2019. URL https://www.aclweb.org/anthology/D19-1371

    work page 2019

  6. [5]

    Efficient intent detection with dual sentence encoders

    Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on NLP for ConvAI - ACL 2020, mar 2020. URL https://arxiv.org/abs/2003.04807. Data available at https://github.com/PolyAI- LDN/task-specific-datasets

    arXiv 2020

  7. [6]

    Legal-bert: The muppets straight out of law school, 2020

    Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. Legal-bert: The muppets straight out of law school, 2020. URL https://arxiv.org/abs/2010. 02559

    work page 2020

  8. [7]

    Lexglue: A benchmark dataset for legal language understanding in english, 2022

    Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. Lexglue: A benchmark dataset for legal language understanding in english, 2022

    work page 2022

Show all 50 references
  1. [8]

    A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society , 28(1):20–28, 1979. ISSN 00359254, 14679876. URL http: //www.jstor.org/stable/2346806

    1979

  2. [9]

    Deepl api, 2024

    DeepL. Deepl api, 2024. URL https://www.deepl.com/pro#api. Accessed: 2024-05-27

    2024

  3. [10]

    Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages, 2022

    Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. Massive: A 1m-example multilingual ...

    2022

  4. [11]

    Ncbi disease corpus dataset, 2023

    National Center for Biotechnology Information (NCBI). Ncbi disease corpus dataset, 2023. URL https: //ncbi.nlm.nih.gov/research/bionlp/Data/disease/. Accessed: 2024-10-31

    2023

  5. [12]

    Fries, Paroma Varma, Vincent S

    Jason A. Fries, Paroma Varma, Vincent S. Chen, Ke Xiao, Heliodoro Tejeda, Priyanka Saha, Jared Dunnmon, Henry Chubb, Shiraz Maskatia, Madalina Fiterau, Scott Delp, Euan Ashley, Christopher Ré, and James R. Priest. Weakly supervised classification of aortic valve malformations ...

    2019 doi

  6. [13]

    Fu, Mayee F

    Daniel Y . Fu, Mayee F. Chen, Frederic Sala, Sarah M. Hooper, Kayvon Fatahalian, and Christopher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods, 2020

    2020

  7. [14]

    BERT-Base, Chinese

    Google. BERT-Base, Chinese. https://huggingface.co/google-bert/ bert-base-chinese. Accessed: 2024-10-31

    2024

  8. [15]

    Can large language models design accurate label functions?, 2023

    Naiqing Guan, Kaiwen Chen, and Nick Koudas. Can large language models design accurate label functions?, 2023

    2023

  9. [16]

    Improved pattern learning for bootstrapped entity extraction

    Sonal Gupta and Christopher Manning. Improved pattern learning for bootstrapped entity extraction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning , pages 98–108, 2014

    2014

  10. [17]

    Automatic acquisition of hyponyms from large text corpora

    Marti A Hearst. Automatic acquisition of hyponyms from large text corpora. In Proceedings of the 14th conference on Computational linguistics-Volume 2, pages 539–545. Association for Computational Linguistics, 1992. 11

    1992

  11. [18]

    Scriptoriumws: A code generation assistant for weak supervision

    Tzu-Heng Huang, Catherine Cao, Spencer Schoenberg, Harit Vishwakarma, Nicholas Roberts, and Frederic Sala. Scriptoriumws: A code generation assistant for weak supervision. ICLR Deep Learning for Code Workshop, 2023

    2023

  12. [19]

    The alchemist: Automated labeling 500x cheaper than llm data annotators, 2024

    Tzu-Heng Huang, Catherine Cao, Vaishnavi Bhargava, and Frederic Sala. The alchemist: Automated labeling 500x cheaper than llm data annotators, 2024. URL https://arxiv.org/abs/2407.11004

    2024 arXiv

  13. [20]

    Learning visual features from large weakly supervised data

    Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14, pages 67–84...

    2016

  14. [21]

    Iterative learning for reliable crowdsourc- ing systems

    David Karger, Sewoong Oh, and Devavrat Shah. Iterative learning for reliable crowdsourc- ing systems. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc., 20...

    2011

  15. [22]

    Nandal, Erin M

    Martin Krallinger, Obdulia Rabal, Saber Ahmad Akhondi, Martín Pérez Pérez, Jesus Santamaría, Gael Pérez Rodríguez, Georgios Tsatsaronis, Ander Intxaurrondo, José Antonio Baso López, Umesh K. Nandal, Erin M. van Buel, Ambika Chandrasekhar, Marleen Rodenburg, Astrid Lægreid, Mar...

    2017

  16. [23]

    Characterizing the impacts of semi- supervised learning for weak supervision

    Jeffrey Li, Jieyu Zhang, Ludwig Schmidt, and Alexander J Ratner. Characterizing the impacts of semi- supervised learning for weak supervision. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, vo...

    2023

  17. [24]

    Jeffrey Li, Jieyu Zhang, Ludwig Schmidt, and Alexander J. Ratner. Characteriz- ing the impacts of semi-supervised learning for weak supervision. 2024. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 0d6270381e018b3d83eb9be7d0b06036-Abstract-Conference.html

    2024

  18. [25]

    Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J

    Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database, 2016:baw068, 05 2016...

    2016 doi

  19. [26]

    Claudette: an automated detector of potentially unfair clauses in online terms of service

    Marco Lippi, Przemysław Pałka, Giuseppe Contissa, Francesca Lagioia, Hans-Wolfgang Micklitz, Giovanni Sartor, and Paolo Torroni. Claudette: an automated detector of potentially unfair clauses in online terms of service. Artificial Intelligence and Law, 27(2):117–139, February ...

    2019 doi

  20. [27]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  21. [28]

    Exploring the limits of weakly supervised pretraining

    Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceed- ings of the European conference on computer vision (ECCV), pages 181–196, 2018

    2018

  22. [29]

    Mintz, Steven Bills, R

    Mike D. Mintz, Steven Bills, R. Snow, and Dan Jurafsky. Distant supervision for relation extraction without labeled data. In ACL/IJCNLP, 2009

    2009

  23. [30]

    Nb-bert-base: A norwegian bert model built on the digital collection at the national library of norway, March 2021

    NbAiLab. Nb-bert-base: A norwegian bert model built on the digital collection at the national library of norway, March 2021. URL https://huggingface.co/NbAiLab/nb-bert-base. Trained on a wide variety of Norwegian text from the last 200 years, intended for fine-tuning on specif...

    2021

  24. [31]

    BERT-Base, Japanese

    Tohoku NLP. BERT-Base, Japanese. https://huggingface.co/tohoku-nlp/ bert-base-japanese. Accessed: 2024-10-31

    2024

  25. [32]

    Robust speech recognition via large-scale weak supervision, 2022

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. URL https://arxiv.org/abs/2212. 04356. 12

    2022

  26. [33]

    A. J. Ratner, B. Hancock, J. Dunnmon, F. Sala, S. Pandey, and C. Ré. Training complex models with multi-task weak supervision. In AAAI, pages 4763–4771, 2019

    2019

  27. [34]

    Data programming: Cre- ating large training sets, quickly

    Alexander Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Cre- ating large training sets, quickly. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 3574–3582, Red Hook, NY , USA,...

    2016

  28. [35]

    Ratner, Stephen H

    Alexander J. Ratner, Stephen H. Bach, Henry R. Ehrenberg, Jason Alan Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases, 11 3:269–282, 2017

    2017

  29. [36]

    Overton: A data system for monitoring and improving machine-learned products

    Christopher Ré, Feng Niu, Pallavi Gudipati, and Charles Srisuwananukorn. Overton: A data system for monitoring and improving machine-learned products. In Proceedings of the 10th Annual Conference on Innovative Data Systems Research, 2020

    2020

  30. [37]

    Autows-bench-101: Benchmarking automated weak supervision with 100 labels, 2023

    Nicholas Roberts, Xintong Li, Tzu-Heng Huang, Dyah Adila, Spencer Schoenberg, Cheng-Yu Liu, Lauren Pick, Haotian Ma, Aws Albarghouthi, and Frederic Sala. Autows-bench-101: Benchmarking automated weak supervision with 100 labels, 2023

    2023

  31. [38]

    Introduction to the conll-2003 shared task: Language-independent named entity recognition

    Erik Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In CoNLL, pages 142–147, 2003

    2003

  32. [39]

    Universalizing weak supervision

    Changho Shin, Winfred Li, Harit Vishwakarma, Nicholas Carl Roberts, and Frederic Sala. Universalizing weak supervision. In International Conference on Learning Representations (ICLR), 2022. URL https: //openreview.net/forum?id=YpPiNigTzMT

    2022

  33. [40]

    Mitigating source bias for fairer weak supervision

    Changho Shin, Sonia Cromp, Dyah Adila, and Frederic Sala. Mitigating source bias for fairer weak supervision. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  34. [41]

    Adaptive ranking-based sample selection for weakly supervised class-imbalanced text classification, 2022

    Linxin Song, Jieyu Zhang, Tianxiang Yang, and Masayuki Goto. Adaptive ranking-based sample selection for weakly supervised class-imbalanced text classification, 2022. URL https://arxiv.org/abs/2210. 03092

    2022

  35. [42]

    Uk biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age

    Cathie Sudlow, John Gallacher, Naomi Allen, Valerie Beral, Paul Burton, John Danesh, Paul Downey, Paul Elliott, Jane Green, Martin Landray, Bette Liu, Paul Matthews, Giok Ong, Jill Pell, Alan Silman, Alan Young, Tim Sprosen, Tim Peakman, and Rory Collins. Uk biobank: an open a...

    2015 doi

  36. [43]

    Snuba: Automating weak supervision to label training data

    Paroma Varma and Christopher Ré. Snuba: Automating weak supervision to label training data. In VLDB, volume 12, page 223. NIH Public Access, 2018

    2018

  37. [44]

    Ontonotes release 5.0

    Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. Ontonotes release 5.0. URL https://catalog.ldc.upenn.edu/LDC2013T19

  38. [45]

    Fine-tuning pre-trained language model with weak supervision: A contrastive-regularized self-training approach

    Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang. Fine-tuning pre-trained language model with weak supervision: A contrastive-regularized self-training approach. In NAACL-HLT, pages 1063–1077, 2021. URL https://www.aclweb.org/anthology/2021.naacl-main.84

    2021

  39. [46]

    Wrench: A comprehensive benchmark for weak supervision, 2021

    Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner. Wrench: A comprehensive benchmark for weak supervision, 2021

    2021

  40. [47]

    A survey on programmatic weak supervision

    Jieyu Zhang, Cheng-Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner. A survey on programmatic weak supervision. arXiv preprint arXiv:2202.05433, 2022

    2022 arXiv

  41. [48]

    Walnut: A benchmark on semi-weakly supervised learning for natural language understanding, 2022

    Guoqing Zheng, Giannis Karamanolakis, Kai Shu, and Ahmed Hassan Awadallah. Walnut: A benchmark on semi-weakly supervised learning for natural language understanding, 2022

    2022

  42. [2019]

    URL https://doi.org/10.1093/database/baz095

    doi: 10.1093/database/baz095. URL https://doi.org/10.1093/database/baz095

    doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.