Pith. sign in

REVIEW 3 major objections 4 minor 5 cited by

Weak-to-Strong Generalization Through the Data-Centric Lens

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that weak-to-strong generalization is governed by overlap density — the fraction of training points containing both an easy pattern and a hard pattern.

desk verdict A genuinely useful data-centric framing of weak-to-strong generalization, with a novel detection algorithm and extensive experiments, but the headline theoretical claim is not actually proven by the theorem. read the letter →

arxiv 2412.03881 v2 pith:6QA5HADJ submitted 2024-12-05 cs.LG cs.AIstat.ML

Changho Shin , John Cooper , Frederic Sala This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords weak-to-stronggeneralizationoverlapdensityeasy-harddataselectionpseudolabelingrobustexpansionsuperalignmentdata-centriclearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that weak-to-strong generalization — when a strong model is trained on labels supplied by a weaker model — is governed by a simple data property called overlap density: the proportion of points that contain both an easy pattern (which the weak model can learn) and a hard pattern (which only the strong model can learn). On such points the weak labels are reliable, and the strong model can use them to learn the hard pattern and then generalize to hard-only points where the weak model is unreliable. The authors provide a theoretical bound showing the generalization benefit is a function of overlap density, a practical algorithm to detect overlap points, and a bandit-style data selection method that chooses data sources maximizing overlap density. They validate the mechanism across large language model and weak-supervision settings. If correct, this refocuses weak-to-strong learning on data acquisition: seek points where easy and hard patterns co-occur, rather than only designing better training algorithms.

What carries the argument

The central object is the overlap density, defined as the fraction of data points containing both an easy pattern and a hard pattern. The argument is carried by the robust expansion property: the collection of correctly pseudolabeled overlap points, $M'_\eta(S^{\mathrm{good}}_i \cap D_{\mathrm{overlap}}, F)$, is assumed to satisfy $(c,q,\eta)$-robust expansion on $(S^{\mathrm{bad}}_i \cap D_{\mathrm{hard\,only}}, S^{\mathrm{good}}_i \cap D_{\mathrm{overlap}})$. This expansion is the mechanism by which reliable labels on overlap points flow through the neighborhood structure to correct mistakes on hard-only points. The overlap detection algorithm operationalizes the mechanism using two signals: weak-model confidence (to isolate hard-only points) and inner-product alignment with hard-only points (to separate overlap from easy-only points).

What would settle it

Construct or find a dataset whose measured overlap density is high but whose overlap points and hard-only points are disconnected in feature space (so the robust expansion coefficient $c$ is near zero); if the weak-to-strong model still improves on hard-only points, overlap density is not the driving mechanism, while if it fails, the mechanism is supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that weak-to-strong generalization is characterized by the overlap density: points that contain both an easy pattern (learnable by the weak model) and a hard pattern (only learnable by the strong model). The weak model reliably labels overlap points via the easy pattern; the strong model then learns the hard pattern from those reliable labels and can correct pseudolabels on hard-only points. Theorem 4.1 makes this precise: under a robust expansion assumption on the correctly pseudolabeled overlap points, the weak-to-strong model's error on hard-only points is bounded by its disagreement with the weak model plus a term that shrinks with the expansion coefficient $c$, so pseudolabel correction is provably achieved when overlap density expands sufficiently. The paper also proves a concentration bound (Theorem 4.2) separating overlap points from easy-only points under a Gaussian mixture, and a regret bound (Theorem 4.3) of $O(\sqrt{K \log T / t})$ for the UCB data-source selection algorithm. Empirically, weak-to-strong performance tracks overlap density in 19 LLM datasets and 9 weak-supervision datasets, showing low, medium, and high overlap regimes.

Load-bearing premise

The load-bearing premise is that correctly pseudolabeled overlap points expand, through the neighborhood structure, to hard-only points where the weak model errs; if that expansion is weak or absent, the theoretical link between overlap density and weak-to-strong generalization collapses.

Editorial extensions

If this is right

  • If overlap density is the primary driver, then data acquisition and curation that maximize overlap density should matter more than algorithmic changes to the weak-to-strong training procedure.
  • The UCB data-source selection algorithm will, with enough rounds, converge to the source with the highest overlap density and thereby improve weak-to-strong generalization, with regret $O(\sqrt{K \log T / t})$.
  • Weak-to-strong generalization should exhibit three regimes: low overlap (performance at or below the weak model), medium overlap (comparable to the weak model), and high overlap (approaching the strong model trained on true labels).
  • Overlap points detected with one model transfer to another model, implying that overlap density is a latent property of the data rather than an artifact of a particular representation.
  • In superalignment-style settings, the practical path is to obtain or construct data where human-accessible (easy) and superhuman-accessible (hard) patterns co-occur.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if real datasets have weak robust expansion, raw overlap density may not suffice; data geometry and neighborhood structure would need to be measured alongside overlap density to predict weak-to-strong gains.
  • Editorial extension: the binary easy/hard split could generalize to a spectrum of difficulty levels, with 'overlap density' becoming a weighted measure across difficulty layers; the same selection framework would then target sources dense in multi-level overlaps.
  • Editorial extension: the overlap-detection and UCB-selection machinery could be applied proactively to active learning or data generation (e.g., synthesizing overlap points) rather than only choosing among existing sources.
  • Editorial extension: a cleaner causal test would intervene on the data — fixing a source, adding extra hard-only or extra overlap points, and checking whether only overlap additions shift hard-only accuracy — which the paper's controlled synthetic experiments approximate but do not fully separate in real datasets.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes that weak-to-strong generalization is governed by 'overlap density,' the fraction of data points containing both an easy pattern learnable by the weak model and a hard pattern learnable only by the strong model. It introduces Algorithm 2 to detect such points from weak-model confidence and inner-product alignment, and Algorithm 1, a UCB-based bandit, to select data sources with maximal overlap density. The theory consists of Theorem 4.1, a conditional error bound for the weak-to-strong model on hard-only points; Theorem 4.2, a separation guarantee for overlap scores in a Gaussian mixture; and Theorem 4.3, a regret bound for data selection. The experiments cover 19 LLM datasets, 9 weak-supervision datasets, and controlled synthetic settings.

Significance. If validated, the overlap-density mechanism would give practitioners a concrete data-acquisition target for weak-to-strong generalization and superalignment, and the paper's data-centric framing is a useful complement to existing algorithmic work. Strengths include released code, broad and reproducible experiments with standard deviations over 20 seeds, controlled synthetic ablations in Appendices F.3 and F.4, and clean regret and detection bounds. The main weakness is that Theorem 4.1 does not actually derive a functional dependence on overlap density; the expansion assumption contains the mechanism. The empirical contributions are substantial, but the central theoretical claim needs either proof or reframing before journal publication.

major comments (3)
  1. [4.1 (Theorem 4.1, Appendix D.1)] The abstract and Section 4.1 claim that 'the generalization benefit is a function of the overlap density,' but Theorem 4.1 does not establish this. The bound contains no p_i^(overlap)=P(Doverlap|Si) term; overlap enters only through the assumption that the family M'_eta(Sgood_i ∩ Doverlap, F) satisfies (c,q,eta)-robust expansion on (Sbad_i ∩ Dhard only, Sgood_i ∩ Doverlap). The theorem therefore shows that if correctly pseudolabeled overlap points expand to hard-only points, the strong model can correct pseudolabels on hard-only points. It does not show that the expansion coefficient c, or the improvement rho defined after the theorem, is increasing in p_i^(overlap). To make the headline claim a theorem consequence, the authors should prove such a dependence, for example under the Gaussian mixture model of Appendix D.3, or explicitly demote the claim to a conditional statement about overlap points that expand.
  2. [Appendix D.1 (robust expansion assumption)] The key assumption that M'_eta(Sgood_i ∩ Doverlap, F) satisfies (c,q,eta)-robust expansion on (Sbad_i ∩ Dhard only, Sgood_i ∩ Doverlap) is essentially the mechanism the paper sets out to establish: it asserts that correct pseudolabels on overlap points propagate through the neighborhood structure to hard-only points. No derivation of this expansion from the data model in Section 3.1 or from the Gaussian mixture setup in Appendix D.3 is provided, so Theorem 4.1 is an application of the Lang et al. (2024) framework rather than a data-centric characterization. A concrete remedy is to derive the expansion from the Gaussian mixture model, or to report empirical estimates of the expansion coefficient c as the overlap density varies and show that it increases monotonically.
  3. [5.1, 5.3, Appendix F.6] The real-data validation in Figure 2 is partially circular: the overlap density on the x-axis is produced by Algorithm 2, whose two detection rules (low weak-model confidence for hard-only points, high inner-product alignment for overlap points) are the same intuitions that define the overlap model in Section 3.1. Thus the observed correlation could reflect the detector's criteria rather than an independent property of the data. The synthetic experiments in Section 5.3, where the partition is known by construction, and the transferability experiment in Appendix F.6 provide important mitigation. I recommend adding one real-data experiment with an independent ground-truth partition of easy/hard/overlap points, or reporting precision/recall of Algorithm 2 against synthetic ground truth under the noise settings of Appendix F.4, to break the circularity.
minor comments (4)
  1. [Section 4.1 / Appendix D.1] The main-text statement of Theorem 4.1 says 'Suppose P satisfies (c,q) expansion on (Sbad_i ∩ Dhard only, Sgood_i ∩ Doverlap)', but Definition 1 defines expansion for sets U ⊂ B, and the formal theorem in Appendix D.1 uses robust expansion of the family M'_eta on the same ordered pair. Please align the notation with the formal statement.
  2. [Algorithm 2 / Appendix C] Algorithm 2 uses inner-product scores in the pseudocode, while Appendix C states that the experiments use absolute cosine similarity on last-layer activations; please reconcile the pseudocode with the implementation.
  3. [Throughout] There are several typos: 'genarated' in Appendix E, 'HelleSwag' in the dataset list, and 'optmal' in Section 5.3.2. A careful copyedit is needed.
  4. [Section 4.1] The sentence after Theorem 4.1 says the simplified version is obtained by setting eta = 0, q = 0, but the stated theorem still contains q in the condition; please clarify the relationship between the simplified and formal statements.

Circularity Check

1 steps flagged · score 4.0 of 10

Theorem 4.1's claimed support for the overlap-density mechanism rests on an expansion assumption that is the mechanism itself, and the theorem's bound contains no overlap-density term.

  1. other [Appendix D.1 (Problem Setup) and Theorem 4.1 in Section 4.1]
    "Our goal is to show how the overlap density can make strong model perform better in p(hard)_i based on (robust) expansion property. Our main assumption is M′_η(Sgood_i ∩ Doverlap, F) satisfies (c,q,η) robust expansion on (Sbad_i ∩ Dhard only, Sgood_i ∩ Doverlap)."

    The claimed result—that the weak-to-strong generalization benefit is a function of overlap density—is not present in Theorem 4.1's bound, which is a function of c, ε2, err(fw2s,fweak|Sgood_i∩Doverlap), and P(Rη(fw2s)^c|Sgood_i∩Doverlap). The only place overlap enters the analysis is the quoted 'main assumption': correctly pseudolabeled overlap points robustly expand to hard-only points. That assumption is exactly the mechanism the paper claims to establish—overlap points teach hard patterns—so the theorem is a conditional restatement of the mechanism rather than a derivation of an overlap-density dependence.

full rationale

The central theoretical claim is partially circular: Theorem 4.1 is an expansion bound (largely following Lang et al., as the paper states) in which overlap density appears only inside the robust-expansion assumption that is the mechanism under test; the bound itself contains no p_i^(overlap) term, so the abstract's claim that 'the generalization benefit is a function of the overlap density' is not derived from the theorem. However, the paper's empirical program is substantially independent: the synthetic experiments in Section 5.3 use known ground-truth overlap composition and show weak-to-strong gains concentrated on hard-only points, and Theorem 4.2's overlap-detection guarantee is derived from an explicit Gaussian model rather than fitted to the outcomes. There is no load-bearing self-citation chain; the citation to Lang et al. (2024) is external and the framework is acknowledged. The mismatch between the theorem and the headline claim is therefore a rigor/interpretation problem as much as a circularity, so the overall score is moderate rather than severe.

Assumptions & free parameters 3 free parameters · 6 assumptions · 1 invented entities

The central claim rests on an assumed easy/hard feature decomposition, a weak model that cannot see hard features, and an expansion property from overlap points to hard-only points. The expansion assumption is close to the claim itself. The overlap detection theory additionally assumes well-separated Gaussian mixtures. Free parameters are the algorithmic thresholds and sample sizes, which are data-dependent but not fitted to the target W2S accuracy.

free parameters (3)
  • tau_hard = data-dependent (binary segmentation on weak-model confidence scores)
    Threshold in Algorithm 2 to separate hard-only points; estimated from each dataset's confidence distribution, not fit to final W2S accuracy.
  • tau_overlap = data-dependent (binary segmentation on overlap scores)
    Threshold in Algorithm 2 to separate overlap from easy-only points; estimated from inner product or cosine similarity scores.
  • n_controlled = varies per dataset (e.g., 1150, 1404 in Table A2)
    Number of points sampled for the controlled overlap-density curves; chosen to manage train set sizes, not fit to theory.
assumptions (6)
  • domain assumption Feature decomposition x = [xeasy, xhard] with easy features learnable by the weak model and hard features not (Assumption A1).
    Section 3.1; the overlap mechanism is defined on this decomposition, which is not given for real data.
  • domain assumption Weak model has no access to hard patterns: fweak([xeasy, xhard]) = fweak([xeasy, 0]) (Assumption A2).
    Section 3.1; ensures weak predictions are uninformative on hard-only points; in real LLMs this is approximate.
  • domain assumption Pseudolabel error on hard-only points is much larger than on easy/overlap points, eps1 << eps2 (Assumption A3).
    Section 3.1; needed for the mechanism, and plausible in many settings.
  • ad hoc to paper Robust expansion: M'_eta(Sgood_i ∩ Doverlap, F) satisfies (c,q,eta)-robust expansion on (Sbad_i ∩ Dhard only, Sgood_i ∩ Doverlap).
    Key assumption in Theorem 4.1 and Appendix D.1; it asserts that correct pseudolabels on overlap points expand to hard-only points, which is essentially the mechanism the paper claims to prove.
  • domain assumption Gaussian mixture model for overlap detection theory in Theorem 4.2.
    Appendix D.3 assumes label-conditioned Gaussian mixtures with shared covariance, enabling subexponential concentration; real data may not satisfy this.
  • domain assumption Weak model's hard-only input is exactly zero and the prediction probability is sigma(0)=0.5, giving perfect hard-only detection in the theoretical model.
    Section 4.2; in practice confidence thresholds are noisy and hard-only points are not exactly zero.
invented entities (1)
  • Overlap point (easy-hard overlap) independent evidence
    purpose: A latent data point containing both an easy pattern (weak-learnable) and a hard pattern (strong-only); posited as the key driver of weak-to-strong generalization.
    Algorithm 2 provides an operational handle: any dataset can be scored for overlap, and W2S improvement can be measured on held-out hard points; this is falsifiable on new datasets, though all validation in the paper is by the authors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Weak-to-Strong Generalization Through the Data-Centric Lens." pith.science (2026). https://pith.science/paper/6QA5HADJ

@misc{pith2026241203881,
  author       = {Pith},
  title        = {Pith review of: Weak-to-Strong Generalization Through the Data-Centric Lens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6QA5HADJ}},
  note         = {Machine review of arXiv:2412.03881}
}
read the original abstract

The weak-to-strong generalization phenomenon is the driver for important machine learning applications including highly data-efficient learning and, most recently, performing superalignment. While decades of research have resulted in numerous algorithms that produce strong empirical performance, understanding what aspects of data enable weak-to-strong generalization has been understudied. We propose a simple data-centric mechanism that characterizes weak-to-strong generalization: the overlap density. Intuitively, generalization tracks the number of points that contain overlaps, i.e., both easy patterns (learnable by a weak model) and challenging patterns (only learnable by a stronger model), as with such points, weak predictions can be used to learn challenging patterns by stronger models. We provide a practical overlap detection algorithm to find such points in datasets and leverage them to learn, among multiple sources of data, which to query when seeking to maximize overlap density and thereby enhance weak-to-strong generalization. We present a theoretical result showing that the generalization benefit is a function of the overlap density and a regret bound for our data selection algorithm. Empirically, we validate the mechanism and the overlap detection algorithm on a wide array of settings.

Figures

Figures reproduced from arXiv: 2412.03881 by the authors.

Figure 1
Figure 1. Left: overlapping easy and hard patterns in our dataset are the key to weak-to-strong [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overlap density versus performance in weak-to-strong generalization with LLMs. Red lines [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Data selection results with Algorithm 1 for Amazon Polarity and DREAM datasets. We [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Accuracy in each data region in synthetic experiments. As expected, the performance gain [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Synthetic data selection experiment. Our algorithm demonstrates better data efficiency than [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions

    cs.LG 2025-02 conditional novelty 8.0 of 10

    Weak-to-strong performance is governed by the overlap between the weak model's unlearnable error space and the strong model's principal-representation space, quantified by ||P_s(I-P_w)||.

  2. On Weak-to-Strong Generalization and f-Divergence

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Replacing cross-entropy with f-divergence losses in weak-to-strong generalization gives modest accuracy gains and improved label-noise tolerance, though the paper's theoretical equivalence result is constructed after ...

  3. Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss

    cs.LG 2025-01 conditional novelty 6.0 of 10

    For convex and approximately convex model classes, the loss gain in weak-to-strong learning is at least the KL misfit between strong and weak models, plus an error term that vanishes as k grows.

  4. Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges

    cs.LG 2025-02 conditional novelty 5.0 of 10

    Iterative self-training on a model's own correct outputs, with simple length and voting filters, lets transformers generalize to far longer arithmetic and path-finding problems than they saw in training.

  5. Super Co-alignment of Human and AI for Sustainable Symbiotic Society

    cs.AI 2025-04 unverdicted novelty 4.0 of 10

    The authors propose 'Super Co-alignment', in which humans and superintelligent AI iteratively co-evolve shared values through external oversight and intrinsic empathy-based alignment.

Reference graph

Works this paper leans on

69 extracted references · 38 canonical work pages · cited by 5 Pith papers

  1. [1]

    Estimating example difficulty using variance of gradients

    Chirag Agarwal, Daniel D'souza, and Sara Hooker. Estimating example difficulty using variance of gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 10368--10378, 2022

    2022

  2. [2]

    Self-training: A survey, 2023

    Massih-Reza Amini, Vasilii Feofanov, Loic Pauletto, Lies Hadjadj, Emilie Devijver, and Yury Maximov. Self-training: A survey, 2023

    work page 2023

  3. [3]

    Perplexed by perplexity: Perplexity-based data pruning with small reference models

    Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity-based data pruning with small reference models. arXiv preprint arXiv:2405.20541, 2024

    arXiv 2024

  4. [4]

    Pseudo-labeling and confirmation bias in deep semi-supervised learning

    Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International joint conference on neural networks (IJCNN), pp.\ 1--8. IEEE, 2020

    work page 2020

  5. [5]

    Finite-time analysis of the multiarmed bandit problem, 2002

    P Auer. Finite-time analysis of the multiarmed bandit problem, 2002

    2002

  6. [6]

    Deep learning through the lens of example difficulty

    Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34: 0 10876--10889, 2021

    work page 2021

  7. [7]

    going on a vacation

    Qiang Ning Ben Zhou, Daniel Khashabi and Dan Roth. “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding. In EMNLP, 2019

    work page 2019

  8. [8]

    Combining labeled and unlabeled data with co-training

    Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory, 1998

    work page 1998

Show all 69 references
  1. [9]

    Weak-to-strong generalization: Eliciting strong capabilities with weak supervision

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023

    2023 arXiv

  2. [10]

    Compression, generalization and learning

    Marco C Campi and Simone Garatti. Compression, generalization and learning. Journal of Machine Learning Research, 24 0 (339): 0 1--74, 2023

    2023

  3. [11]

    Quantifying the gain in weak-to-strong generalization

    Moses Charikar, Chirag Pabbaraju, and Kirankumar Shiragur. Quantifying the gain in weak-to-strong generalization. arXiv preprint arXiv:2405.15116, 2024

    2024 arXiv

  4. [12]

    Xgboost: A scalable tree boosting system

    Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp.\ 785--794, 2016

    2016

  5. [13]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computati...

    2019

  6. [14]

    Analysis of perceptron-based active learning

    Sanjoy Dasgupta, Adam Tauman Kalai, and Claire Monteleoni. Analysis of perceptron-based active learning. In International conference on computational learning theory, pp.\ 249--263. Springer, 2005

    2005

  7. [15]

    On the foundations of noise-free selective classification

    Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. JMLR, 11: 0 1605–1641, aug 2010. ISSN 1532-4435

    2010

  8. [16]

    Weak to strong generalization: Some theoretical perspectives

    EleutherAI. Weak to strong generalization: Some theoretical perspectives. https://blog.eleuther.ai/weak-to-strong/, 2021. Accessed: 2024-09-26

    2021

  9. [17]

    Fu, Mayee F

    Daniel Y. Fu, Mayee F. Chen, Frederic Sala, Sarah M. Hooper, Kayvon Fatahalian, and Christopher R\'e. Fast and three-rious: Speeding up weak supervision with triplet methods. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020), 2020

    2020

  10. [18]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022

    2022 arXiv

  11. [19]

    Selective classification for deep neural networks

    Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems, volume 30, 2017

    2017

  12. [20]

    Data shapley: Equitable valuation of data for machine learning

    Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning, pp.\ 2242--2251. PMLR, 2019

    2019

  13. [21]

    Theory of disagreement-based active learning

    Steve Hanneke. Theory of disagreement-based active learning. Found. Trends Mach. Learn., 7 0 (2–3): 0 131–309, jun 2014. ISSN 1935-8237

    2014

  14. [22]

    Cosmos qa: Machine reading comprehension with contextual commonsense reasoning

    Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference ...

    2019

  15. [23]

    Data quality: A statistical perspective

    Alan F Karr, Ashish P Sanil, and David L Banks. Data quality: A statistical perspective. Statistical Methodology, 3 0 (2): 0 137--173, 2006

    2006

  16. [24]

    Looking beyond the surface: A challenge set for reading comprehension over multiple sentences

    Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computatio...

    2018

  17. [25]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp.\ 1885--1894. PMLR, 2017

    2017

  18. [26]

    Theoretical analysis of weak-to-strong generalization, 2024

    Hunter Lang, David Sontag, and Aravindan Vijayaraghavan. Theoretical analysis of weak-to-strong generalization, 2024. URL https://arxiv.org/abs/2405.16043

    2024 arXiv

  19. [27]

    Bandit algorithms

    Tor Lattimore and Csaba Szepesv \'a ri. Bandit algorithms. Cambridge University Press, 2020

    2020

  20. [28]

    Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks

    Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop on Challenges in Representation Learning, 2013

    2013

  21. [29]

    Datacomp-lm: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794, 2024

    2024 arXiv

  22. [30]

    Ling, Jun Du, and Zhi-Hua Zhou

    Charles X. Ling, Jun Du, and Zhi-Hua Zhou. When does co-training work in real data? In Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD) 2009, 2009

    2009

  23. [31]

    Teacher–student curriculum learning

    Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher–student curriculum learning. IEEE Transactions on Neural Networks and Learning Systems, 2020

    2020

  24. [32]

    TSATC: Twitter Sentiment Analysis Training Corpus

    Ibrahim Naji. TSATC: Twitter Sentiment Analysis Training Corpus . In thinknook, 2012

    2012

  25. [33]

    Learning with noisy labels

    Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26, 2013

    2013

  26. [34]

    Adversarial nli: A new benchmark for natural language understanding

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational L...

    2020

  27. [35]

    An overview of deep semi-supervised learning

    Yassine Ouali, C \'e line Hudelot, and Myriam Tami. An overview of deep semi-supervised learning. arXiv preprint arXiv:2006.05278, 2020

    2006 arXiv

  28. [36]

    Campi, and Simone Garatti

    Dario Paccagnan, Marco C. Campi, and Simone Garatti. The pick-to-learn algorithm: empowering compression for tight generalization bounds and improved post-training performance. NIPS '23. Curran Associates Inc., 2024

    2024

  29. [37]

    Wic: the word-in-context dataset for evaluating context-sensitive meaning representations

    Mohammad Taher Pilehvar and Jose Camacho-Collados. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te...

    2019

  30. [38]

    A. J. Ratner, Christopher M. De Sa, Sen Wu, Daniel Selsam, and C. R\' e . Data programming: Creating large training sets, quickly. In Proceedings of the 29th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, 2016

    2016

  31. [39]

    Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher R\' e

    Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher R\' e . Snorkel: Rapid training data creation with weak supervision. In Proceedings of the 44th International Conference on Very Large Data Bases (VLDB), Rio de Janeiro, Brazil, 2018

    2018

  32. [40]

    Getting closer to AI complete question answering: A set of prerequisite real tasks

    Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. Getting closer to AI complete question answering: A set of prerequisite real tasks. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artifici...

    2020

  33. [41]

    Social iqa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural ...

    2019

  34. [42]

    H. Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 1965

    1965

  35. [43]

    Dissecting sample hardness: A fine-grained analysis of hardness characterization methods for data-centric ai

    Nabeel Seedat, Fergus Imrie, and Mihaela van der Schaar. Dissecting sample hardness: A fine-grained analysis of hardness characterization methods for data-centric ai. arXiv preprint arXiv:2403.04551, 2024

    2024 arXiv

  36. [44]

    On tests for detecting change in mean

    Ashish Sen and Muni S Srivastava. On tests for detecting change in mean. The Annals of statistics, pp.\ 98--108, 1975

    1975

  37. [45]

    Universalizing weak supervision

    Changho Shin, Winfred Li, Harit Vishwakarma, Nicholas Carl Roberts, and Frederic Sala. Universalizing weak supervision. In International Conference on Learning Representations, 2022

    2022

  38. [46]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Proces...

    2013

  39. [47]

    A statistical framework for weak-to-strong generalization

    Seamus Somerstep, Felipe Maia Polo, Moulinath Banerjee, Ya'acov Ritov, Mikhail Yurochkin, and Yuekai Sun. A statistical framework for weak-to-strong generalization. arXiv preprint arXiv:2405.16236, 2024

    2024 arXiv

  40. [48]

    Dream: A challenge data set and models for dialogue-based reading comprehension

    Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. Dream: A challenge data set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 7: 0 217--231, 2019

    2019

  41. [49]

    Easy-to-hard generalization: Scalable alignment beyond human supervision

    Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-hard generalization: Scalable alignment beyond human supervision. arXiv preprint arXiv:2403.09472, 2024

    2024 arXiv

  42. [50]

    quartz: An open-domain dataset of qualitative relationship questions

    Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. "quartz: An open-domain dataset of qualitative relationship questions". "2019"

    2019

  43. [51]

    Selective review of offline change point detection methods

    Charles Truong, Laurent Oudre, and Nicolas Vayatis. Selective review of offline change point detection methods. Signal Processing, 167: 0 107299, 2020

    2020

  44. [52]

    High-dimensional statistics: A non-asymptotic viewpoint, volume 48

    Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019

    2019

  45. [53]

    Neural network acceptability judgments

    Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471, 2018

    2018 arXiv

  46. [54]

    Theoretical analysis of self-training with deep networks on unlabeled data

    Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In International Conference on Learning Representations, 2022

    2022

  47. [55]

    Crowdsourcing multiple choice science questions

    Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pp.\ 94--106, 2017

    2017

  48. [56]

    Qurating: Selecting high-quality data for training language models

    Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.09739, 2024

    2024 arXiv

  49. [57]

    Less: Selecting influential data for targeted instruction tuning

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. In Forty-first International Conference on Machine Learning, 2024

    2024

  50. [58]

    Data selection for language models via importance resampling

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 0 34201--34227, 2023

    2023

  51. [59]

    An empirical study of gpt-3 for few-shot knowledge-based vqa

    Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In AAAI, 2022

    2022

  52. [60]

    Data valuation using reinforcement learning

    Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference on Machine Learning, pp.\ 10842--10851. PMLR, 2020

    2020

  53. [61]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4800, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4800, 2019

    2019

  54. [62]

    Wrench: A comprehensive benchmark for weak supervision

    Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner. Wrench: A comprehensive benchmark for weak supervision. arXiv preprint arXiv:2109.11377, 2021

    2021 arXiv

  55. [63]

    Character-level convolutional networks for text classification

    Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015

    2015

  56. [64]

    PAWS: Paraphrase Adversaries from Word Scrambling

    Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase Adversaries from Word Scrambling . In Proc. of NAACL, 2019

    2019

  57. [65]

    Introduction to semi-supervised learning

    Xiaojin Zhu and Andrew B Goldberg. Introduction to semi-supervised learning. Springer Nature, 2022

    2022

  58. [66]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  59. [67]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  60. [68]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  61. [69]

    Then train f_ weak on \ P_w x_i, y_i\ _i , the projection of the dataset onto specifically the weak features

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.