Pith. sign in

REVIEW 4 cited by

A transfer learning framework for weak-to-strong generalization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.16236 v3 pith:HYDLLURU submitted 2024-05-25 stat.ML cs.LG

Seamus Somerstep , Felipe Maia Polo , Moulinath Banerjee , Ya'acov Ritov , Mikhail Yurochkin , Yuekai Sun This is my paper · ORCID
classification stat.MLcs.LG
keywords modelgeneralizationproblemweak-to-strongapproachcapabilitiesfeedbackllms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Modern large language model (LLM) alignment techniques rely on human feedback, but it is unclear whether these techniques fundamentally limit the capabilities of aligned LLMs. In particular, it is unknown if it is possible to align (stronger) LLMs with superhuman capabilities with (weaker) human feedback without degrading their capabilities. This is an instance of the weak-to-strong generalization problem: using feedback from a weaker (less capable) model to train a stronger (more capable) model. We prove that weak-to-strong generalization is possible by eliciting latent knowledge from pre-trained LLMs. In particular, we cast the weak-to-strong generalization problem as a transfer learning problem in which we wish to transfer a latent concept prior from a weak model to a strong pre-trained model. We prove that a naive fine-tuning approach suffers from fundamental limitations, but an alternative refinement-based approach suggested by the problem structure provably overcomes the limitations of fine-tuning. Finally, we demonstrate the practical applicability of the refinement approach in multiple LLM alignment tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions

    cs.LG 2025-02 conditional novelty 8.0 of 10

    Weak-to-strong performance is governed by the overlap between the weak model's unlearnable error space and the strong model's principal-representation space, quantified by ||P_s(I-P_w)||.

  2. On Weak-to-Strong Generalization and f-Divergence

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Replacing cross-entropy with f-divergence losses in weak-to-strong generalization gives modest accuracy gains and improved label-noise tolerance, though the paper's theoretical equivalence result is constructed after ...

  3. Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss

    cs.LG 2025-01 conditional novelty 6.0 of 10

    For convex and approximately convex model classes, the loss gain in weak-to-strong learning is at least the KL misfit between strong and weak models, plus an error term that vanishes as k grows.

  4. The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibration

    cs.LG 2025-02 reject novelty 5.0 of 10

    The paper derives generalization and calibration bounds for weak-to-strong generalization and extends a known regression result from squared loss to KL divergence.

Pith tools