Pith. sign in

REVIEW 4 cited by

How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.08391 v2 pith:6JZGEL67 submitted 2023-10-12 stat.ML cs.LG

Jingfeng Wu , Difan Zou , Zixiang Chen , Vladimir Braverman , Quanquan Gu , Peter L. Bartlett This is my paper · ORCID
classification stat.MLcs.LG
keywords tasksmodelpretraininglinearregressionattentionbayesin-context
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Transformers pretrained on diverse tasks exhibit remarkable in-context learning (ICL) capabilities, enabling them to solve unseen tasks solely based on input contexts without adjusting model parameters. In this paper, we study ICL in one of its simplest setups: pretraining a linearly parameterized single-layer linear attention model for linear regression with a Gaussian prior. We establish a statistical task complexity bound for the attention model pretraining, showing that effective pretraining only requires a small number of independent tasks. Furthermore, we prove that the pretrained model closely matches the Bayes optimal algorithm, i.e., optimally tuned ridge regression, by achieving nearly Bayes optimal risk on unseen tasks under a fixed context length. These theoretical findings complement prior experimental research and shed light on the statistical foundations of ICL.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling

    cs.LG 2025-10 conditional novelty 6.0 of 10

    When a cosine schedule would halve the learning rate, Seesaw cuts it by √2 and doubles the batch, matching loss curves with ~36% fewer serial steps.

  2. How Can Mamba Learn In Context with Outliers and Generalize Provably?

    cs.LG 2025-10 conditional novelty 6.0 of 10

    A simplified one-layer Mamba provably learns in-context binary classification tolerating outlier fractions approaching 1, whereas a linear Transformer can only tolerate α < 1/2.

  3. Towards Theoretical Understanding of Transformer Test-Time Computing: Investigation on In-Context Linear Regression

    cs.LG 2025-08 conditional novelty 6.0 of 10

    A one-layer linear-attention transformer implementing noisy gradient descent gives provable bounds showing linear noise plus ensembling avoids label-noise overfitting, and majority voting beats greedy decoding in spar...

  4. Re-examining learning linear functions in context

    cs.LG 2024-11 conditional novelty 6.0 of 10

    Transformer models trained from scratch on in-context linear function prediction learn boundary-limited interpolation, not general linear regression.

Pith tools