Pith. sign in

REVIEW 5 major objections 5 minor 67 references

Optimization-Inspired Few-Shot Adaptation for Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Tuning only LayerNorm parameters beats in-context learning on few-shot tasks.

desk verdict Solid empirical package with a genuinely new regularizer combination, but the paper undersells its own evidential gap: the ablation never reports final accuracies for the individual objective terms, so the headline gains are not yet attributed to the method's core components. read the letter →

arxiv 2505.19107 v1 pith:KTDH73EB submitted 2025-05-25 cs.LG

Boyan Gao , Xin Wang , Yibo Yang , David Clifton This is my paper · ORCID
classification cs.LG
keywords few-shotadaptationin-contextlearningparameter-efficientfine-tuningLayerNormpreconditioningpreconditionedgradientdescentsharpnessminimizationrandomizedtraceestimationlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the forward pass of an LLM can be viewed not just as gradient descent but as preconditioned gradient descent, and that the LayerNorm scale vectors are the preconditioners. On this view, few-shot adaptation reduces to tuning only those LayerNorm parameters with an objective that penalizes uneven step sizes across layers and the trace of the preconditioned Hessian, steering the internal trajectory into flat regions of the loss landscape. The authors report consistent 4%-10% accuracy gains over the leading implicit-in-context-learning baseline on nine classification benchmarks across Llama2-7B, Llama3-8B, Llama3-8B-Instruct, and GPT2-XL, with no added inference cost and only 0.27M trainable parameters.

What carries the argument

The central object is the LayerNorm-as-preconditioner parameterization $P_t=\Gamma_t\cdot 1/\sigma_t$, where $\Gamma_t=\operatorname{diag}(\gamma_t)$ is the learnable LayerNorm scale vector. The argument is carried by two objective terms: the step-ratio penalty (Eq. 3), which uses the ratio of consecutive layer-output differences to favor smooth contraction, and the sharpness penalty (Eq. 4), which estimates the trace of the preconditioned Hessian with Gaussian perturbations and a Softplus regularizer. The LayerNorm parameters are the only trainable parameters, so no extra parameters are introduced at inference, and the sharpness estimate requires only forward passes because the attention output difference $Z_{t+1}-Z_t$ is used as a proxy for $P_t\nabla L(Z_t)$.

What would settle it

Train OFA on a linear transformer where the true gradient and Hessian are exactly computable, and compare the randomized trace proxy with exact preconditioned curvature; if the proxy fails to reproduce the experimental gains, or if removing LayerNorm's mean-subtraction term changes the result, the load-bearing equivalence is falsified.

Watch

Extended reading notes

Core claim

The central claim is that you can steer the implicit optimization an LLM performs during in-context learning by learning layer-wise diagonal preconditioning matrices, and that LayerNorm parameters are exactly such matrices. The paper shows that treating each attention layer as one preconditioned gradient step, with $Z_{t+1}=Z_t-P_t\nabla L(Z_t)$ and $P_t=\Gamma_t/\sigma_t$, lets the LayerNorm scaling vector $\Gamma_t$ act as the optimizer's preconditioner. The proposed objective combines cross-entropy with a step-ratio penalty $\sum_t \|Z_t-Z_{t+1}\|/\|Z_t-Z_{t-1}\|$ that promotes contraction, and a sharpness penalty using a randomized trace estimator of $\operatorname{tr}(P_t\nabla^2 L(Z_t)P_t^\top)$ that promotes flat minima. The authors argue theoretically that the step-ratio term reduces the spectral radius of $I-\eta P_t H_t$ and that the sharpness term bounds the generalization gap by $\sqrt{(1/n)\sum_t \|P_t\nabla^2 L_{\text{train}}(Z_t)\|_F^2}$. Empirically, tuning only the LayerNorm parameters this way outperforms both PEFT (including LoRA at ranks 1-128) and ICL-based methods, with the same inference cost as zero-shot.

Load-bearing premise

The forward pass of a real transformer, including LayerNorm with its mean subtraction, is accurately modeled as preconditioned gradient descent on a well-defined loss, so that the LayerNorm scale vector genuinely acts as a diagonal preconditioner and the attention output difference $Z_{t+1}-Z_t$ is a valid proxy for the gradient in the sharpness estimator.

Editorial extensions

If this is right

  • On Llama2-7B and Llama3-8B-Instruct, OFA improves accuracy by 4-10% over the I2CL baseline on all nine benchmarks tested, with the largest gains on Subj and DBPedia.
  • Adaptation requires only 0.27M trainable parameters, about half the size of a rank-1 LoRA adapter, and no additional inference-time demonstration storage or compute.
  • Layer-wise probe analysis shows that features produced by OFA become linearly separable earlier in the network, and final layers exhibit lower sharpness and smoother step ratios than the cross-entropy baseline.
  • The method transfers across decoder-only architectures without the need for model-specific layer selection, unlike task-vector based approaches.
  • The sharpness regularizer can be evaluated with only forward passes, making the approach scalable to large LLMs where Hessian-based sharpness estimation would be prohibitive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the preconditioner interpretation is correct, the same LayerNorm-tuning recipe could apply to encoder-only and vision transformers, and to generation tasks beyond classification, though the paper only tests decoder-only classification.
  • The use of $Z_{t+1}-Z_t$ as a gradient proxy suggests a cheaper family of sharpness-aware objectives that avoid backpropagation through the Hessian; a testable extension is to compare this proxy against exact finite-difference gradients on a linearized transformer.
  • The learned $\Gamma_t$ values could be interpreted as adaptive per-layer learning rates, potentially unifying OFA with standard fine-tuning optimizers and suggesting a way to derive LayerNorm updates from optimizer state rather than from a separate loss.
  • A direct comparison on regression-style in-context learning tasks, where the implicit loss is known exactly, would clarify whether the optimizer-equivalence assumption or the sharpness regularizer is responsible for the observed gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript proposes Optimization-Inspired Few-Shot Adaptation (OFA), a method that treats selected layers of a transformer's forward pass as preconditioned gradient descent steps and tunes LayerNorm scaling parameters as learnable diagonal preconditioners. The training objective, Eq. (5), combines cross-entropy with a step-ratio penalty (Eq. 3) and a Hutchinson-estimated sharpness penalty (Eq. 4). The paper reports experiments on four decoder-only LLMs and nine classification datasets, claiming consistent accuracy improvements over ICL and PEFT baselines, including a 4%-10% gain over I2CL on Llama2-7B and Llama3-8B-Instruct, with no additional inference cost.

Significance. If the empirical claims hold, OFA is a practically attractive contribution: it achieves few-shot adaptation with about 0.27M trainable parameters and no inference-time overhead, and the comparison across four model families and nine datasets is broad and internally consistent. The paper also ships readable ablations of per-layer probe accuracy, sharpness, and step ratio, which help connect the proposed objective terms to the optimization story. However, the significance is currently limited by two gaps: the theoretical theorems are not rigorously proven, and the causal role of the two regularizers for the final test accuracy is not directly established.

major comments (5)
  1. [Section 4, Figure 1 vs. Table 1] The central claim that the Eq. (5) objective produces the Table 1 gains is not supported by a final-accuracy ablation. Figure 1 reports only per-layer probe accuracy and probe loss for CE, CE+step-ratio, CE+sharpness, and OFA; no table or paragraph gives the end-task classification accuracy for these four objective variants. Since Figures 2 and 3 show lower values of the very quantities that Eq. (5) explicitly minimizes, those plots cannot by themselves establish improved generalization. I request an accuracy table (with standard deviations over the same five seeds) for CE, CE+step-ratio, CE+sharpness, and OFA on at least the main benchmark sets, or an explicit statement if the gains in Table 1 are driven by cross-entropy alone.
  2. [Section 3.4, Eq. (4) and Algorithm 1] The Hutchinson estimator in Eq. (4) uses the identification Pt∇L(Zt) = Zt+1 − Zt (Algorithm 1, step 6), i.e., the residual output of an attention layer is treated as the preconditioned gradient of a well-defined loss. This is inherited from linearized-transformer results, but the paper does not address the mean-subtraction term in LayerNorm, and the attention output difference is not shown to equal a true gradient for the actual nonlinear transformer. Because the sharpness penalty and Theorem 3.2 both depend on this proxy, I ask for either a formal justification under the paper's assumptions or an empirical validation (e.g., comparing the estimator against a finite-difference gradient on a small linearized model). Without this, the theoretical motivation for the sharpness term is fragile even if the method works empirically.
  3. [Appendix B, Theorem 3.1] The proof of Theorem 3.1 is not a rigorous derivation. It asserts a local quadratic approximation, then states that 'minimizing J(P) ensures ρt decreases over time' without showing how the step-ratio objective in Eq. (3) controls the spectral radius of I − ηPtHt. In particular, Eq. (3) involves ratios of consecutive update norms and does not explicitly contain η, Pt, or Ht; the step from minimizing that ratio to reducing ρt is not established. The theorem's conclusion about 'faster local contraction and improved convergence' is therefore unsupported as written. Either supply a complete proof with precise assumptions connecting Eq. (3) to the spectral radius, or present the step-ratio penalty as a heuristic and adjust the claims accordingly.
  4. [Appendix C, Theorem 3.2] The generalization bound in Theorem 3.2 is not proven. The proof consists of informal Taylor and PAC-Bayes statements: no stability bound is defined, no Rademacher complexity or PAC-Bayes argument is actually instantiated, and the step from 'Σ is shaped by optimization history through {∆t}' to the final O(·) bound is a gap. Additionally, the statement bounds E[Ltest(ZT) − Ltrain(ZT)] but the derivation concerns perturbations of the weights, not the sampling of training data, and the final expression depends on the training Hessian without explaining how that controls the test gap. The theorem should either be replaced by a precise statement with a complete proof or be explicitly labeled as an informal motivation rather than a proven result.
  5. [Section 4 and Appendix F, hyperparameter selection] The manuscript reports a grid-search pool for λ1 and λ2 (Table 7) but does not state the selected values per dataset/model or the selection criterion. Since the main results are reported on test sets, it is important to clarify whether hyperparameters were chosen on a held-out validation split, by cross-validation, or by test-set performance; the same applies to the LoRA baseline tuning described in Appendix E. The absence of this information, together with no released code, makes it difficult to rule out test-set hyperparameter selection and to reproduce the results.
minor comments (5)
  1. [Eq. (3) and Appendix B] The notation is inconsistent: Eq. (3) is written with Zt, while the proof of Theorem 3.1 switches to xt without defining the relation between the two. Please unify the notation.
  2. [Section 3.4, Eq. (5)] The objective uses 'lCE (F (Z0))' but the subscript CE is not defined at that point (it is defined only in Figure 1). Also define the Softplus function δ(·) explicitly.
  3. [Table 3] The table reports 0 'introduced parameters' for OFA, which is correct for inference-time overhead but potentially misleading because OFA does train 0.27M LayerNorm parameters during adaptation. Please clarify in the caption or table that the measure is inference-time additional context parameters, not total trainable parameters.
  4. [References] References [34] and [35] are duplicates of the same work (Liu et al., 'Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning'). Please merge or differentiate them.
  5. [Figure 3 caption] The caption says 'AGNews, Subj, and TREC' but the body text in Section 4 says 'AGNews, Subj'; please align the caption with the actual panels.

Circularity Check

2 steps flagged · score 4.0 of 10

OFA's mechanistic checks measure the exact terms its objective minimizes, but the headline accuracy comparisons are not circular.

  1. self definitional [Section 3.4, Eq. (5); Section 4, 'Layer-wise Sharpness Analysis' (Figure 2)]
    "Ψ(P ) = lCE (F (Z0)) + λ1 ... λ2 ... δ(tr(Pt∇2L(Zt)P T t )) ... One can observe that in Figure 2, the model trained by OFA consistently illustrates the lowest sharpness among the baseline models across all the layers."

    Eq. (5) directly minimizes the sharpness penalty λ2 δ(tr(Pt∇2L(Zt)P T t )), and the analysis section then measures that same preconditioned-Hessian trace (Eq. 4) and reports that OFA has the lowest value. Because the CE and base-model baselines do not include this penalty, their higher sharpness is a direct consequence of the objective definitions, not an independent empirical discovery about generalization. The paper presents this as confirmation that OFA reaches flat minima, but the measurement is the optimization target itself; without end-task accuracy for the individual penalty terms, the flatness story is not independently tested.

  2. self definitional [Section 3.3, Eq. (3); Section 4, 'Layer-wise Step Ratio Analysis' (Figure 3)]
    "J (P ) = T −1X t=1 ∥Zt − Zt+1∥/∥Zt − Zt−1∥ ... Notably, optimizing the step-ratio objective in OFA results in smoother and more consistent contraction across layers, highlighting the effectiveness of our learned preconditioning mechanism."

    The step-ratio J(P) is defined in Eq. (3) and then added to the learning objective with weight λ1 in Eq. (5). The step-ratio analysis therefore compares a model trained to minimize J(P) against models that do not contain this term, and observes that J(P) is smaller for OFA. This is a restatement of the training objective, not a prediction about convergence or generalization. The paper cites the smoother profiles as evidence of improved optimization efficiency, but no held-out convergence metric or final-accuracy ablation for the ablated objectives is supplied, so the figure cannot distinguish a genuine benefit from the tautology that optimizing a term reduces that term.

full rationale

The headline results (Table 1 and Table 4) are genuine external evaluations: the same LayerNorm-preconditioner tuning is scored on held-out classification benchmarks against ICL, PEFT, and task-vector baselines, and those accuracy numbers do not follow by construction from the OFA objective. The theoretical framing that transformer attention layers implement gradient descent is imported from unrelated prior work [3, 56, 64]; there is no load-bearing self-citation chain, and the LayerNorm-as-preconditioner parameterization is an architectural choice rather than a fitted quantity. The real circular component is confined to the mechanistic validation: Eq. (5) explicitly minimizes the step-ratio term (Eq. 3) and the preconditioned-Hessian trace (Eq. 4), and the 'Layer-wise Sharpness Analysis' and 'Layer-wise Step Ratio Analysis' then present lower values of those same two quantities as evidence that OFA improves flatness and optimization efficiency. Those measurements are restatements of the optimization objective and are tautological as support for the regularizers' value. In addition, the paper reports no final-accuracy ablation for CE, CE+step-ratio, CE+sharpness, and OFA, so the causal attribution of Table 1's gains to the two penalty terms is not directly established; this is an experimental-support gap rather than a circular derivation, and it does not push the score higher by itself.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method's central claim relies primarily on existing results (transformers as gradient descent) and on the paper's own approximations (LayerNorm as preconditioner, attention output as gradient). No new physical or mathematical entities are introduced.

free parameters (4)
  • lambda1 = tuned per dataset via grid search (Appendix F)
    Regularization weight for the step-ratio objective in Eq. 5; the paper does not report the selected value per dataset.
  • lambda2 = tuned per dataset via grid search (Appendix F)
    Regularization weight for the sharpness objective in Eq. 5; selected values not reported.
  • epsilon = small scale number, value not reported in main text
    Noise scale in the Hutchinson approximation (Eq. 4); affects the bias-variance tradeoff of the Hessian trace estimate.
  • N = number of Hutchinson samples; not reported in main text
    Controls the variance of the trace estimator in Algorithm 1.
assumptions (5)
  • domain assumption The forward pass of an attention layer implements one step of gradient descent on a loss defined over the prompt (cited from prior work).
    Section 3.1 adopts Eq. 2 from [3, 56, 64]; this is not proven for real LLMs and is approximate.
  • ad hoc to paper LayerNorm's operation can be absorbed into a diagonal preconditioner with the mean subtraction ignored.
    Section 3.2 writes Z_{t+1} = Z_t - Gamma_t grad L(Z_t) - mu_t/sigma_t and drops the -mu_t/sigma_t term when defining P_t.
  • standard math The loss is locally quadratic with locally Lipschitz gradients (Theorem 3.1) and smooth with bounded Hessian and bounded gradients (Theorem 3.2).
    Assumed for the proofs in Appendices B and C without empirical verification.
  • ad hoc to paper The attention output difference Z_{t+1} - Z_t is a valid proxy for the gradient grad L(Z_t) used in the Hutchinson estimator.
    Algorithm 1 uses this substitution to compute P_t grad L(Z_t + epsilon P_t nu), which equates attention output differences with gradients.
  • ad hoc to paper The stated generalization bound follows from stability-based generalization bounds and Taylor expansion.
    Appendix C's proof of Theorem 3.2 is a sketch and does not complete the stability or PAC-Bayes argument.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimization-Inspired Few-Shot Adaptation for Large Language Models." pith.science (2026). https://pith.science/paper/KTDH73EB

@misc{pith2026250519107,
  author       = {Pith},
  title        = {Pith review of: Optimization-Inspired Few-Shot Adaptation for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KTDH73EB}},
  note         = {Machine review of arXiv:2505.19107}
}
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable performance in real-world applications. However, adapting LLMs to novel tasks via fine-tuning often requires substantial training data and computational resources that are impractical in few-shot scenarios. Existing approaches, such as in-context learning and Parameter-Efficient Fine-Tuning (PEFT), face key limitations: in-context learning introduces additional inference computational overhead with limited performance gains, while PEFT models are prone to overfitting on the few demonstration examples. In this work, we reinterpret the forward pass of LLMs as an optimization process, a sequence of preconditioned gradient descent steps refining internal representations. Based on this connection, we propose Optimization-Inspired Few-Shot Adaptation (OFA), integrating a parameterization that learns preconditioners without introducing additional trainable parameters, and an objective that improves optimization efficiency by learning preconditioners based on a convergence bound, while simultaneously steering the optimization path toward the flat local minimum. Our method overcomes both issues of ICL-based and PEFT-based methods, and demonstrates superior performance over the existing methods on a variety of few-shot adaptation tasks in experiments.

Figures

Figures reproduced from arXiv: 2505.19107 by the authors.

Figure 1
Figure 1. Probe Analysis on EMO, SST, and TREC. The layer-wise prediction accuracy (%) and loss [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Sharpness comparison on MR, Subj and TREC. The average sharpness over the test samples [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Step ratio comparison across the test sets of AGNews, Subj, and TREC over each layer of [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 51 canonical work pages

  1. [1]

    A mechanism for sample-efficient in-context learning for sparse retrieval tasks

    Jacob Abernethy, Alekh Agarwal, Teodor Vanislavov Marinov, and Manfred K Warmuth. A mechanism for sample-efficient in-context learning for sparse retrieval tasks. In ALT, 2024

    work page 2024

  2. [2]

    Second-order stochastic optimization for machine learning in linear time

    Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research, 18(116):1–40, 2017

    work page 2017

  3. [3]

    Transformers learn to implement preconditioned gradient descent for in-context learning

    Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. In NeurIPS, 2023

    work page 2023

  4. [4]

    What learning algorithm is in-context learning? investigations with linear models

    Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In ICLR, 2023

    work page 2023

  5. [5]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022

    2022

  6. [6]

    Infinite mixture prototypes for few-shot learning

    Kenneth Allen, Evan Shelhamer, Joshua B Tenenbaum, and Trevor Darrell. Infinite mixture prototypes for few-shot learning. In ICML, 2019

    work page 2019

  7. [7]

    Transformers as statisticians: Provable in-context learning with in-context algorithm selection

    Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. In NeurIPS, 2023

    work page 2023

  8. [8]

    Peyman Bateni, Hadi Ghasemzadeh, Farnood Barati, Amir Gholami, Kurt Keutzer, Trevor Darrell, Ali Farhadi, and Farzan F. Dabaghi. Improved few-shot visual classification. In CVPR, 2020

    work page 2020

Show all 67 references
  1. [9]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Deirdre Jiang, Sergey Levine, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

    2023 arXiv

  2. [10]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020

    2020

  3. [11]

    Semeval- 2019 task 3: Emocontext contextual emotion detection in text

    Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi, and Puneet Agrawal. Semeval- 2019 task 3: Emocontext contextual emotion detection in text. In Proceedings of the 13th international workshop on semantic evaluation, pp. 39–48, 2019

    2019

  4. [12]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. In NeurIPS, 2021

    2021

  5. [13]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christo- pher Hesse, and John Schulman. Training verifiers to solve math word problems. In NeurIPS, 2021

    2021

  6. [14]

    Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers

    Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. arXiv preprint arXiv:2212.10559, 2022

    2022 arXiv

  7. [15]

    Hate speech dataset from a white supremacy forum

    Ona de Gibert, Naiara Perez, Aitor García-Pablos, and Montse Cuadros. Hate speech dataset from a white supremacy forum. arXiv preprint arXiv:1809.04444, 2018

    2018 arXiv

  8. [16]

    Sharp minima can generalize for deep nets

    Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In ICML, 2017

    2017

  9. [17]

    Incorporat- ing second-order functional knowledge for better option pricing

    Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incorporat- ing second-order functional knowledge for better option pricing. In NeurIPS, 2000

    2000

  10. [18]

    Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data

    Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017. 10

    2017 arXiv

  11. [19]

    Model-agnostic meta-learning for fast adapta- tion of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In ICLR, 2017

    2017

  12. [20]

    Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell

    Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. In ICLR, 2020

    2020

  13. [21]

    Sharpness-aware mini- mization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware mini- mization for efficiently improving generalization. In ICLR, 2021

    2021

  14. [22]

    Transformers are universal in-context learners

    Takashi Furuya, Maarten V de Hoop, and Gabriel Peyré. Transformers are universal in-context learners. arXiv preprint arXiv:2408.01367, 2024

    2024 arXiv

  15. [23]

    The impact of initialization on lora finetuning dynamics

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. The impact of initialization on lora finetuning dynamics. In NeurIPS, 2024

    2024

  16. [24]

    In-context learning creates task vectors

    Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In EMNLP, 2023

    2023

  17. [25]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022

    2022

  18. [26]

    Transformers are minimax optimal nonparametric in-context learners

    Juno Kim, Tai Nakamaki, and Taiji Suzuki. Transformers are minimax optimal nonparametric in-context learners. In NeurIPS, 2024

    2024

  19. [27]

    Asam: Adaptive sharpness-aware minimization for scale-invariant learning

    Yugeun Kwon, Junbeom Kim, and Jaehong Yun. Asam: Adaptive sharpness-aware minimization for scale-invariant learning. In NeurIPS, 2021

    2021

  20. [28]

    Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia

    Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6 (2):...

    2015

  21. [29]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  22. [30]

    In-context learning state vector with inner and momentum optimization

    Dongfang Li, Xinshuo Hu, Zetian Sun, Baotian Hu, Min Zhang, et al. In-context learning state vector with inner and momentum optimization. In NeurIPS, 2024

    2024

  23. [31]

    Learning question classification with support vector machines, 2002

    Xin Li and Dan Roth. Learning question classification with support vector machines, 2002. https://cogcomp.seas.upenn.edu/Data/QA/QC/

    2002

  24. [32]

    Meta-sgd: Learning to learn quickly for few-shot learning

    Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09835, 2017

    2017 arXiv

  25. [33]

    Zhuowei Li, Zihao Xu, Ligong Han, Yunhe Gao, Song Wen, Di Liu, Hao Wang, and Dimitris N. Metaxas. Implicit in-context learning. In ICLR, 2025

    2025

  26. [35]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In NeurIPS, 2022

    2022

  27. [36]

    DoRA: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In ICML, 2024

    2024

  28. [37]

    Codet: Code generation with generated tests

    Shuyan Lu, Bowen Fu, Ziniu Zhang, Wenpeng Yin, Hongxia Zhao, Xin Geng, Yizhou Sun, and Yizhou Wu. Codet: Code generation with generated tests. In ICLR, 2024

    2024

  29. [38]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Amir Yazdanbakhsh, Huan Wu, Shiyue Yao, Amir Gholami, Kurt Keutzer, and Wen-mei Hwu. Self-refine: Iterative refinement with self-feedback. In NeurIPS, 2023. 11

    2023

  30. [39]

    Pissa: Principal singular values and singular vectors adaptation of large language models

    Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. In NeurIPS, 2024

    2024

  31. [40]

    A PAC-bayesian approach to spectrally-normalized margin bounds for neural networks

    Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A PAC-bayesian approach to spectrally-normalized margin bounds for neural networks. In ICLR, 2018

    2018

  32. [41]

    Reptile: a scalable metalearning algorithm

    Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999, 2018

    2018 arXiv

  33. [42]

    A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts

    Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In ACL, 2004

    2004

  34. [43]

    Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales

    Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, 2005

    2005

  35. [44]

    Meta-curvature

    Eunbyung Park and Junier B Oliva. Meta-curvature. In NeurIPS, 2019

    2019

  36. [45]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog , 1(8),

  37. [46]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  38. [47]

    Meta-learning with implicit gradients

    Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In NeurIPS, 2019

    2019

  39. [48]

    Robocat: A self-improving foundation agent for robotics

    Scott Reed, Diego de Las Casas, Yuxuan Lu, Emilio Parisotto, Andre Barreto, et al. Robocat: A self-improving foundation agent for robotics. arXiv preprint arXiv: 2305.19328, 2023

    2023 arXiv

  40. [49]

    Large language models encode clinical knowledge

    Karan Singhal, Shekoofeh Azizi, Tien-Ju Tu, Soroush Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Anil Tanwani, Hunter Cole, Jemin Lee, et al. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138, 2022

    2022 arXiv

  41. [50]

    Prototypical networks for few-shot learning

    Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurIPS, 2017

    2017

  42. [51]

    Recursive deep models for semantic compositionality over a sentiment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013

    2013

  43. [52]

    Learning to compare: Relation network for few-shot learning

    Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In CVPR, 2018

    2018

  44. [53]

    de la Cerda, John Dodson, Kyle Kosic, Thomas Lavril, Matthew Leavitt, Jenia Jitsev, and Gabriel Lample

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Y-Lan Boureau, Vishwavirat B. de la Cerda, John Dodson, Kyle Kosic, Thomas Lavril, Matthew Leavitt, Jenia Jitsev, and Gabriel Lample. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arX...

    2023 arXiv

  45. [54]

    Llama 3: Open foundation and instruction-tuned language models

    Hugo Touvron, Louis Martin, Kevin Lu, Kamel Benjelloun, Myle Ott, Marc-Alexandre Côté, Sam Shleifer, Thomas Wang, Tianyi Zhang, Edouard Grave, Angela Fan, Luke Zettlemoyer, and Guillaume Lample. Llama 3: Open foundation and instruction-tuned language models. arXiv preprint arX...

    2024 arXiv

  46. [55]

    Matching networks for one shot learning

    Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NeurIPS), 2016

    2016

  47. [56]

    Transformers learn in-context by gradient descent

    Johannes V on Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In ICML, 2023

    2023

  48. [57]

    Label words are anchors: An information flow perspective for understanding in-context learning

    Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In EMNLP, 2023. 12

    2023

  49. [58]

    Do prompt-based models really understand the meaning of their prompts? In NAACL, 2022

    Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts? In NAACL, 2022

    2022

  50. [59]

    Chain of thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In NeurIPS, 2022

    2022

  51. [60]

    How many pretraining tasks are needed for in-context learning of linear regression? In ICLR, 2024

    Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? In ICLR, 2024

    2024

  52. [61]

    Corda: Context-oriented decomposition adaptation of large language models for task-aware parameter-efficient fine-tuning

    Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. Corda: Context-oriented decomposition adaptation of large language models for task-aware parameter-efficient fine-tuning. In NeurIPS, 2024

    2024

  53. [62]

    Free lunch for few-shot learning: Distribution calibration

    Yingxiang Yang, Zhi Zhang, Timothy Hospedales, and Tao Xiang. Free lunch for few-shot learning: Distribution calibration. In ICLR, 2021

    2021

  54. [63]

    Improving generalization by controlling label-noise information in neural network weights

    Kaidi Zhang, Ruijia Zhang, Behnam Neyshabur, and Olivier Bousquet. Improving generalization by controlling label-noise information in neural network weights. In ICLR, 2022

    2022

  55. [64]

    In-context learning of a linear transformer block: benefits of the mlp component and one-step gd initialization

    Ruiqi Zhang, Jingfeng Wu, and Peter Bartlett. In-context learning of a linear transformer block: benefits of the mlp component and one-step gd initialization. In NeurIPS, 2024

    2024

  56. [65]

    Character-level convolutional networks for text classification

    Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NeurIPS, 2015

    2015

  57. [66]

    Calibrate before use: Improving few-shot performance of language models

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In ICML, 2021

    2021

  58. [67]

    Dvornek, Sekhar Tatikonda, James S

    Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha C. Dvornek, Sekhar Tatikonda, James S. Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. In ICLR, 2022. 13 A Few-shot performance We report the entire few-shot performanc...

    2022

  59. [2019]

    https://cdn.openai.com/better-language-models/language_models_are_ unsupervised_multitask_learners.pdf

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.