{"id":"a5739b1d-725c-4768-8971-0283b8d37398","arxiv_id":"2505.19107","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"OFA tunes LayerNorm parameters as optimization preconditioners and adds step-ratio and sharpness penalties, reporting consistent few-shot accuracy gains over baselines on Llama and GPT-2 models.","lead":"This paper introduces OFA, a method for adapting large language models (LLMs) to new tasks with only a few examples. It tunes only the LayerNorm parameters, framed as learned preconditioners in an optimization view of the forward pass, and adds two penalties to encourage fast convergence and flat minima. The authors report consistent accuracy gains over existing few-shot baselines on several LLMs, with no added inference cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OFA's central claim lacks a final-accuracy ablation: Figure 1 reports only per-layer probe metrics for CE, CE+step-ratio, CE+sharpness, and OFA, so the contribution of the two regularizers to the Table 1 gains is untested.","rationale":"The reader's weakest_assumption focused on the theoretical equivalence between LayerNorm and a true preconditioned gradient descent update (Section 3.2), and on the attention-output difference as a gradient proxy in the Hutchinson estimator (Eq. 4). That is a legitimate concern, but it is not the most load-bearing for the paper's empirical claim: even if the optimization-inspired interpretation is imperfect, the method could still work, and the paper's own limitation section (Section G) concedes that a rigorous theoretical foundation is future work. The more decisive gap is that the ablation evidence for the two regularizers is indirect. Figure 1 reports probe-based per-layer accuracy/loss, not the final task accuracies that Table 1 is about, and Figures 2 and 3 report the very quantities being minimized. Consequently, we cannot tell whether the reported gains come from the proposed objective or simply from tuning LayerNorm with cross-entropy. This is a concrete, testable omission in the empirical core of the paper. Since the reader already assigned a CONDITIONAL verdict, my concern does not change the verdict; it identifies a specific missing experiment that should be required before acceptance, and it partially agrees with the reader's emphasis on rigor while shifting focus from the theoretical framing to the empirical attribution.","tokens_in":19278,"tokens_out":6560,"duration_ms":63492,"concrete_test":"Run the identical Table 1 protocol (same 9 datasets, Llama2-7B and Llama3-8B-Instruct, same 5 seeds) for four variants: CE-only, CE + step-ratio (Eq. 3), CE + sharpness (Eq. 4), and full OFA (Eq. 5), all tuning only LayerNorm parameters. Select lambda1 and lambda2 on a validation split (e.g., 20% of the demonstration set) rather than on the test set, and report final test accuracy for each variant. If full OFA does not exceed CE-only by more than the seed-level standard deviation, the central claim that the optimization-inspired penalties drive the improvement is unsupported; if it does, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the OFA objective in Eq. 5—cross-entropy plus the step-ratio and sharpness penalties—produces the consistent improvements in Table 1. For this claim to hold, each added penalty must actually improve end-task accuracy. The manuscript never tests this directly: the ablation in Figure 1 shows per-layer probe accuracy and loss for CE, CE+step-ratio, CE+sharpness, and OFA, but no corresponding final classification accuracies are given. Figures 2 and 3 only demonstrate that OFA lowers the sharpness and step-ratio quantities that the objective explicitly minimizes; lower values of these quantities are not evidence of better classification, since they are the optimization targets themselves. The causal attribution of Table 1's gains to the regularizers is therefore unsupported. This gap is compounded by the absence of released code and of the selected lambda1/lambda2 values, so test-set hyperparameter selection cannot be excluded. If CE-only LayerNorm tuning already matches OFA's accuracy, the proposed objective is unnecessary for the headline result; if CE-only is much worse, the regularizers' role is confirmed. The current text does not distinguish these possibilities.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Optimization-Inspired Few-Shot Adaptation (OFA), a method that treats selected layers of a transformer's forward pass as preconditioned gradient descent steps and tunes LayerNorm scaling parameters as learnable diagonal preconditioners. The training objective, Eq. (5), combines cross-entropy with a step-ratio penalty (Eq. 3) and a Hutchinson-estimated sharpness penalty (Eq. 4). The paper reports experiments on four decoder-only LLMs and nine classification datasets, claiming consistent accuracy improvements over ICL and PEFT baselines, including a 4%-10% gain over I2CL on Llama2-7B and Llama3-8B-Instruct, with no additional inference cost.","tokens_in":19467,"tokens_out":2934,"duration_ms":29878,"significance":"If the empirical claims hold, OFA is a practically attractive contribution: it achieves few-shot adaptation with about 0.27M trainable parameters and no inference-time overhead, and the comparison across four model families and nine datasets is broad and internally consistent. The paper also ships readable ablations of per-layer probe accuracy, sharpness, and step ratio, which help connect the proposed objective terms to the optimization story. However, the significance is currently limited by two gaps: the theoretical theorems are not rigorously proven, and the causal role of the two regularizers for the final test accuracy is not directly established.","major_comments":[{"comment":"The central claim that the Eq. (5) objective produces the Table 1 gains is not supported by a final-accuracy ablation. Figure 1 reports only per-layer probe accuracy and probe loss for CE, CE+step-ratio, CE+sharpness, and OFA; no table or paragraph gives the end-task classification accuracy for these four objective variants. Since Figures 2 and 3 show lower values of the very quantities that Eq. (5) explicitly minimizes, those plots cannot by themselves establish improved generalization. I request an accuracy table (with standard deviations over the same five seeds) for CE, CE+step-ratio, CE+sharpness, and OFA on at least the main benchmark sets, or an explicit statement if the gains in Table 1 are driven by cross-entropy alone.","section":"Section 4, Figure 1 vs. Table 1"},{"comment":"The Hutchinson estimator in Eq. (4) uses the identification Pt∇L(Zt) = Zt+1 − Zt (Algorithm 1, step 6), i.e., the residual output of an attention layer is treated as the preconditioned gradient of a well-defined loss. This is inherited from linearized-transformer results, but the paper does not address the mean-subtraction term in LayerNorm, and the attention output difference is not shown to equal a true gradient for the actual nonlinear transformer. Because the sharpness penalty and Theorem 3.2 both depend on this proxy, I ask for either a formal justification under the paper's assumptions or an empirical validation (e.g., comparing the estimator against a finite-difference gradient on a small linearized model). Without this, the theoretical motivation for the sharpness term is fragile even if the method works empirically.","section":"Section 3.4, Eq. (4) and Algorithm 1"},{"comment":"The proof of Theorem 3.1 is not a rigorous derivation. It asserts a local quadratic approximation, then states that 'minimizing J(P) ensures ρt decreases over time' without showing how the step-ratio objective in Eq. (3) controls the spectral radius of I − ηPtHt. In particular, Eq. (3) involves ratios of consecutive update norms and does not explicitly contain η, Pt, or Ht; the step from minimizing that ratio to reducing ρt is not established. The theorem's conclusion about 'faster local contraction and improved convergence' is therefore unsupported as written. Either supply a complete proof with precise assumptions connecting Eq. (3) to the spectral radius, or present the step-ratio penalty as a heuristic and adjust the claims accordingly.","section":"Appendix B, Theorem 3.1"},{"comment":"The generalization bound in Theorem 3.2 is not proven. The proof consists of informal Taylor and PAC-Bayes statements: no stability bound is defined, no Rademacher complexity or PAC-Bayes argument is actually instantiated, and the step from 'Σ is shaped by optimization history through {∆t}' to the final O(·) bound is a gap. Additionally, the statement bounds E[Ltest(ZT) − Ltrain(ZT)] but the derivation concerns perturbations of the weights, not the sampling of training data, and the final expression depends on the training Hessian without explaining how that controls the test gap. The theorem should either be replaced by a precise statement with a complete proof or be explicitly labeled as an informal motivation rather than a proven result.","section":"Appendix C, Theorem 3.2"},{"comment":"The manuscript reports a grid-search pool for λ1 and λ2 (Table 7) but does not state the selected values per dataset/model or the selection criterion. Since the main results are reported on test sets, it is important to clarify whether hyperparameters were chosen on a held-out validation split, by cross-validation, or by test-set performance; the same applies to the LoRA baseline tuning described in Appendix E. The absence of this information, together with no released code, makes it difficult to rule out test-set hyperparameter selection and to reproduce the results.","section":"Section 4 and Appendix F, hyperparameter selection"}],"minor_comments":[{"comment":"The notation is inconsistent: Eq. (3) is written with Zt, while the proof of Theorem 3.1 switches to xt without defining the relation between the two. Please unify the notation.","section":"Eq. (3) and Appendix B"},{"comment":"The objective uses 'lCE (F (Z0))' but the subscript CE is not defined at that point (it is defined only in Figure 1). Also define the Softplus function δ(·) explicitly.","section":"Section 3.4, Eq. (5)"},{"comment":"The table reports 0 'introduced parameters' for OFA, which is correct for inference-time overhead but potentially misleading because OFA does train 0.27M LayerNorm parameters during adaptation. Please clarify in the caption or table that the measure is inference-time additional context parameters, not total trainable parameters.","section":"Table 3"},{"comment":"References [34] and [35] are duplicates of the same work (Liu et al., 'Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning'). Please merge or differentiate them.","section":"References"},{"comment":"The caption says 'AGNews, Subj, and TREC' but the body text in Section 4 says 'AGNews, Subj'; please align the caption with the actual panels.","section":"Figure 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper has a plausible empirical story and a useful practical contribution if the ablation and hyperparameter issues are resolved. However, the two theorems are currently more decorative than load-bearing, and the missing final-accuracy ablation for the objective components is a substantive gap that a revision must fill. I did not find evidence of misconduct, but the absence of selected hyperparameter values and code is a reproducibility risk for a paper whose claims depend on grid-searched regularizers. Please weigh the theoretical claims carefully in the revision outcome."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth taking seriously. The method is simple and the empirical story is broad: tuning only LayerNorm parameters, with no added inference cost, beats ICL and PEFT baselines across four LLMs and nine datasets. The LoRA comparison is also fair and is a real point in favor. The combination of LayerNorm-as-preconditioner, the step-ratio penalty, and the Hutchinson sharpness penalty is genuinely new as far as I can tell, and the probe analysis is a nice attempt to show per-layer effects. Credit where it's due: the experiments are consistent, with multiple seeds, and the method's premise is clearly stated.\n\nNow the soft spots, in proportion. The stress-test note is exactly right: the ablation in Figure 1 shows only per-layer probe accuracy and loss for CE, CE+step-ratio, CE+sharpness, and OFA, but never reports the final classification accuracy for each variant. That means we cannot attribute the Table 1 gains to the two regularizers. It is entirely possible that CE-only LayerNorm tuning already gets most of the benefit, and the penalties are cosmetic. Figures 2 and 3 do not resolve this because they plot the very quantities the objective minimizes; those plots are evidence of optimization, not generalization. The theory also does not carry much weight. Theorem 3.1 is a one-paragraph heuristic, and Theorem 3.2's proof jumps from a Taylor expansion to a PAC-Bayes bound without making the connection clear. The LayerNorm-as-preconditioner formula silently drops the mean subtraction term, which is a known fragile point in the linearized-transformer literature.\n\nReproducibility is a practical concern: no code is released, and although the hyperparameter pool is listed, the selected values for lambda1, lambda2, epsilon, and N are not reported. For a paper whose main selling point is empirical, that is a serious omission. The abstract's claim that OFA 'overcomes both issues' of ICL and PEFT is an overclaim, though the text is somewhat more careful later. The reader's conditional verdict is fair.\n\nWho is this for? Researchers working on few-shot adaptation and PEFT will want to read it and, if the code appears, try it. It does not reshape the field, but it is a plausible, cheap alternative. My recommendation: send it to peer review, but require a proper ablation with final accuracies, released code, and a more honest framing of the theoretical results. I would not cite it as a dependable result until those pieces are in place.","headline":"Solid empirical package with a genuinely new regularizer combination, but the paper undersells its own evidential gap: the ablation never reports final accuracies for the individual objective terms, so the headline gains are not yet attributed to the method's core components.","tokens_in":20050,"tokens_out":1523,"would_cite":false,"duration_ms":16105,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tuning only LayerNorm parameters beats in-context learning on few-shot tasks.","keywords":["few-shot adaptation","in-context learning","parameter-efficient fine-tuning","LayerNorm preconditioning","preconditioned gradient descent","sharpness minimization","randomized trace estimation","large language models"],"falsifier":"Train OFA on a linear transformer where the true gradient and Hessian are exactly computable, and compare the randomized trace proxy with exact preconditioned curvature; if the proxy fails to reproduce the experimental gains, or if removing LayerNorm's mean-subtraction term changes the result, the load-bearing equivalence is falsified.","tokens_in":18962,"feed_emoji":"🎯","tokens_out":5201,"duration_ms":44124,"temperature":0.7,"pith_summary":"The paper claims that the forward pass of an LLM can be viewed not just as gradient descent but as preconditioned gradient descent, and that the LayerNorm scale vectors are the preconditioners. On this view, few-shot adaptation reduces to tuning only those LayerNorm parameters with an objective that penalizes uneven step sizes across layers and the trace of the preconditioned Hessian, steering the internal trajectory into flat regions of the loss landscape. The authors report consistent 4%-10% accuracy gains over the leading implicit-in-context-learning baseline on nine classification benchmarks across Llama2-7B, Llama3-8B, Llama3-8B-Instruct, and GPT2-XL, with no added inference cost and only 0.27M trainable parameters.","feed_headline":"LayerNorm tuning beats in-context learning on few-shot tasks","feed_subtitle":"Adapting only 0.27M LayerNorm parameters lifts Llama2-7B and Llama3-8B accuracy 4–10% over the leading baseline.","key_machinery":"The central object is the LayerNorm-as-preconditioner parameterization $P_t=\\Gamma_t\\cdot 1/\\sigma_t$, where $\\Gamma_t=\\operatorname{diag}(\\gamma_t)$ is the learnable LayerNorm scale vector. The argument is carried by two objective terms: the step-ratio penalty (Eq. 3), which uses the ratio of consecutive layer-output differences to favor smooth contraction, and the sharpness penalty (Eq. 4), which estimates the trace of the preconditioned Hessian with Gaussian perturbations and a Softplus regularizer. The LayerNorm parameters are the only trainable parameters, so no extra parameters are introduced at inference, and the sharpness estimate requires only forward passes because the attention output difference $Z_{t+1}-Z_t$ is used as a proxy for $P_t\\nabla L(Z_t)$.","core_discovery":"The central claim is that you can steer the implicit optimization an LLM performs during in-context learning by learning layer-wise diagonal preconditioning matrices, and that LayerNorm parameters are exactly such matrices. The paper shows that treating each attention layer as one preconditioned gradient step, with $Z_{t+1}=Z_t-P_t\\nabla L(Z_t)$ and $P_t=\\Gamma_t/\\sigma_t$, lets the LayerNorm scaling vector $\\Gamma_t$ act as the optimizer's preconditioner. The proposed objective combines cross-entropy with a step-ratio penalty $\\sum_t \\|Z_t-Z_{t+1}\\|/\\|Z_t-Z_{t-1}\\|$ that promotes contraction, and a sharpness penalty using a randomized trace estimator of $\\operatorname{tr}(P_t\\nabla^2 L(Z_t)P_t^\\top)$ that promotes flat minima. The authors argue theoretically that the step-ratio term reduces the spectral radius of $I-\\eta P_t H_t$ and that the sharpness term bounds the generalization gap by $\\sqrt{(1/n)\\sum_t \\|P_t\\nabla^2 L_{\\text{train}}(Z_t)\\|_F^2}$. Empirically, tuning only the LayerNorm parameters this way outperforms both PEFT (including LoRA at ranks 1-128) and ICL-based methods, with the same inference cost as zero-shot.","pith_inferences":["If the preconditioner interpretation is correct, the same LayerNorm-tuning recipe could apply to encoder-only and vision transformers, and to generation tasks beyond classification, though the paper only tests decoder-only classification.","The use of $Z_{t+1}-Z_t$ as a gradient proxy suggests a cheaper family of sharpness-aware objectives that avoid backpropagation through the Hessian; a testable extension is to compare this proxy against exact finite-difference gradients on a linearized transformer.","The learned $\\Gamma_t$ values could be interpreted as adaptive per-layer learning rates, potentially unifying OFA with standard fine-tuning optimizers and suggesting a way to derive LayerNorm updates from optimizer state rather than from a separate loss.","A direct comparison on regression-style in-context learning tasks, where the implicit loss is known exactly, would clarify whether the optimizer-equivalence assumption or the sharpness regularizer is responsible for the observed gains."],"forward_implications":["On Llama2-7B and Llama3-8B-Instruct, OFA improves accuracy by 4-10% over the I2CL baseline on all nine benchmarks tested, with the largest gains on Subj and DBPedia.","Adaptation requires only 0.27M trainable parameters, about half the size of a rank-1 LoRA adapter, and no additional inference-time demonstration storage or compute.","Layer-wise probe analysis shows that features produced by OFA become linearly separable earlier in the network, and final layers exhibit lower sharpness and smoother step ratios than the cross-entropy baseline.","The method transfers across decoder-only architectures without the need for model-specific layer selection, unlike task-vector based approaches.","The sharpness regularizer can be evaluated with only forward passes, making the approach scalable to large LLMs where Hessian-based sharpness estimation would be prohibitive."],"supporting_citations":[{"why":"Establishes that transformers can implement preconditioned gradient descent for in-context learning, providing the basis for treating LayerNorm as a learnable preconditioner.","marker":"[3]"},{"why":"Shows that transformers learn in-context by gradient descent, supplying the core equivalence used to frame each attention layer as an optimization step.","marker":"[56]"},{"why":"Analyzes in-context learning in a linear transformer block with one-step GD initialization, giving theoretical support for the gradient-descent interpretation.","marker":"[64]"},{"why":"Supplies the randomized trace estimator used in Eq. 4 to approximate the preconditioned Hessian trace without explicit Hessian computation.","marker":"[2]"},{"why":"Provides the I2CL baseline and the evaluation protocol that OFA follows, and is the state-of-the-art method that OFA claims to beat by 4-10%.","marker":"[33]"},{"why":"Introduces LoRA, the main PEFT baseline used in the parameter-efficiency comparison tables.","marker":"[25]"},{"why":"Identifies the overfitting problem of PEFT in few-shot settings and provides the FPET method, framing the motivation for avoiding PEFT in low-data regimes.","marker":"[35]"},{"why":"Discusses the relationship between sharp minima and generalization, used in the proof of the generalization bound in Theorem 3.2.","marker":"[16]"},{"why":"Provides PAC-Bayesian bounds that the proof of Theorem 3.2 relies on to relate preconditioned curvature to the generalization gap.","marker":"[40]"},{"why":"Supplies the PAC-Bayes-based argument used to turn the accumulated preconditioned curvature into a generalization bound.","marker":"[18]"}],"fun_headline_variants":["Few-shot: tune LayerNorm, beat ICL and PEFT","Optimization-inspired LLM adaptation: just LayerNorm","LayerNorm as preconditioner: few-shot beats ICL","Don't tune weights: few-shot with LayerNorm only"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The forward pass of a real transformer, including LayerNorm with its mean subtraction, is accurately modeled as preconditioned gradient descent on a well-defined loss, so that the LayerNorm scale vector genuinely acts as a diagonal preconditioner and the attention output difference $Z_{t+1}-Z_t$ is a valid proxy for the gradient in the sharpness estimator.","fun_headline_variants_meta":{"raw":{"variants":["Few-shot: tune LayerNorm, beat ICL and PEFT","Optimization-inspired LLM adaptation: just LayerNorm","LayerNorm as preconditioner: few-shot beats ICL","Don't tune weights: few-shot with LayerNorm only"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000627,"raw_usage":{"total_tokens":2945,"prompt_tokens":1037,"completion_tokens":1908,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":1840}},"tokens_in":653,"tokens_out":1908,"duration_ms":13158,"temperature":1.0,"reasoning_tokens":1840,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:20:17.803357+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train OFA on a linear transformer where the true gradient and Hessian are exactly computable, and compare the randomized trace proxy with exact preconditioned curvature; if the proxy fails to reproduce the experimental gains, or if removing LayerNorm's mean-subtraction term changes the result, the load-bearing equivalence is falsified.","supporting_citations":[{"cited_title":"Transformers learn to implement preconditioned gradient descent for in-context learning","cited_arxiv_id":null,"evidence_quote":"Establishes that transformers can implement preconditioned gradient descent for in-context learning, providing the basis for treating LayerNorm as a learnable preconditioner."},{"cited_title":"Transformers learn in-context by gradient descent","cited_arxiv_id":null,"evidence_quote":"Shows that transformers learn in-context by gradient descent, supplying the core equivalence used to frame each attention layer as an optimization step."},{"cited_title":"In-context learning of a linear transformer block: benefits of the mlp component and one-step gd initialization","cited_arxiv_id":null,"evidence_quote":"Analyzes in-context learning in a linear transformer block with one-step GD initialization, giving theoretical support for the gradient-descent interpretation."},{"cited_title":"Second-order stochastic optimization for machine learning in linear time","cited_arxiv_id":null,"evidence_quote":"Supplies the randomized trace estimator used in Eq. 4 to approximate the preconditioned Hessian trace without explicit Hessian computation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the I2CL baseline and the evaluation protocol that OFA follows, and is the state-of-the-art method that OFA claims to beat by 4-10%."},{"cited_title":"Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning","cited_arxiv_id":null,"evidence_quote":"Identifies the overfitting problem of PEFT in few-shot settings and provides the FPET method, framing the motivation for avoiding PEFT in low-data regimes."},{"cited_title":"Sharp minima can generalize for deep nets","cited_arxiv_id":null,"evidence_quote":"Discusses the relationship between sharp minima and generalization, used in the proof of the generalization bound in Theorem 3.2."},{"cited_title":"A PAC-bayesian approach to spectrally-normalized margin bounds for neural networks","cited_arxiv_id":null,"evidence_quote":"Provides PAC-Bayesian bounds that the proof of Theorem 3.2 relies on to relate preconditioned curvature to the generalization gap."}],"review_version":1}