Pith. sign in

REVIEW 3 major objections 4 minor 46 references

LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that repeated backbone backpropagation in LLM adaptation can be replaced by a one-time calibration: after fitting low-rank feedback maps from final-layer error to blockwise hidden-state corrections, adapters are tuned by…

desk verdict A cleanly derived forward-only tuning method with honest scoping, but the unmeasured calibration-drift assumption and missing synthetic-gradient baseline keep it a strong revise-and-resubmit rather than an accept. read the letter →

arxiv 2608.03020 v2 pith:AXJ673CR submitted 2026-08-04 cs.AI

Linhan Xia , Rui Liu , Zhaofeng Zhang , Yihao Wang , Binrui Shen , Shengxin Zhu This is my paper · ORCID
classification cs.AI
keywords backpropagation-freetrainingfeedbackalignmentparameter-efficientfine-tuningforward-onlytuninglow-rankadaptersridgeregressionlargelanguagemodelsmemory-efficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that repeated end-to-end backpropagation through a frozen LLM backbone can be replaced by a one-time calibration. LoCA fits a low-rank feedback map per transformer block, from the final prediction error to a blockwise hidden-state correction, using a single probe backward pass. After that, each outer iteration runs one forward pass, forms blockwise regression targets, and solves closed-form ridge problems for low-rank adapters without any backbone backward pass. On five discriminative tasks across Qwen2.5 models from 0.5B to 14B, LoCA reports lower evaluation cross-entropy than LoRA in 16 of 25 task-scale cells, with 26-29 percent lower full-run GPU peak memory and 36-52 percent lower post-calibration CPU steady-state memory. The paper itself restricts the claim to small-shift adaptation and treats the cell-level counts as descriptive rather than as evidence of general superiority.

What carries the argument

The load-bearing object is the calibrated low-rank feedback operator $F_\ell$, a rank-8 truncation of a ridge fit of hidden-state gradients on top-layer error, together with the closed-form per-block ridge solve $B_\ell^* = C_\ell(G_\ell + \lambda I_r)^{-1}$. $F_\ell$ converts the frozen head's prediction error into a blockwise hidden-state correction without differentiating through the backbone, and the solve then fits the adapter to that correction. The streaming statistics $G_\ell$ and $C_\ell$ are additive over batches and require $O(L(r^2+dr))$ storage, so the per-pass cost after calibration is essentially one forward pass plus small linear solves.

What would settle it

Run LoCA on a task where LoRA's improvement over the frozen model is large, then measure the blockwise cosine alignment $\cos\angle(F_\ell e, g_\ell)$ at outer iterations 1, 5, 20, and 40. If the alignment drops to near zero or negative at the point where held-out cross-entropy stops improving while a LoRA run continues to improve, the one-shot calibration assumption is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that a rank-8 truncated ridge fit of the map from top-layer error $e = W_u^\top(\mathrm{softmax}(z)-\mathrm{onehot}(y))$ to exact hidden-state gradients, computed once per block, carries enough credit information for subsequent forward-only adapter tuning. LoCA's outer loop computes targets $\tau_\ell = h_\ell - \bar{\eta} F_\ell e$ with $\bar{\eta} = \eta\, \mathrm{RMS}(h_{\ell-1})/\mathrm{RMS}(F_\ell e)$, then solves $B_\ell^* = C_\ell(G_\ell+\lambda I_r)^{-1}$. Because the block objective is a strictly convex quadratic for fixed targets, each solve is exact for its local objective; the paper does not claim global cross-entropy decreases monotonically and selects snapshots by held-out CE. The fitted maps show in-probe cosine alignment 0.3-0.5 with true hidden-state gradients, versus near zero for random feedback, which is the quantitative reason the local targets are informative. The method therefore amortizes global credit assignment into one calibration and leaves a backward-free outer loop.

Load-bearing premise

The whole method rests on the assumption that one-time calibration stays valid: as adapters update, the fitted feedback maps must keep pointing roughly toward the true hidden-state gradients, and the paper concedes that alignment can degrade when tuning moves far from the calibrated state.

Editorial extensions

If this is right

  • Post-calibration tuning can run on hardware without backward support, since each outer iteration uses only a forward pass, the frozen-head error, and local linear solves.
  • Compared with LoRA's backward-based loop, LoCA's measured full-run GPU peak memory is 26-29 percent lower, while its post-calibration CPU steady-state memory and per-pass time are 36-52 percent and 43-48 percent lower.
  • The same scale-normalized target-size candidate set transfers across Qwen2.5 sizes from 0.5B to 14B and to SmolLM2-1.7B, with held-out selection inside the set.
  • Because the local ridge solve is exact for its fixed target, LoCA has no learning-rate schedule or optimizer state inside a block update, although the paper limits this claim to the local objective rather than to global convergence.
  • LoCA can return the frozen model as a fallback, protecting the selected validation cross-entropy when no adapted snapshot improves on it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the fitted feedback maps degrade predictably with distance from calibration, a cheap recalibration trigger based on the residual between realized adapter displacements and target displacements could extend LoCA to larger shifts than the paper tests.
  • Because the per-block solves can be computed independently for a shared forward pass, the method may distribute naturally across devices that cannot run a global backward pass.
  • The reported BoolQ pattern, in which LoRA lowers cross-entropy while ranking accuracy drops and LoCA preserves accuracy, suggests the local-target bias may act as a regularizer against completion-format overfitting; that hypothesis is testable by comparing validation CE and ranking accuracy across datasets with and without format shift.
  • A direct comparison against a version that recomputes the feedback maps every few outer iterations would quantify how much of the method's value comes from the one-shot fit rather than from the local solve structure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces LoCA, a two-stage method for parameter-efficient fine-tuning of large language models. In the first stage, a single backward pass through the frozen backbone fits low-rank feedback operators mapping the top-layer error to per-block hidden-state corrections. In the second stage, these operators are fixed, and each low-rank adapter is updated by solving a local ridge regression in closed form, requiring only forward passes through the backbone. The authors evaluate LoCA on Qwen2.5 models from 0.5B to 14B and on SmolLM2-1.7B, reporting that LoCA achieves lower evaluation cross-entropy than the corresponding LoRA run in 16 of 25 task-scale comparisons, with 26–29% lower GPU peak memory including calibration. The paper is careful to describe most cells as single runs and to state that the results are descriptive rather than statistical evidence of general superiority.

Significance. If validated, LoCA offers a practical forward-only tuning regime with meaningful memory savings, and it occupies a distinct niche between zeroth-order methods and feedback alignment. The mathematical core is sound: the ridge solutions in Theorems E.1 and F.1 are derived cleanly and exactly, and Proposition D.1 makes the descent claim conditional on explicit alignment and smoothness assumptions rather than asserting unconditional convergence. The paper ships code, gives a detailed appendix of derivations, and is unusual in explicitly labeling the limits of its own evidence. The main weaknesses are empirical: the central small-shift assumption is never directly measured over the adaptation trajectory, and the headline comparisons rest on single runs without variance estimates.

major comments (3)
  1. [Section 3.2 and Section 5] The central load-bearing assumption of LoCA is that the once-fitted rank-8 feedback maps F_l maintain sufficient alignment with the true hidden-state gradients throughout the up to 40 outer iterations. The paper reports only in-probe cosine alignment of 0.3–0.5 and acknowledges in Section 5 that "the approximation may weaken when tuning moves far from the calibration point," but no measurement is provided of how the alignment evolves over the adaptation trajectory. Without such a measurement, the forward-only claim rests on an unquantified small-shift assumption that could fail in practice. Please add a small-scale diagnostic, for example computing the cosine alignment alpha_l of Appendix D between F_l e and the exact hidden-state gradient G_l at several outer iterations on at least one task/model, or provide a quantitative bound on the shift within which the fixed feedback remains reliable.
  2. [Section 4.1 and Table 1] All main benchmark cells are single runs at seed 0, and no confidence intervals are reported. The paper explicitly states that the 16/25 count is descriptive, but this count is the primary evidence that LoCA is "not merely an inaccurate copy of the LoRA update." Without any measure of variability, it is impossible to judge whether a 16/25 split is distinguishable from chance or from run-to-run noise. Please provide variance estimates for the headline comparison, for example by running 3 seeds on a representative subset of the 25 cells or by reporting bootstrap intervals for the recovery ratios. This is load-bearing because the qualitative conclusion that LoCA follows a different optimization path depends on the pattern of wins being a stable signal rather than noise.
  3. [Section 5 and Appendix H] The "small-shift regime" is never defined quantitatively, despite being the scope condition for the method. The paper repeatedly refers to "small, targeted changes" but does not state a measurable threshold on, for example, parameter distance, activation norm change, or loss reduction within which LoCA is expected to be valid. Without an operational definition, the method's applicability is not falsifiable. Please provide a concrete shift metric or stability condition that can be checked in experiments, or report the measured shift in the evaluated settings so that the reader can connect the empirical results to the assumed regime.
minor comments (4)
  1. [Introduction and Discussion] The text contains placeholder citations marked "[?]" where references are missing, specifically for preference alignment in the introduction and for variance-reduced MeZO variants in the Discussion. These should be completed before publication.
  2. [Table 1] The column header "LoCAR CE Racc" appears to be a formatting error; it should likely read "LoCA R_CE R_acc" or similar. The table layout also makes it difficult to distinguish the model-size blocks; consider clearer vertical spacing or rule lines.
  3. [Appendix C] Equation (24) sums over t from s to K for the positionwise gradient. It would be helpful to state explicitly that this relies on the causal mask making the Jacobian zero for t < s, as this justifies the sum over later positions only.
  4. [Section 3.3] The phrase "This solve is exact for a fixed local target" is correct but could be misread as a global claim. Consider adding a brief sentence clarifying that "exact" means the exact minimizer of the local ridge quadratic, not an exact solution to the global cross-entropy objective.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: LoCA's closed-form solves are exact only for their own local targets, and the LoRA-relative CE results are measured empirically, not forced by construction.

full rationale

LoCA's derivation chain is self-contained. The feedback operator F_l is fitted by ridge regression in Eq. (5) from probe gradients, the targets tau_l are defined in Eq. (4) via that fitted operator, and the closed-form adapter solve in Eq. (8) minimizes the resulting local quadratic. None of these quantities is defined in terms of the evaluation cross-entropy reported in Table 1, and the paper explicitly disclaims global descent: "This solve is exact for a fixed local target. It does not imply that global cross-entropy decreases at every outer iteration." The 16-of-25 lower-CE comparison with LoRA is a measured result, not a consequence of the equations by construction. The appendix theorems (E.1 and F.1) are proven in the paper rather than imported from self-citation, and the reference list contains no load-bearing citations to the authors' own prior work. The paper itself flags the main limitations—fixed feedback may weaken far from calibration, calibration latency was not recorded separately, and most cells are single runs—but these are honest scope restrictions, not circular reasoning. The only arguably close concern is that the feedback map is calibrated on the same task distribution used for adaptation, but that is ordinary supervised calibration on training data and does not force the held-out evaluation outcome. No specific reduction of a claimed prediction to its fitted input can be exhibited, so the appropriate score is 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. Its free parameters are the target-size coefficient eta, ridge regularization lambda, feedback ridge beta, ranks r and k, and the outer iteration count T. The main domain assumption is that the model stays in a small-shift regime, because the fitted feedback operators are only locally valid. The theoretical results are standard convex optimization statements and a descent lemma under unverified smoothness.

free parameters (6)
  • eta = selected from {0.003, 0.01, 0.03} on held-out CE
    Target-size hyperparameter that scales the feedback correction in Eq. (10).
  • lambda = not specified
    Ridge regularization coefficient in Eq. (8) and (51); value is never reported in the paper.
  • beta = not specified
    Ridge regularization for the feedback fit in Eq. (5); value is never reported.
  • feedback rank k = 8
    Retained rank of the fitted feedback map in Eq. (47).
  • adapter rank r = 32
    Low-rank adapter dimension used for all experiments.
  • outer iterations T = at most 40, selected by held-out CE
    Number of forward-only outer iterations.
assumptions (4)
  • domain assumption Adaptation is a small shift from the calibrated model state, so a fixed linear feedback map remains a useful credit signal.
    Invoked when Eq. (4) replaces the true gradient g_l with F_l e; Section 3.2 notes alignment may degrade away from the calibrated state.
  • domain assumption The probe backward pass provides accurate per-token gradients g_l for the frozen model on the adaptation loss.
    The fitted F_l in Eq. (5) is only as good as these gradient targets; the paper reports in-probe cosine alignment of 0.3 to 0.5 for the fitted map, not a guarantee.
  • domain assumption The frozen language-model head error e is the correct top-level signal for adaptation.
    Lemma C.1 derives e from the CE loss, but the assumption that the head remains frozen and suitable for the task is a modeling choice.
  • ad hoc to paper Optional Lipschitz smoothness condition in Proposition D.1 holds when needed.
    Used to prove single-block virtual descent, but no empirical verification; the paper notes it is not a global convergence result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment." pith.science (2026). https://pith.science/paper/AXJ673CR

@misc{pith2026260803020,
  author       = {Pith},
  title        = {Pith review of: LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AXJ673CR}},
  note         = {Machine review of arXiv:2608.03020}
}
read the original abstract

Parameter-efficient post-training reduces the number of trainable parameters, but still requires repeated end-to-end backpropagation through the frozen backbone. Every adaptation step therefore needs backward-capable hardware and must store or recompute activations. We ask whether this repeated backward chain can be replaced by a one-time calibration. We introduce Local Credit Assignment (LoCA), a two-stage method for small-shift adaptation. One probe backward pass fits a low-rank map at each transformer block from the final prediction error to a local hidden-state correction. LoCA then reuses these maps to form blockwise regression targets from forward activations and fits low-rank adapters with closed-form ridge solves. No further backbone backward pass is required. We evaluate LoCA on five discriminative benchmarks with Qwen2.5 models from 0.5B to 14B. In 16 of 25 reported task--scale comparisons, LoCA yields lower evaluation cross-entropy than the corresponding LoRA run. Its measured full-run GPU peak, including calibration, is 26--29\% lower than LoRA's. After calibration, its CPU steady-state memory is 36--52\% lower and its per-pass time is 43--48\% lower. A shared scale-normalized candidate set is reused across all tested Qwen2.5 sizes and on SmolLM2-1.7B. LoCA thus amortizes global credit assignment into one calibration and enables later forward-only tuning when repeated backpropagation is impractical. The code associated with this paper is available \href{https://github.com/Xia12121/LoCA}{here}.

Figures

Figures reproduced from arXiv: 2608.03020 by the authors.

Figure 1
Figure 1. LoRA and the post-calibration LoCA loop. One probe backward pass fits [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Full-parameter MeZO on BoolQ across the tested model sizes. Left: best evaluation CE within the reported [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 2 canonical work pages

  1. [1]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback....

    2022

  2. [2]

    Suchin Gururangan, Ana Marasovi´c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don’t stop pretraining: Adapt language models to domains and tasks. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, 2020

    2020

  3. [3]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InInternational Conference on Learning Representations, 2015. 8 LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment

    2015

  4. [4]

    Lee, Danqi Chen, and Sanjeev Arora

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. InAdvances in Neural Information Processing Systems, volume 36, 2023

    2023

  5. [5]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Represen- tations, 2022

    2022

  6. [6]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. InProceedings of the 36th International Conference on Machine Learning, pages 2790–2799, 2019

    2019

  7. [7]

    Prefix-Tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing continuous prompts for generation. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 4582–4597, 2021

    2021

  8. [8]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, 2021

    2021

Show all 46 references
  1. [9]

    James C. Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control, 37(3):332–341, 1992

    1992

  2. [10]

    Duchi, Michael I

    John C. Duchi, Michael I. Jordan, Martin J. Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations.IEEE Transactions on Information Theory, 61(5): 2788–2806, 2015

    2015

  3. [11]

    Random gradient-free minimization of convex functions.F oundations of Computational Mathematics, 17(2):527–566, 2017

    Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions.F oundations of Computational Mathematics, 17(2):527–566, 2017

    2017

  4. [12]

    Lillicrap, Daniel Cownden, Douglas B

    Timothy P. Lillicrap, Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. Random synaptic feedback weights support error backpropagation for deep learning.Nature Communications, 7:13276, 2016

    2016

  5. [13]

    Direct feedback alignment provides learning in deep neural networks

    Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. InAdvances in Neural Information Processing Systems, volume 29, 2016

    2016

  6. [14]

    Align, then memorise: The dynamics of learning with feedback alignment

    Maria Refinetti, Stéphane d’Ascoli, Ruben Ohana, and Sebastian Goldt. Align, then memorise: The dynamics of learning with feedback alignment. InProceedings of the 38th International Conference on Machine Learning, pages 8925–8935, 2021

    2021

  7. [15]

    Greedy layerwise learning can scale to ImageNet

    Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning can scale to ImageNet. InProceedings of the 36th International Conference on Machine Learning, pages 583–593, 2019

    2019

  8. [16]

    Training neural networks with local error signals

    Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. InProceedings of the 36th International Conference on Machine Learning, pages 4839–4850, 2019

    2019

  9. [17]

    The forward-forward algorithm: Some preliminary investigations.arXiv preprint arXiv:2212.13345, 2022

    Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations.arXiv preprint arXiv:2212.13345, 2022

    2022 arXiv

  10. [18]

    TinyBERT: Distilling BERT for natural language understanding

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174, 2020

    2020

  11. [19]

    Measuring the intrinsic dimension of objective landscapes

    Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. InInternational Conference on Learning Representations, 2018

    2018

  12. [20]

    Intrinsic dimensionality explains the effectiveness of language model fine-tuning

    Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 7319–7328, 2021

    2021

  13. [21]

    GaLore: Memory-efficient LLM training by gradient low-rank projection

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. GaLore: Memory-efficient LLM training by gradient low-rank projection. InProceedings of the 41st International Conference on Machine Learning, 2024

    2024

  14. [22]

    Full parameter fine-tuning for large language models with limited resources

    Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 8187–8198, 2024

    2024

  15. [23]

    QLoRA: Efficient finetuning of quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. InAdvances in Neural Information Processing Systems, volume 36, 2023

    2023

  16. [24]

    Training deep nets with sublinear memory cost

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv:1604.06174, 2016. 9 LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment

    2016 arXiv

  17. [25]

    BAdam: A memory efficient full parameter optimization method for large language models

    Qijun Luo, Hengxu Yu, and Xiao Li. BAdam: A memory efficient full parameter optimization method for large language models. InAdvances in Neural Information Processing Systems, volume 37, 2024

    2024

  18. [26]

    Zeroth-order fine-tuning of LLMs in random subspaces

    Ziming Yu, Pan Zhou, Sike Wang, Jia Li, and Hua Huang. Zeroth-order fine-tuning of LLMs in random subspaces. InProceedings of the IEEE/CVF International Conference on Computer Vision, 2025

    2025

  19. [27]

    Enhancing zeroth-order fine-tuning for language models with low-rank structures

    Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. Enhancing zeroth-order fine-tuning for language models with low-rank structures. InInternational Conference on Learning Representations, 2025

    2025

  20. [28]

    Sparse MeZO: Less parameters for better performance in zeroth-order LLM fine-tuning.arXiv preprint arXiv:2402.15751, 2024

    Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse MeZO: Less parameters for better performance in zeroth-order LLM fine-tuning.arXiv preprint arXiv:2402.15751, 2024

    2024

  21. [29]

    Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor W. Tsang. Second-order fine-tuning without pain for LLMs: A Hessian informed zeroth-order optimizer. InInternational Conference on Learning Representations, 2025

    2025

  22. [30]

    Direct feedback alignment scales to modern deep learning tasks and architectures

    Julien Launay, Iacopo Poli, François Boniface, and Florent Krzakala. Direct feedback alignment scales to modern deep learning tasks and architectures. InAdvances in Neural Information Processing Systems, volume 33, 2020

    2020

  23. [31]

    Scaling forward gradient with local losses

    Mengye Ren, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. Scaling forward gradient with local losses. In International Conference on Learning Representations, 2023

    2023

  24. [32]

    Qwen2.5 technical report

    Qwen Team. Qwen2.5 technical report. arXiv:2412.15115, 2024

    2024 arXiv

  25. [33]

    SmolLM2: When smol goes big – data-centric training of a small language model

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, et al. SmolLM2: When smol goes big – data-centric training of a small language model. arXiv:2502.02737, 2025

    2025 arXiv

  26. [34]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. InProceedings of the 2013 Conference on Empirical Methods in Natural Language Process...

    2013

  27. [35]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computatio...

    2019

  28. [36]

    Think you have solved question answering? try ARC, the AI2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv:1803.05457, 2018

    2018 arXiv

  29. [37]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, 2018

    2018

  30. [38]

    HellaSwag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019. 10 LoCA: Forward-Only LLM Tuning afte...

    2019

  31. [39]

    the language-model-head error and the two ridge minimizers are exact for the stated fixed data

  32. [40]

    fixed feedback defines a virtual hidden-state descent step only under explicit alignment and smoothness conditions

  33. [41]

    Local Targets

    the fitted adapter need not realize that virtual step exactly, and the full outer iteration is not claimed to decrease global cross-entropy monotonically. Sections S9–S13 provide experimental context for the setup, main benchmark, resource study, scale normalization, and MeZO ...

  34. [42]

    a forward pass through the adapted frozen model

  35. [43]

    evaluation of (20) at the frozen head

  36. [44]

    application ofF ℓeusing fixed low-rank factors

  37. [45]

    streaming accumulation ofG ℓ andC ℓ

  38. [46]

    Related Work,

    positive-definite linear solves forB ℓ. 18 LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment None of these steps differentiates through a backbone block. Therefore the adaptation loop is backward-free after calibration. The complete method i...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.