{"id":"7ba03912-ba94-4ec2-85f2-6a330e49da9b","arxiv_id":"2608.03020","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LoCA replaces repeated backpropagation in adapter tuning with one calibrated linear feedback map per block, reporting lower memory than LoRA and lower evaluation cross-entropy in 16 of 25 tested cells.","lead":"LoCA is a new way to adapt large language models that uses one backward pass to calibrate per-block credit maps, then tunes low-rank adapters with forward passes and closed-form solves. It reports comparable or better cross-entropy than LoRA on most tested cells with lower memory, but the quality comparisons are mostly single runs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The fixed feedback map F_l is only validated in-probe; the paper never measures whether its alignment with true gradients survives the up-to-40 outer iterations, so the core forward-only claim rests on an unquantified small-shift assumption.","rationale":"The reader identified the same load-bearing assumption: the fixed feedback map may lose alignment as adapters update. I agree that this is the most fundamental risk, because the entire forward-only mechanism depends on F_l e remaining a faithful proxy for the true gradient g_l throughout the tuning trajectory. The paper acknowledges the risk but provides no empirical check of alignment over time, no quantitative definition of the small-shift regime, and no measurement of how far the model actually moves in the tested tasks. Without this, the 16/25 cross-entropy result could be driven by early iterations or by the held-out snapshot selection rather than by sustained valid credit assignment. A secondary concern is that LoCA uses rank 32 but the paper does not state LoRA's rank; if LoRA uses a smaller rank, the quality comparison is partly a parameter-count effect. However, that affects the empirical comparison rather than the algorithm's core validity. The algebraic derivations are correct, the memory and forward-only claims are structurally sound, and the limitations are honestly stated, so the verdict should remain conditional pending additional evidence. The concrete diagnostic test would directly resolve whether the central approximation holds for the full adaptation run.","tokens_in":17424,"tokens_out":7465,"duration_ms":70714,"concrete_test":"Re-run LoCA on at least the 3B and 14B SST-2 and BoolQ cells while inserting a diagnostic backward pass at each outer iteration (used only for measurement). Compute the mean cosine similarity ⟨F_l e, g_l⟩/(||F_l e|| ||g_l||) over blocks and tokens, and the relative RMS shift of hidden states from the calibrated state. If the cosine drops below, say, 0.1 before the selected snapshot, or if the shift exceeds a threshold, the small-shift assumption fails and the claimed success cannot be attributed to the calibrated feedback; if cosine stays in the 0.3–0.5 range across iterations, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"LoCA's practical guarantee—that one calibration pass suffices for the entire adaptation run—depends on the fixed rank-8 feedback map F_l continuing to approximate the true input-dependent Jacobian (∂h_L/∂h_l)^T as the adapters B_l evolve. The paper reports only in-probe cosine alignment of 0.3–0.5 (Section 3.2) and acknowledges in Section 5 that the approximation 'may weaken when tuning moves far from the calibration point.' However, no measurement is provided of how this alignment evolves over the up-to-40 outer iterations, nor is any quantitative definition of the 'small-shift' regime given. If alignment decays quickly, the targets τ_l in Eq. (4) become increasingly wrong, and the closed-form solves in Eq. (8) optimize a local surrogate that is no longer a descent direction for global cross-entropy. The empirical results (16/25 lower CE) could then be an artifact of early-iteration behavior or of held-out snapshot selection, rather than evidence that the calibrated credit remains valid. Because the method's forward-only benefit only matters if the calibration is valid for the whole tuning trajectory, this unquantified assumption is the most load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LoCA, a two-stage method for parameter-efficient fine-tuning of large language models. In the first stage, a single backward pass through the frozen backbone fits low-rank feedback operators mapping the top-layer error to per-block hidden-state corrections. In the second stage, these operators are fixed, and each low-rank adapter is updated by solving a local ridge regression in closed form, requiring only forward passes through the backbone. The authors evaluate LoCA on Qwen2.5 models from 0.5B to 14B and on SmolLM2-1.7B, reporting that LoCA achieves lower evaluation cross-entropy than the corresponding LoRA run in 16 of 25 task-scale comparisons, with 26–29% lower GPU peak memory including calibration. The paper is careful to describe most cells as single runs and to state that the results are descriptive rather than statistical evidence of general superiority.","tokens_in":17705,"tokens_out":4308,"duration_ms":40854,"significance":"If validated, LoCA offers a practical forward-only tuning regime with meaningful memory savings, and it occupies a distinct niche between zeroth-order methods and feedback alignment. The mathematical core is sound: the ridge solutions in Theorems E.1 and F.1 are derived cleanly and exactly, and Proposition D.1 makes the descent claim conditional on explicit alignment and smoothness assumptions rather than asserting unconditional convergence. The paper ships code, gives a detailed appendix of derivations, and is unusual in explicitly labeling the limits of its own evidence. The main weaknesses are empirical: the central small-shift assumption is never directly measured over the adaptation trajectory, and the headline comparisons rest on single runs without variance estimates.","major_comments":[{"comment":"The central load-bearing assumption of LoCA is that the once-fitted rank-8 feedback maps F_l maintain sufficient alignment with the true hidden-state gradients throughout the up to 40 outer iterations. The paper reports only in-probe cosine alignment of 0.3–0.5 and acknowledges in Section 5 that \"the approximation may weaken when tuning moves far from the calibration point,\" but no measurement is provided of how the alignment evolves over the adaptation trajectory. Without such a measurement, the forward-only claim rests on an unquantified small-shift assumption that could fail in practice. Please add a small-scale diagnostic, for example computing the cosine alignment alpha_l of Appendix D between F_l e and the exact hidden-state gradient G_l at several outer iterations on at least one task/model, or provide a quantitative bound on the shift within which the fixed feedback remains reliable.","section":"Section 3.2 and Section 5"},{"comment":"All main benchmark cells are single runs at seed 0, and no confidence intervals are reported. The paper explicitly states that the 16/25 count is descriptive, but this count is the primary evidence that LoCA is \"not merely an inaccurate copy of the LoRA update.\" Without any measure of variability, it is impossible to judge whether a 16/25 split is distinguishable from chance or from run-to-run noise. Please provide variance estimates for the headline comparison, for example by running 3 seeds on a representative subset of the 25 cells or by reporting bootstrap intervals for the recovery ratios. This is load-bearing because the qualitative conclusion that LoCA follows a different optimization path depends on the pattern of wins being a stable signal rather than noise.","section":"Section 4.1 and Table 1"},{"comment":"The \"small-shift regime\" is never defined quantitatively, despite being the scope condition for the method. The paper repeatedly refers to \"small, targeted changes\" but does not state a measurable threshold on, for example, parameter distance, activation norm change, or loss reduction within which LoCA is expected to be valid. Without an operational definition, the method's applicability is not falsifiable. Please provide a concrete shift metric or stability condition that can be checked in experiments, or report the measured shift in the evaluated settings so that the reader can connect the empirical results to the assumed regime.","section":"Section 5 and Appendix H"}],"minor_comments":[{"comment":"The text contains placeholder citations marked \"[?]\" where references are missing, specifically for preference alignment in the introduction and for variance-reduced MeZO variants in the Discussion. These should be completed before publication.","section":"Introduction and Discussion"},{"comment":"The column header \"LoCAR CE Racc\" appears to be a formatting error; it should likely read \"LoCA R_CE R_acc\" or similar. The table layout also makes it difficult to distinguish the model-size blocks; consider clearer vertical spacing or rule lines.","section":"Table 1"},{"comment":"Equation (24) sums over t from s to K for the positionwise gradient. It would be helpful to state explicitly that this relies on the causal mask making the Jacobian zero for t < s, as this justifies the sum over later positions only.","section":"Appendix C"},{"comment":"The phrase \"This solve is exact for a fixed local target\" is correct but could be misread as a global claim. Consider adding a brief sentence clarifying that \"exact\" means the exact minimizer of the local ridge quadratic, not an exact solution to the global cross-entropy objective.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is interesting and the mathematical presentation is careful, but the empirical validation is thin in ways that are not fully offset by the paper's honest caveats. The single-seed design and the absence of any measurement of feedback-alignment decay over outer iterations are both directly relevant to the paper's main claim. I would encourage the editor to request a revision that adds at least one alignment-over-time diagnostic and a limited seed- or interval-based variability check for the headline comparisons. The placeholder citations should also be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: LoCA is a real attempt at making post-training backward-free after one calibration pass, and the algebra is clean. The paper deserves a serious referee, but the current version needs one more round of empirical hardening before the central claim fully lands.\n\nWhat's new: the specific combination—one probe backward pass to fit rank-8 linear maps from top-layer error to blockwise hidden-state corrections, then closed-form ridge solves per block with scale-normalized targets—isn't in the cited prior work. The math in the appendices is careful: Theorems E.1 and F.1 derive the ridge solutions correctly, and Proposition D.1 is the standard descent lemma, honestly labeled as applying to a virtual single-block step, not the outer loop. Credit goes to distinguishing exact local optimization from global convergence, which the paper never overclaims.\n\nThe empirical work is transparent about its limits: most cells are single runs, the 16/25 count is called descriptive, and the BoolQ CE-accuracy divergence is flagged as unexplained. The memory measurements include calibration in GPU peak, which is the right thing to do. The scale-normalized target transfer across model families is a nice practical result.\n\nSoft spots: the stress-test concern is real. The fixed feedback map's alignment with true gradients is measured only in-probe (cosine 0.3–0.5). The paper says alignment may degrade but never measures how it behaves over up to 40 outer iterations. Since the forward-only benefit depends on the calibration remaining useful for the whole run, this is the load-bearing unquantified assumption. It's not fatal—the method is explicitly scoped to small shifts and keeps the frozen model as a fallback—but a quantitative alignment-vs-iteration plot would be the obvious missing figure.\n\nSecond, the novelty discussion skips synthetic gradients, the most related line of work. A linear map predicting gradients from a top error is essentially synthetic gradients with a closed-form fit. Omitting that baseline and discussion weakens the positioning.\n\nThird, the tables would carry more weight with repeated seeds and error bars, at least for the main 25-cell comparison. Also missing: the values of lambda, beta, and the probe-data size. Minor but easy to fix.\n\nVerdict: this is a legitimate contribution for people working on memory-constrained adaptation and local learning. I'd send it to peer review with a request for the alignment-over-iterations measurement and repeated runs. I wouldn't desk-reject it.","headline":"A cleanly derived forward-only tuning method with honest scoping, but the unmeasured calibration-drift assumption and missing synthetic-gradient baseline keep it a strong revise-and-resubmit rather than an accept.","tokens_in":18205,"tokens_out":2598,"would_cite":true,"duration_ms":24298,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that repeated backbone backpropagation in LLM adaptation can be replaced by a one-time calibration: after fitting low-rank feedback maps from final-layer error to blockwise hidden-state corrections, adapters are tuned by…","keywords":["backpropagation-free training","feedback alignment","parameter-efficient fine-tuning","forward-only tuning","low-rank adapters","ridge regression","large language models","memory-efficient fine-tuning"],"falsifier":"Run LoCA on a task where LoRA's improvement over the frozen model is large, then measure the blockwise cosine alignment $\\cos\\angle(F_\\ell e, g_\\ell)$ at outer iterations 1, 5, 20, and 40. If the alignment drops to near zero or negative at the point where held-out cross-entropy stops improving while a LoRA run continues to improve, the one-shot calibration assumption is falsified.","tokens_in":17250,"feed_emoji":"⚡","tokens_out":8013,"duration_ms":70322,"temperature":0.7,"pith_summary":"This paper tries to establish that repeated end-to-end backpropagation through a frozen LLM backbone can be replaced by a one-time calibration. LoCA fits a low-rank feedback map per transformer block, from the final prediction error to a blockwise hidden-state correction, using a single probe backward pass. After that, each outer iteration runs one forward pass, forms blockwise regression targets, and solves closed-form ridge problems for low-rank adapters without any backbone backward pass. On five discriminative tasks across Qwen2.5 models from 0.5B to 14B, LoCA reports lower evaluation cross-entropy than LoRA in 16 of 25 task-scale cells, with 26-29 percent lower full-run GPU peak memory and 36-52 percent lower post-calibration CPU steady-state memory. The paper itself restricts the claim to small-shift adaptation and treats the cell-level counts as descriptive rather than as evidence of general superiority.","feed_headline":"One calibration pass replaces every backward pass in LLM tuning","feed_subtitle":"After one probe backward pass, adapters tune forward-only with 26-29 percent lower peak GPU memory than LoRA.","key_machinery":"The load-bearing object is the calibrated low-rank feedback operator $F_\\ell$, a rank-8 truncation of a ridge fit of hidden-state gradients on top-layer error, together with the closed-form per-block ridge solve $B_\\ell^* = C_\\ell(G_\\ell + \\lambda I_r)^{-1}$. $F_\\ell$ converts the frozen head's prediction error into a blockwise hidden-state correction without differentiating through the backbone, and the solve then fits the adapter to that correction. The streaming statistics $G_\\ell$ and $C_\\ell$ are additive over batches and require $O(L(r^2+dr))$ storage, so the per-pass cost after calibration is essentially one forward pass plus small linear solves.","core_discovery":"The central discovery is that a rank-8 truncated ridge fit of the map from top-layer error $e = W_u^\\top(\\mathrm{softmax}(z)-\\mathrm{onehot}(y))$ to exact hidden-state gradients, computed once per block, carries enough credit information for subsequent forward-only adapter tuning. LoCA's outer loop computes targets $\\tau_\\ell = h_\\ell - \\bar{\\eta} F_\\ell e$ with $\\bar{\\eta} = \\eta\\, \\mathrm{RMS}(h_{\\ell-1})/\\mathrm{RMS}(F_\\ell e)$, then solves $B_\\ell^* = C_\\ell(G_\\ell+\\lambda I_r)^{-1}$. Because the block objective is a strictly convex quadratic for fixed targets, each solve is exact for its local objective; the paper does not claim global cross-entropy decreases monotonically and selects snapshots by held-out CE. The fitted maps show in-probe cosine alignment 0.3-0.5 with true hidden-state gradients, versus near zero for random feedback, which is the quantitative reason the local targets are informative. The method therefore amortizes global credit assignment into one calibration and leaves a backward-free outer loop.","pith_inferences":["If the fitted feedback maps degrade predictably with distance from calibration, a cheap recalibration trigger based on the residual between realized adapter displacements and target displacements could extend LoCA to larger shifts than the paper tests.","Because the per-block solves can be computed independently for a shared forward pass, the method may distribute naturally across devices that cannot run a global backward pass.","The reported BoolQ pattern, in which LoRA lowers cross-entropy while ranking accuracy drops and LoCA preserves accuracy, suggests the local-target bias may act as a regularizer against completion-format overfitting; that hypothesis is testable by comparing validation CE and ranking accuracy across datasets with and without format shift.","A direct comparison against a version that recomputes the feedback maps every few outer iterations would quantify how much of the method's value comes from the one-shot fit rather than from the local solve structure."],"forward_implications":["Post-calibration tuning can run on hardware without backward support, since each outer iteration uses only a forward pass, the frozen-head error, and local linear solves.","Compared with LoRA's backward-based loop, LoCA's measured full-run GPU peak memory is 26-29 percent lower, while its post-calibration CPU steady-state memory and per-pass time are 36-52 percent and 43-48 percent lower.","The same scale-normalized target-size candidate set transfers across Qwen2.5 sizes from 0.5B to 14B and to SmolLM2-1.7B, with held-out selection inside the set.","Because the local ridge solve is exact for its fixed target, LoCA has no learning-rate schedule or optimizer state inside a block update, although the paper limits this claim to the local objective rather than to global convergence.","LoCA can return the frozen model as a fallback, protecting the selected validation cross-entropy when no adapted snapshot improves on it."],"supporting_citations":[{"why":"Defines the frozen-backbone low-rank adapter parameterization and serves as the main quality and memory baseline.","marker":"[5]"},{"why":"Provides the forward-only zeroth-order baseline, and Adapter-MeZO perturbs the same low-rank adapters as LoCA.","marker":"[4]"},{"why":"Introduces the random-feedback idea that LoCA replaces with a fitted operator.","marker":"[12]"},{"why":"Sends top-layer error directly to each layer, the blockwise credit pattern LoCA adopts.","marker":"[13]"},{"why":"Shows feedback alignment relies on forward weights adapting, which a frozen backbone cannot do, motivating the fitted feedback maps.","marker":"[14]"},{"why":"Supplies the Qwen2.5 model family used in the 25 task-scale benchmark cells.","marker":"[32]"},{"why":"Provides the second model family used to test the scale-normalized target candidate set.","marker":"[33]"}],"fun_headline_variants":["One backward pass enables forward-only LLM tuning","Calibrate once, then tune without backward passes","LoCA: one-shot calibration, forward-only adaptation","Replace repeated backprop with a single calibration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that one-time calibration stays valid: as adapters update, the fitted feedback maps must keep pointing roughly toward the true hidden-state gradients, and the paper concedes that alignment can degrade when tuning moves far from the calibrated state.","fun_headline_variants_meta":{"raw":{"variants":["One backward pass enables forward-only LLM tuning","Calibrate once, then tune without backward passes","LoCA: one-shot calibration, forward-only adaptation","Replace repeated backprop with a single calibration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001388,"raw_usage":{"total_tokens":5690,"prompt_tokens":1091,"completion_tokens":4599,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":707,"completion_tokens_details":{"reasoning_tokens":4539}},"tokens_in":707,"tokens_out":4599,"duration_ms":31057,"temperature":1.0,"reasoning_tokens":4539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:53:10.068622+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LoCA on a task where LoRA's improvement over the frozen model is large, then measure the blockwise cosine alignment $\\cos\\angle(F_\\ell e, g_\\ell)$ at outer iterations 1, 5, 20, and 40. If the alignment drops to near zero or negative at the point where held-out cross-entropy stops improving while a LoRA run continues to improve, the one-shot calibration assumption is falsified.","supporting_citations":[{"cited_title":"Lee, Danqi Chen, and Sanjeev Arora","cited_arxiv_id":null,"evidence_quote":"Provides the forward-only zeroth-order baseline, and Adapter-MeZO perturbs the same low-rank adapters as LoCA."},{"cited_title":"Align, then memorise: The dynamics of learning with feedback alignment","cited_arxiv_id":null,"evidence_quote":"Shows feedback alignment relies on forward weights adapting, which a frozen backbone cannot do, motivating the fitted feedback maps."}],"review_version":2}