{"id":"a055630d-cfd9-484f-b1a4-560d2a13f854","arxiv_id":"2607.19391","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"LAARA allocates LoRA ranks per layer from diagonal Fisher (gradient-based) estimates, reporting improved accuracy with fewer trainable parameters on GLUE and MathInstruct.","lead":"This paper proposes LAARA, a method that gives each layer of a fine-tuned transformer its own LoRA adapter size, using cheap Fisher-information estimates from gradients. It reports matching or beating standard baselines on GLUE and math reasoning while training fewer parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central premise that Fisher scores from low-rank adapters early in training predict optimal final ranks after resizing is untested; dynamic allocation may chase a moving target, and Theorem 2 does not rescue it because its monotonicity inference is invalid.","rationale":"The paper's strongest practical claim is that LAARA matches or outperforms fixed-rank and adaptive baselines at lower parameter counts. For that claim to be more than a particular monotone allocation trick, the dynamic, Fisher-driven rank adaptation must be doing real work. The weakest point is exactly the predictivity of early Fisher estimates under the very resizing LAARA performs: the paper demonstrates stability of layer ordering but never demonstrates that the ranks chosen are optimal after previous rank changes. This is not an external-consensus disagreement; it is a missing control within the paper's own logic. The theoretical theorems, while presented as supporting 'provably' suboptimal uniform allocation, contain invalid inference steps: Lemma 2's lower-bound argument cannot establish inequality of optimal ranks, and Theorem 2's Part 3 uses a malformed inequality chain. I agree with the reader's identification of the moving-target premise as the weakest assumption, and I also note the theorem flaw as a compounding issue. A concrete ablation with static versus dynamic rank schedules would settle whether the dynamic loop matters. Since the reader already issued CONDITIONAL, my stress-test does not move the verdict; it sharpens the condition: add the static-vs-dynamic ablation and correct or downgrade the theoretical claims. I am not raising concerns about author intent or integrity; the concern is about missing experimental controls and invalid proof steps.","tokens_in":19127,"tokens_out":6696,"duration_ms":77620,"concrete_test":"Run LAARA on RTE, CoLA, and one MathInstruct split with three variants under identical hyperparameters: (A) StaticEarly: compute Fisher scores once at step 200, apply Eq. 17 to assign final ranks, freeze ranks for the rest of training; (B) FullLAARA: current algorithm with Δ=200 and τ=2; (C) StaticLate: fine-tune one epoch with fixed ranks, then compute Fisher scores once, map to final ranks, and freeze. If (A) is within 0.5 GLUE points of (B) at equal or lower final parameter count, then dynamic rank tracking is not load-bearing and the predictivity premise is unnecessary. If (B) clearly beats both static variants, the moving-target concern is mitigated. Additionally, at each update step record the proposed rank vs. the rank that would be chosen from a fresh Fisher estimate at that moment; large divergence indicates the allocation is chasing a moving target.","verdict_should_be":"UNCHANGED","load_bearing_attack":"LAARA's rank-allocation loop is only justified if the Fisher importance measured with adapters at their current ranks remains predictive of the best ranks after those adapters are resized. The paper shows layer ordering stabilizes after epoch 3 (Observation 2, Figs. 2, 6), but ordering stability is not rank optimality. When a layer's rank changes, the adapter's gradient geometry changes, so the Fisher target itself shifts; Eq. 18's vote-to-change mechanism only damps oscillations, it does not correct the target. The experiments compare full LAARA only against fixed-rank baselines, never against an ablation with ranks frozen after the first calibration or against a static allocation of the same final parameter budget. Without such controls, the reported gains could stem from any reasonable monotone final rank profile rather than from the dynamic tracking that is LAARA's claimed contribution.\n\nThe theoretical section does not fill this gap. Lemma 2 claims optimal ranks differ from heterogeneous effective ranks, but its proof uses only lower bounds on r*_l and r*_{l'}; larger lower bound for layer l does not imply r*_l > r*_{l'} — both true optimal ranks could still be equal. Theorem 2's Part 3 is also malformed: from r*_l ≥ c·erank(F_l) ≤ c·erank(F_{l+1}) ≤ r*_{l+1} one cannot conclude r*_l ≤ r*_{l+1}. Moreover, erank(Fl) ≤ tr(Fl)/||Fl||_2 does not imply erank(Fl) ≤ erank(F_{l+1}) when the spectral norm can also grow with depth. Thus the theoretical monotonicity claim is unsupported, and the paper's 'provably' language overstates what is established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LAARA, a search-free method for allocating per-layer LoRA ranks based on diagonal Fisher information estimates computed during fine-tuning. It claims two theoretical results: Theorem 1 states that uniform rank allocation is provably suboptimal under heterogeneous Fisher spectra, and Theorem 2 claims that gradient norms / Fisher traces induce a monotonic non-decreasing rank profile with layer depth. The method combines EMA-smoothed diagonal Fisher traces, per-projection normalization, log compression, a blended A/B importance score, and a vote-to-change dampening mechanism. Experiments on GLUE with DeBERTa-v3-base and on MathInstruct with Llama-3.2-3B report that LAARA matches or outperforms LoRA, AdaLoRA, DyLoRA, and BitFit while using fewer trainable parameters.","tokens_in":19484,"tokens_out":6488,"duration_ms":63456,"significance":"If the empirical claims hold, LAARA is a practically useful, search-free alternative to fixed-rank LoRA and SVD-based adaptive rank methods. The paper provides a clearly specified algorithm and releases code, and the design choices (EMA smoothing, per-projection normalization, dampening) are reasonable and reproducible from the text. The theoretical machinery, however, is central to the paper's framing: the abstract and introduction claim that uniform rank allocation is 'provably suboptimal' and that the proposed monotone rank schedule is theoretically justified. Those theoretical claims are not established by the proofs as written, and the empirical validation is partly circular because the final rank profile is generated by the allocation rule itself. The contribution is therefore best viewed as a promising empirical recipe needing a corrected or weakened theoretical narrative and additional ablation controls.","major_comments":[{"comment":"The proof of Part 3 is invalid. It derives r*_l ≥ c·erank(F_l) ≤ c·erank(F_{l+1}) ≤ r*_{l+1}, which does not imply r*_l ≤ r*_{l+1}; a chain of lower bounds cannot establish an ordering of the optimal ranks. In addition, Eq. (33) gives erank(M) ≤ tr(M)/||M||_2, but this does not imply erank(F_l) ≤ erank(F_{l+1}) from tr(F_l) ≤ tr(F_{l+1}) unless the spectral norms are controlled; Assumption 4 bounds the weight matrices W_l, not the Fisher matrices F_l. The monotonicity claim is therefore unsupported and cannot be used to justify the final rank profile.","section":"Appendix A.5, Theorem 2 (Eq. 7)"},{"comment":"The proof of Lemma 2 relies on a reversed inequality. After assuming erank(F_l) > erank(F_{l'}), the proof writes r = r*_{l'} ≤ c·erank(F_{l'}) < c·erank(F_l) ≤ r*_l, but Lemma 1 is a lower bound r*_{l'} ≥ c·erank(F_{l'}), not an upper bound. A larger lower bound for layer l than for layer l' does not imply r*_l ≠ r*_{l'}; both optimal ranks could be large. Since Theorem 1's Part 1 invokes Lemma 2, the proof that uniform rank always under- or over-parameterizes a layer is not established. This needs a correct argument or the theoretical claim should be downgraded.","section":"Appendix A.5, Lemma 2"},{"comment":"Observation 3 states that the effective rank of the weight updates decreases with depth (from ~6 at layer 0 to ~2 at layer 11), i.e., late layers concentrate their adaptation in few directions. Under the paper's own Lemma 1 logic, concentrated spectra should require smaller ranks, yet Theorem 2 and the LAARA allocation (Eq. 17, Fig. 4) assign larger ranks to deeper layers. The paper never reconciles this tension. The empirical motivation therefore supports either increasing or decreasing rank schedules depending on whether one follows Fisher trace magnitude or update effective rank. This contradiction must be resolved or the theoretical interpretation revised.","section":"§1 Observation 3 versus §2.3 / Fig. 4"},{"comment":"Figure 4 is presented as evidence that final ranks 'strongly validate' the monotonicity result, but this is partly circular: Eq. (17) defines r_l as a monotone linear function of the compressed Fisher score, so any monotone Fisher profile yields a monotone rank profile by construction. The figure is not an independent test. More importantly, the experimental section never compares LAARA against (i) an ablation with ranks frozen after the first calibration pass, or (ii) a static monotone allocation with the same final parameter budget. Without such controls, it is unclear whether the reported gains come from the dynamic vote-based tracking emphasized in Section 3.4 or simply from any depth-increasing rank profile. Please add these ablations or soften the attribution.","section":"§4.2, Eq. (17), Fig. 4"}],"minor_comments":[{"comment":"Algorithm 1 initializes r_l ← r_max, while Section A.2 says LAARA uses initial rank r_init=4. Please clarify which initialization is actually used; this affects the early-training parameter budget and reproducibility.","section":"Algorithm 1 and §A.2"},{"comment":"Several references are duplicated or near-duplicated: DyLoRA appears as Valipour et al. 2022 and 2023; FLoE appears as Wang et al. 2025a and 2025b; AutoLoRA appears as Zhou et al. 2024b and Zhang et al. 2024. This creates ambiguity about which papers/versions were used as baselines and should be cleaned.","section":"References"},{"comment":"The text says comparisons are made 'at identical parameter budget', but the parameter counts differ: Table 1 has LAARA 851,522 vs LoRA 887,042; Table 2 has 987,042 vs 1,034,498; Table 4 has 6,104,512 vs 8,275,984. Since the selling point is lower parameter use, the comparisons are still informative, but the wording should be corrected or matched-budget runs should be added.","section":"§4.2, Tables 1, 2, 4"},{"comment":"Results are reported as averages over three runs, but no standard deviations, confidence intervals, or significance tests are provided. Given the small gaps on some GLUE tasks, adding variance information would strengthen the empirical claims.","section":"§A.2, Tables 1–4"}],"recommendation":"major_revision","confidential_remarks":"The empirical recipe is plausible and the code release is a positive feature, but the theoretical core needs substantial repair: Theorem 2's monotonicity step is invalid, Lemma 2's proof uses a reversed inequality, and the empirical validation of the rank profile is partly by construction. These are load-bearing because the abstract and introduction present the theory as proof that uniform rank allocation is fundamentally suboptimal. The authors could fix the manuscript by replacing the flawed theorems with a weaker empirical claim and adding the missing ablations, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: LAARA is a cheap, clearly specified way to allocate per-layer LoRA ranks from diagonal Fisher estimates, and the empirical story is plausible — a few points on GLUE tasks and 2–3% on GSM8K with fewer trainable parameters. The recipe itself is the contribution: EMA-smoothed squared gradients, per-projection min-max normalization, log compression, a blend of A/B adapter scores, and a vote-to-change dampening rule. That specific combination is not in the prior work I know, and the code link is a plus. The experimental setup is reproducible enough that I could re-run it.\n\nThe soft spots are real but mostly fixable. The theoretical section is the weakest part. Theorem 2's proof contains an invalid inequality chain: from r*_l ≥ c·erank(F_l) ≤ c·erank(F_{l+1}) ≤ r*_{l+1} you cannot conclude r*_l ≤ r*_{l+1}. And the step from trace monotonicity to effective-rank monotonicity assumes the spectral norm is constant across layers, which Assumption 4 doesn't actually give you. So the \"provably\" monotonic rank profile is not established. The paper would be stronger if it presented Theorem 2 as an empirical regularity rather than a proof.\n\nFigure 4 is also circular: the final ranks are produced by Eq. 17, a monotone transform of Fisher scores, so observing that deeper layers get higher ranks is just reading back the allocation rule, not independent validation. That should be reworded or removed. The evaluation gaps are more consequential: no error bars (only 3-seed averages) and no direct comparison to FiLoRA, FLoE, or Fish-tuning, which are the closest Fisher-guided baselines and are cited but never run. Given the abstract claims \"consistently matches or outperforms,\" that omission matters. The stress-test concern about dynamic allocation chasing a moving target is worth checking, but I don't see it as fatal — layer ordering stabilizes early, and the dampening mechanism limits oscillation. An ablation that freezes ranks after the first calibration would settle it.\n\nWho is this for? People working on parameter-efficient fine-tuning who want a low-overhead alternative to fixed-rank LoRA. It deserves a serious referee: the method is novel enough and the empirical results are interesting enough to warrant a careful look, but the paper needs a major revision before acceptance — fix or soften the theorems, add variance and the missing baselines, and stop claiming Figure 4 validates the theory. I'd send it to review with that expectation.","headline":"Useful search-free PEFT recipe with plausible gains, but the theory overreaches and the evaluation skips the closest Fisher-based competitors.","tokens_in":20118,"tokens_out":1614,"would_cite":true,"duration_ms":17297,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that uniform LoRA rank allocation across transformer layers is provably suboptimal, and that per-layer ranks derived from diagonal Fisher information match or beat existing parameter-efficient fine-tuning methods with fewer","keywords":["parameter-efficient fine-tuning","low-rank adaptation","layer-wise rank allocation","Fisher information","diagonal Fisher approximation","LoRA","GLUE benchmark","adaptive rank"],"falsifier":"The decisive test is an ablation that fixes the per-layer rank ordering after the early calibration epoch and disables all later resizing; if accuracy does not drop, the dynamic adaptivity is not doing the work. A complementary check: recompute the per-layer Fisher ordering at the end of training and compare it with the ordering at the calibration epoch — a near-zero rank correlation would show the calibration is chasing a moving target.","tokens_in":18913,"feed_emoji":"🎯","tokens_out":5894,"duration_ms":60064,"temperature":0.7,"pith_summary":"The paper sets out to prove that the common practice of giving every transformer layer the same LoRA rank is not merely wasteful but fundamentally suboptimal. It proposes LAARA, which estimates each layer's importance from diagonal Fisher information — approximated by squared LoRA gradient norms already computed during fine-tuning — and maps those scores to per-layer integer ranks. Because deeper layers accumulate larger Fisher traces, LAARA concentrates capacity near the output and compresses earlier layers, producing a rank profile that grows with depth. The paper reports that LAARA consistently matches or outperforms LoRA, AdaLoRA, DyLoRA, and BitFit on GLUE and MathInstruct while using substantially fewer trainable parameters. If right, adaptive rank allocation can be made principled, cheap, and stable without search or auxiliary objectives.","feed_headline":"Fisher-guided layer ranks beat uniform LoRA on GLUE","feed_subtitle":"Per-layer ranks from gradient-based Fisher estimates match or beat LoRA, AdaLoRA, DyLoRA, and BitFit with fewer trainable parameters.","key_machinery":"The load-bearing object is the layer-wise diagonal Fisher information, approximated by the EMA of squared LoRA gradient norms — the expected squared Frobenius norm of the gradient, claimed to be a consistent estimator of the Fisher trace. Each projection type is normalized across layers, log-compressed, blended between A and B adapters, and linearly mapped to an integer rank in [rmin, rmax]; a vote-to-change mechanism requiring τ consecutive identical proposals prevents rank oscillation. Theorems 1 and 2 use this Fisher geometry to argue that uniform rank is suboptimal and that optimal rank is non-decreasing with layer depth.","core_discovery":"LAARA's central claim is that the correct adapter rank for a transformer layer is set by that layer's diagonal Fisher information, measured during ordinary LoRA fine-tuning as an EMA of squared gradients. Formalized: under positive-definite and heterogeneous per-layer Fisher spectra, any uniform rank either starves a sensitive layer or wastes budget on an insensitive one (Theorem 1); under Lipschitz activations, bounded weights, and a contraction condition, expected gradient norms — and therefore optimal ranks — are non-decreasing toward the task loss (Theorem 2). The method converts these scores into stable integer ranks through projection-wise normalization, log-compression, blending of A/","pith_inferences":["A test the paper does not run: freeze ranks after the first calibration epoch and skip all resizing; if accuracy holds, the mid-training dynamics are a robustness feature rather than the source of gains.","The same Fisher trace that sets rank could also inform layer-wise quantization or pruning thresholds for the merged adapter, extending the method beyond rank selection.","The empirical Fisher ordering should be compared with exact Fisher or K-FAC estimates on a small model; disagreement would weaken the proxy but not necessarily the rank-allocation principle.","The paper's own remark that the depth-rank profile reverses when gradients explode invites a direct test: weaken or remove layer normalization and check whether early layers then need the higher ranks."],"forward_implications":["Uniform-rank LoRA is systematically misallocating capacity; adopting Fisher-guided layer ranks should improve accuracy at equal parameter budgets without any search.","A single early calibration pass can set a stable per-layer rank profile for the duration of fine-tuning, since the layer ordering stabilizes after about three epochs.","The monotone depth profile provides a principled default: assign deeper layers higher ranks and early layers lower ranks, widening the spread as the total rank budget grows.","The allocation mechanism is independent of target projection choice, so the same recipe transfers across Wq/Wv, Wq/Wk/Wv/Wo, and generative reasoning fine-tuning."],"fun_headline_variants":["Fisher ranks beat uniform LoRA with fewer parameters","Layer-aware rank allocation outperforms fixed-rank adapters","Adaptive LoRA ranks via Fisher: less trainable, same score","LAARA: Fisher-guided rank allocation beats LoRA baselines"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method's load-bearing premise is that Fisher importance measured from small rank-4 adapters early in training still predicts which layers deserve the most capacity after the adapters themselves have been resized.","fun_headline_variants_meta":{"raw":{"variants":["Fisher ranks beat uniform LoRA with fewer parameters","Layer-aware rank allocation outperforms fixed-rank adapters","Adaptive LoRA ranks via Fisher: less trainable, same score","LAARA: Fisher-guided rank allocation beats LoRA baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":9.3e-05,"raw_usage":{"total_tokens":815,"prompt_tokens":716,"completion_tokens":99,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":43}},"tokens_in":460,"tokens_out":99,"duration_ms":2043,"temperature":1.0,"reasoning_tokens":43,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T08:56:50.885396+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The decisive test is an ablation that fixes the per-layer rank ordering after the early calibration epoch and disables all later resizing; if accuracy does not drop, the dynamic adaptivity is not doing the work. A complementary check: recompute the per-layer Fisher ordering at the end of training and compare it with the ordering at the calibration epoch — a near-zero rank correlation would show the calibration is chasing a moving target.","supporting_citations":[],"review_version":1}