REVIEW 3 major objections 2 minor 2 references
FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning
T0 review · 3 major / 2 minor · reviewed 2026-06-29 · grok-4.3
Pith's one-line read FoRA selects task-informative layers via cheap Fisher scores and enforces orthonormality on LoRA down-projections to cut parameters while preserving accuracy.
desk verdict FoRA pairs cheap Fisher layer selection with Stiefel orthogonality to cut adapted layers rather than rank, but the selection step's reliability is the part that needs checking. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Fisher-based layer selection combined with Stiefel manifold training of the LoRA down-projection to enforce orthonormality.
What would settle it
Running the method with randomly chosen layers instead of Fisher-selected ones and observing no drop in performance would show that the selection step is not necessary.
Extended reading notes
Core claim
FoRA selects task-informative layers with a single-pass diagonal Fisher score computed at under 1% training cost and trains the LoRA down-projection matrices at those layers on the Stiefel manifold to preserve column orthonormality and effective rank, resulting in consistent outperformance of LoRA and DoRA at half the parameter budget and near parity with AdaLoRA at one-quarter the count across five LLaMA-family backbones and twelve models from LLaMA, Qwen3, and Gemma families.
Load-bearing premise
The diagonal Fisher score from a single forward-backward pass on a small data subset accurately identifies the most task-informative layers for adaptation.
Editorial extensions
If this is right
- The Fisher selection and Stiefel constraint combine super-additively for the accuracy gains.
- Fisher selection alone can match the performance of rank reduction at the same budget.
- The method scales consistently from 270M to 32B parameters.
- Gains hold across different model families including Qwen3 and Gemma.
Reading between the lines
- The layer selection strategy could be applied to other adaptation methods like full fine-tuning or other adapters.
- The low-cost Fisher approximation might enable dynamic layer selection during training.
- This suggests that many layers in LLMs are less critical for specific tasks, pointing to potential for more sparse adaptation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces FoRA, a parameter-efficient fine-tuning method that selects task-informative layers via a single-pass diagonal Fisher score (at <1% training cost) and constrains the LoRA down-projection at those layers to the Stiefel manifold to enforce column orthonormality. It claims FoRA outperforms LoRA and DoRA at half their parameter budget while approaching AdaLoRA performance at one-quarter the budget, with consistent results across five LLaMA-family models and twelve backbones from 270M to 32B parameters; the two components are reported to combine super-additively.
Significance. If the empirical claims and isolation of the Fisher-plus-Stiefel contribution hold under rigorous controls, the work would be significant for PEFT research by shifting focus from rank reduction to layer selection with cheap Fisher approximation and orthogonal constraints, potentially enabling more parameter-efficient adaptation without sacrificing accuracy.
major comments (3)
- [§3 (Method) and §4 (Experiments)] The central performance claims rest on the reliability of single-pass diagonal Fisher layer selection, yet the manuscript provides no direct validation (e.g., comparison to multi-pass or full Fisher, or sensitivity analysis across training distributions) that this approximation surfaces layers whose adaptation produces the reported gains rather than baseline or hyperparameter effects.
- [§4 (Experiments) and Table/Figure reporting ablations] The abstract and results assert super-additive gains from Fisher selection plus Stiefel constraint, but without explicit ablations isolating each component at matched budgets (Fisher alone vs. rank reduction alone vs. combined), it is unclear whether the layer-selection mechanism or the orthonormality constraint drives the outperformance over LoRA/DoRA at half budget.
- [§4 (Experiments)] No experimental protocol, statistical tests, error bars, or ablation details are referenced in the provided abstract or summary, undermining verification of the consistent numerical gains across backbones; this is load-bearing for the efficiency-accuracy claims.
minor comments (2)
- [Abstract] Typo in abstract: 'receivedcomparatively' should be 'received comparatively'.
- [§3 (Method)] Notation for the diagonal Fisher score and Stiefel constraint should be formalized with equations early in the method section for clarity.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback on validation of the Fisher approximation, ablation clarity, and experimental reporting. We address each major comment below with clarifications from the manuscript and indicate planned revisions where appropriate.
read point-by-point responses
-
Referee: [§3 (Method) and §4 (Experiments)] The central performance claims rest on the reliability of single-pass diagonal Fisher layer selection, yet the manuscript provides no direct validation (e.g., comparison to multi-pass or full Fisher, or sensitivity analysis across training distributions) that this approximation surfaces layers whose adaptation produces the reported gains rather than baseline or hyperparameter effects.
Authors: The manuscript motivates the single-pass diagonal Fisher by its negligible cost (<1% of training) and its use in prior PEFT literature for layer importance. While direct head-to-head comparisons against multi-pass or full Fisher are not present, the selection's effectiveness is evidenced by the consistent outperformance over random or uniform layer selection baselines at matched budgets. We will add a limited sensitivity study (on 2-3 tasks) comparing single-pass vs. multi-pass Fisher in the revision to further isolate its contribution. revision: yes
-
Referee: [§4 (Experiments) and Table/Figure reporting ablations] The abstract and results assert super-additive gains from Fisher selection plus Stiefel constraint, but without explicit ablations isolating each component at matched budgets (Fisher alone vs. rank reduction alone vs. combined), it is unclear whether the layer-selection mechanism or the orthonormality constraint drives the outperformance over LoRA/DoRA at half budget.
Authors: The abstract already states that 'Fisher selection alone matches rank reduction at the same budget, while the Stiefel constraint provides the decisive additional gain.' These component-wise results appear in the full experimental section with matched-parameter comparisons. To improve clarity, we will expand the ablation tables/figures in the revision to explicitly label and juxtapose Fisher-only, Stiefel-only, and combined variants against LoRA/DoRA at identical budgets. revision: yes
-
Referee: [§4 (Experiments)] No experimental protocol, statistical tests, error bars, or ablation details are referenced in the provided abstract or summary, undermining verification of the consistent numerical gains across backbones; this is load-bearing for the efficiency-accuracy claims.
Authors: Section 4 of the full manuscript specifies the protocol (hyperparameters, datasets, evaluation), reports mean ± std over three random seeds with error bars in all tables/figures, and includes paired statistical tests for key comparisons. The abstract and summary are necessarily concise; we will add a brief pointer to these details in the revised abstract and ensure all figures/tables reference the seed count and significance testing. revision: partial
Circularity Check
No circularity; empirical claims rest on external benchmarks
full rationale
The paper presents FoRA as an empirical PEFT method combining diagonal Fisher layer selection with a Stiefel manifold constraint on selected LoRA adapters. All performance claims (outperforming LoRA/DoRA at half budget, near AdaLoRA at quarter budget) are supported by direct comparisons on LLaMA-family and other backbones, not by any internal derivation, equation, or fitted quantity that reduces to itself by construction. No self-citations, uniqueness theorems, or ansatzes are invoked as load-bearing steps. The single-pass Fisher heuristic is presented as a practical choice under 1% training cost, without claiming it is derived from or equivalent to the accuracy results.
Assumptions & free parameters
assumptions (2)
- domain assumption A single-pass diagonal Fisher score identifies task-informative layers at under 1% of training cost
- domain assumption Training the LoRA down-projection on the Stiefel manifold preserves column orthonormality and effective rank
Cite this review
Pith. "Pith review of FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning." pith.science (2026). https://pith.science/paper/55COIABH
@misc{pith2026260529317,
author = {Pith},
title = {Pith review of: FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning},
year = {2026},
howpublished = {\url{https://pith.science/paper/55COIABH}},
note = {Machine review of arXiv:2605.29317}
}
read the original abstract
Parameter-efficient fine-tuning(PEFT) has largely focused on LoRA and its accuracy-oriented variants, leaving the original goal of reducing trainable parameters has receivedcomparatively little attention. We introduce FoRA, which revisits this goal by reducing the number of adapted layers rather than adapter rank. FoRA selects task-informative layers via a single-pass diagonal Fisher score (under 1% of training cost) and trains the LoRA down-projection at selected layers on the Stiefel manifold, preserving column orthonormality and effective rank. FoRA consistently outperforms LoRA and DoRA at half their parameter budget, and falls within 0.7-0.8 accuracy points of AdaLoRA at one-quarter its parameter count, across five LLaMA-family backbones. Cross-architecture experiments on twelve backbones from the LLaMA, Qwen3, and Gemma families confirm consistent gains from 270M to 32B parameters. The two components combine super-additively: Fisher selection alone matches rank reduction at the same budget, while the Stiefel constraint provides the decisive additional gain.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
LoRA learns less and forgets less.TMLR. Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about physical commonsense in natural language. InAAAI. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural ...
work page Pith review arXiv 2020
-
[2]
Limitations of the empirical Fisher approxima- tion for natural gradient descent. InNeurIPS. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024a. DoRA: Weight- decomposed low-rank adaptation. InICML. Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zh...
Reviewed June 29, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.