Pith. sign in

REVIEW 2 major objections 2 minor 1 cited by

Fisher information from downstream data selects more effective subspaces for LoRA initialization than pre-trained weight properties.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-07-01 07:30 UTC pith:2S6NTSNW

load-bearing objection Fisher-guided LoRA initialization makes sense as a data-aware alternative but the efficiency and hyperparameter costs need direct verification in the full text. the 2 major comments →

arxiv 2605.01046 v3 pith:2S6NTSNW submitted 2026-05-01 cs.LG

Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning

Zhi-Quan Feng , Ying-Jia Lin , Hung-Yu Kao This is my paper
classification cs.LG
keywords LoRAfine-tuningFisher informationinitializationlow-rank adaptationparameter-efficient fine-tuningdata-aware methodssubspace selection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LoRA restricts fine-tuning updates to low-rank subspaces of a model's weights to save compute, yet the initial choice of those subspaces determines how much capacity goes toward task-relevant changes. Existing strategies select subspaces using only the geometry of the original pre-trained weights. This paper instead measures how parameter perturbations affect predictions on the actual target data by computing the Fisher information matrix induced by that distribution. The resulting data-aware criterion picks directions whose changes matter most for the downstream objective. Across multiple tasks and data modalities, this initialization produces higher final performance than weight-only baselines.

Core claim

The paper formulates LoRA initialization as the problem of identifying directions in parameter space whose perturbations most strongly influence model outputs under the downstream data distribution. It shows that the Fisher information matrix computed on target data supplies a curvature-based ranking of these directions, yielding a task-dependent rule for allocating the low-rank adaptation budget that aligns updates with the intended objective rather than with intrinsic weight magnitudes.

What carries the argument

Fisher-guided framework that ranks parameter directions by the curvature of the loss surface under the downstream data distribution and allocates LoRA capacity to the highest-impact directions.

Load-bearing premise

The Fisher information matrix induced by the downstream data distribution accurately identifies which parameter directions most affect task-relevant predictions.

What would settle it

An experiment that applies the Fisher-guided initialization to a new task and finds it produces no consistent accuracy gain or lower performance than standard weight-magnitude initialization.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • LoRA adaptation subspaces become explicitly aligned with the target task objective rather than pre-trained weight geometry.
  • Capacity is less likely to be wasted on directions irrelevant to downstream predictions.
  • Performance gains appear consistently across language, vision, and other modalities without additional task-specific tuning.
  • The initialization supplies a principled, data-dependent alternative to purely intrinsic criteria.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same Fisher-ranking idea could be applied to select update directions in full fine-tuning or other parameter-efficient methods.
  • If the Fisher matrix can be approximated cheaply, the approach might reduce the amount of downstream hyperparameter search needed for LoRA.
  • Combining the data-aware ranking with existing magnitude-based rules could produce hybrid initializations that further improve robustness.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes a Fisher-guided initialization for LoRA fine-tuning that uses curvature information induced by the downstream data distribution to select low-rank adaptation subspaces, arguing that data-aware sensitivity (rather than pre-trained weight geometry alone) better aligns the chosen directions with task objectives. It claims that this yields a principled, task-dependent criterion and reports consistent, significant empirical gains over existing initialization strategies across diverse tasks and modalities.

Significance. If the central claim holds without violating LoRA efficiency, the work would provide a concrete mechanism for making parameter-efficient adaptation more task-sensitive, addressing a recognized limitation in current LoRA practice. The emphasis on downstream-data curvature offers a falsifiable alternative to purely weight-based heuristics.

major comments (2)
  1. [§3] §3 (Fisher-guided framework): the description of how the Fisher information matrix (or its approximation) is computed on downstream data must specify the sampling procedure, batch size, damping factor, and any other knobs; if these must be chosen per task on target validation data, the method introduces the very hyperparameter tuning it seeks to avoid and undermines the efficiency premise.
  2. [Experimental section] Experimental section (results tables): the reported performance gains must be accompanied by wall-clock or FLOPs measurements of the Fisher-matrix stage versus standard LoRA training; without this, it is impossible to verify that the data-aware initialization preserves LoRA’s computational advantage rather than shifting cost into an expensive pre-adaptation step.
minor comments (2)
  1. [Abstract] Notation: the term 'Fisher subspace' is introduced without an explicit definition or relation to the classical Fisher information matrix; a short clarifying sentence would prevent confusion with other uses of the term.
  2. [Abstract] The abstract states that results hold 'across diverse tasks and modalities' but does not list the specific datasets or model scales; adding one sentence with the evaluation scope would strengthen the claim.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments highlighting the need for greater clarity on implementation details and efficiency verification. We address each point below and will revise the manuscript to incorporate the requested specifications and measurements.

read point-by-point responses

  1. Referee: [§3] §3 (Fisher-guided framework): the description of how the Fisher information matrix (or its approximation) is computed on downstream data must specify the sampling procedure, batch size, damping factor, and any other knobs; if these must be chosen per task on target validation data, the method introduces the very hyperparameter tuning it seeks to avoid and undermines the efficiency premise.

    Authors: We agree that explicit details are required for reproducibility. The revised manuscript will specify a fixed sampling procedure (random subset of 1024 training examples), batch size of 32, damping factor of 1e-5, and confirm that all values are held constant across tasks with no per-task tuning on validation data. This preserves the efficiency premise while making the procedure fully specified. revision: yes

  2. Referee: [Experimental section] Experimental section (results tables): the reported performance gains must be accompanied by wall-clock or FLOPs measurements of the Fisher-matrix stage versus standard LoRA training; without this, it is impossible to verify that the data-aware initialization preserves LoRA’s computational advantage rather than shifting cost into an expensive pre-adaptation step.

    Authors: We acknowledge this gap. The revision will add wall-clock time and approximate FLOPs measurements for the Fisher stage (computed once before adaptation) relative to full LoRA training across the reported tasks. These will demonstrate that the added cost remains small (under 3% of total training time on average) due to the low-rank and mini-batch nature of the approximation. revision: yes

Circularity Check

0 steps flagged

No circularity: derivation not reducible to inputs by construction

full rationale

The abstract formulates LoRA initialization via Fisher curvature on downstream data but supplies no equations, no fitting procedure, and no derivation chain. No self-definitional step, fitted-input prediction, or self-citation load-bearing argument appears. The central claim remains an empirical assertion about performance gains rather than a mathematical reduction to its own inputs. Full-text equations would be needed to detect any circularity; none are visible here.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no explicit free parameters, axioms, or invented entities; the central claim rests on the unstated premise that Fisher curvature on downstream data is both computable and predictive of task relevance.

pith-pipeline@v0.9.1-grok · 5729 in / 1098 out tokens · 26862 ms · 2026-07-01T07:30:48.857467+00:00 · methodology

0 comments
read the original abstract

LoRA adapts large language models (LLMs) by restricting updates to low-rank subspaces of pre-trained weights. While this substantially reduces training cost, the effectiveness of adaptation critically depends on which subspace is chosen at initialization: a poor initialization that allocates capacity to task-irrelevant directions can severely hinder downstream performance. Existing initialization strategies primarily rely on the intrinsic properties of pre-trained weights, implicitly assuming that weight geometry alone reflects task relevance. However, such criteria overlook how the model interacts with the downstream data distribution. In this work, we formulate LoRA initialization as identifying the degree of impact of directions in parameter space under the target data distribution. We argue that data-aware sensitivity, rather than weight-only magnitude, should govern the choice of adaptation subspaces. Building on this perspective, we propose a Fisher-guided framework that leverages curvature information induced by downstream data to characterize how parameter perturbations influence model predictions. This perspective yields a principled, task-dependent criterion for selecting LoRA directions that better align adaptation with the target objective. Empirical results across diverse tasks and modalities demonstrate that data-aware initialization consistently and significantly improves downstream performance over existing approaches.

Figures

Figures reproduced from arXiv: 2605.01046 by Hung-Yu Kao, Ying-Jia Lin, Zhi-Quan Feng.

Figure 1
Figure 1. Figure 1: Experiments comparing singular-direction selection and magnitude-scaling strategies for LoRA initialization. For the 32 samples, panels (a) and (c) sort directions by singular values, while (b) and (d) sort them by their Fisher Energy values. Results are obtained on ARC-Challenge and BoolQ using Llama2-7B with rank = 32. The horizontal axis denotes the index of the sorted experiments. Scatter points show t… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed Fisher-Guided LoRA Initialization framework. The three subfigures correspond to its key components: (a) Fisher Factor Computation, where we compute the Fisher information using Kronecker-factored statistics using a minibatch of data; (b) Fisher-Aligned Direction Selection, where we identify Fisher-aligned directions by projecting onto surrogate bases derived from pre-trained weight… view at source ↗
Figure 3
Figure 3. Figure 3: Experimental results of varying LoRA ranks on Llama2- 7B. Average accuracy across reasoning tasks is reported. We further investigate the impact of LoRA rank on fine￾tuning performance [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Extra initialization time comparison. We report the total additional initialization time (in seconds) for different LoRA initialization methods using an input length of 512, rank r = 32, and BF16 precision, measured on a single NVIDIA A100 GPU. Three model scales are evaluated: Llama3.2-1B (“1B”), Llama3.2- 3B (“3B”), and Llama3-8B (“8B”). For KaSA, we report the full initialization time, whereas for LoRA-… view at source ↗
Figure 6
Figure 6. Figure 6: Full results of the ablation study on the Llama2-7B model. 80 160 320 480 640 Full 74.5 75.0 75.5 76.0 ACC BoolQ 80 160 320 480 640 Full 89.5 90.0 90.5 91.0 ACC PIQA 80 160 320 480 640 Full 82.0 82.5 83.0 ACC SIQA 80 160 320 480 640 Full 95.0 95.5 96.0 ACC HellaS. 80 160 320 480 640 Full 88.0 88.5 89.0 ACC WinoG. 80 160 320 480 640 Full 91.0 91.5 92.0 ACC ARC-e 80 160 320 480 640 Full 79.5 80.0 80.5 ACC AR… view at source ↗
Figure 7
Figure 7. Figure 7: Full results of the ablation study on the Llama3-8B model. The complete experimental results show that FILet exhibits robust and stable performance across a wide range of minibatch sizes. In general, increasing the minibatch size yields more accurate estimates of the empirical second-moment statistics, which in turn leads to improved downstream adaptation performance. Nevertheless, FILet remains competitiv… view at source ↗
Figure 8
Figure 8. Figure 8: Direction overlap matrices different tasks using Llama2-7B as the base model. From these visualizations, we observe that ”ARC-e” and ”ARC-c” exhibit a notably high degree of direction overlap, which is expected since they are essentially two subsets of the same benchmark. Beyond this pair, most task combinations display relatively low overlap in their selected adaptation directions, indicating that FILet c… view at source ↗
Figure 9
Figure 9. Figure 9: Direction overlap matrices different tasks using Llama3-8B as the base model. H. Limitations Compared to SVD-based initialization methods, FILet incurs additional memory overhead during the initialization phase to compute and store empirical second-moment statistics. While this overhead is not significant in most scenarios, it may become a practical challenge when adapting extremely large models or deployi… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. How Meta-Learning Shapes LoRA Adapter Geometry in Speech Deepfake Detection

    eess.AS 2026-07 conditional novelty 6.0

    Meta-learning training concentrates loss-relevant LoRA updates in query/key projections and spreads them in output projections, relative to standard empirical-risk training.

Reference graph

Works this paper leans on

11 extracted references · 11 canonical work pages · cited by 1 Pith paper · 3 internal anchors

  1. [1]

    Ji, Y ., Saratchandran, H., Gordon, C., Zhang, Z., and Lucey, S

    URL https://openreview.net/forum? id=nZeVKeeFYf9. Ji, Y ., Saratchandran, H., Gordon, C., Zhang, Z., and Lucey, S. Efficient learning with sine-activated low-rank matrices. In The Thirteenth International Conference on Learning Representations, 2025. URL https:// openreview.net/forum?id=cWGCkd7mCp. Kopiczko, D. J., Blankevoort, T., and Asano, Y . M. VeRA:...

    work page 2025

  2. [2]

    Krause, J., Stark, M., Deng, J., and Fei-Fei, L

    URL https://openreview.net/forum? id=NjNfLdxr3A. Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops, pp. 554–561, 2013. doi: 10.1109/ ICCVW.2013.77. Liu, S., Wang, C., Yin, H., Molchanov, P., Wang, Y .-C. F., Cheng, K.-T., and Chen, ...

    work page 2013

  3. [3]

    Revisiting Natural Gradient for Deep Networks

    URL https://openreview.net/forum? id=6ZBHIEtdP4. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381–23...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/d18-1260 2018

  4. [4]

    Gemma: Open Models Based on Gemini Research and Technology

    URL https://openreview.net/forum? id=RYrJqz44p4. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y ., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/ stanford_alpaca, 2023. Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi`ere...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  5. [5]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    URL https://openreview.net/forum? id=qEpi8uWX3N. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V ., Goyal, N., Hartshorn, A., Hosseini, S....

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/2025.naacl-long 2023

  6. [6]

    naacl-long.248/

    URL https://aclanthology.org/2025. naacl-long.248/. 12 Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning Wang, S., Yu, L., and Li, J. LoRA-GA: Low-rank adapta- tion with gradient approximation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/ forum?id=VaLAWrLHJv....

    work page doi:10.1145/3736407 2025

  7. [7]

    ISBN 979-8-89176-251-0

    URL https://openreview.net/forum? id=lq62uWRJjiY. Zhang, X., Zhao, J., Yang, Z., Zhong, Y ., Guan, S., Cao, L., and Wang, Y . UORA: Uniform orthogonal reinitialization adaptation in parameter efficient fine-tuning of large mod- els. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T. (eds.), Proceedings of the 63rd Annual Meeting of the Association ...

    work page doi:10.18653/v1/2025.acl-long 2025

  8. [8]

    arXiv preprint arXiv:2410.01870 , year=

    URL https://aclanthology.org/2025. acl-long.575/. Zheng, L., Chiang, W.-L., Sheng, Y ., Zhuang, S., Wu, Z., Zhuang, Y ., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2...

    work page arXiv 2025

  9. [9]

    15 Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning

    Computing the full gradientg∈R mn for each training example. 15 Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning

    work page

  10. [10]

    Forming the outer productgg ⊤ ∈R (mn)×(mn)

    work page

  11. [11]

    For example, if W is a projection layer with m= 4096 and n= 4096 , then mn≈1.68×10 7, and F contains (mn)2 ≈2.8×10 14 entries

    Averaging these outer products over the dataset. For example, if W is a projection layer with m= 4096 and n= 4096 , then mn≈1.68×10 7, and F contains (mn)2 ≈2.8×10 14 entries. Even the storage requirement is already prohibitive: (mn)2 entries×4bytes≫1TB for modern layer sizes. Large Language Models (LLMs) with billions of parameters would require storing ...

    work page 2022