Pith. sign in

REVIEW 3 major objections 6 minor 17 references

LoRA Is Slower Than You Think

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Freezing the lower eight layers and tuning only the upper 24 with PaCA matches LoRA on MMLU and cuts fine-tuning time by about two hours.

desk verdict Layer-frozen PaCA is a sensible idea, but the paper's own table shows the speedup depends entirely on an unexplained K=24 choice, and a single MMLU run is too thin to validate it. read the letter →

arxiv 2507.08833 v1 pith:KECBCWRV submitted 2025-07-06 cs.LG cs.AIcs.CL

Seokmin Ko This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords parameter-efficientfine-tuningLoRApartialconnectionadaptationlayerfreezingLLaMA2-7BMMLUGPUkernelbottlenecktrainingspeed
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that LoRA's efficiency is not a dependable training-speed win, and that a simpler scheme can beat it on wall-clock time without losing accuracy. On an A100 GPU, the authors measured LoRA forward and backward passes that were slower than full fine-tuning for several models, which they trace to the adapter branch running as extra sequential GPU kernels. They then freeze the lower layers of LLaMA2-7B and apply PaCA only to the upper 24 layers, doubling the column-mask density so the number of updated parameters stays equal to a full-model LoRA. On the MMLU benchmark (a broad multiple-choice knowledge test), this 'Three Quarters PaCA' setup reaches 52.02 average accuracy versus LoRA's 52.15, while training takes 5 hours 16 minutes instead of 7 hours 17 minutes. The practical point is that practitioners with limited GPUs may get LoRA-level accuracy faster by skipping adapters and updating only the most task-relevant upper layers.

What carries the argument

The carrying mechanism is a layer-freezing schedule combined with PaCA's column mask. PaCA updates weights by $W = W_0 - \eta (M \odot \Delta W)$, where $M$ is a binary mask selecting columns of the weight matrix; the paper freezes the bottom $L-K$ transformer layers and applies this masked update only to the top $K$ layers, setting the mask width to $2r$ columns per layer to keep total trainable parameters equal to LoRA of rank $r$. On the speed side, the argument turns on eliminating LoRA's sequential adapter kernels: LoRA's forward path computes $W_0 x + B A x$, which adds two small matrix multiplications per layer, whereas PaCA keeps the original weight path and only masks its gradient update, so frozen lower layers also remove their backward pass entirely.

What would settle it

On the accuracy side: fix K=24 before seeing results and rerun the LLaMA2-7B MMLU experiment; if Three Quarters PaCA falls more than about one point behind LoRA on a new seed or split, the upper-layer-priority premise fails. On the speed side: profile LoRA with the low-rank branch fused into the base GEMM; if the slowdown disappears, the sequential-kernel explanation is falsified.

Watch

Extended reading notes

Core claim

The central claim is that LoRA's parameter-count advantage does not automatically become a speed advantage: measured on an A100, LoRA's forward and backward passes took longer than full fine-tuning for GPT2-xl, GPT2-medium, and TinyLlama at batch size 4 (for GPT2-xl, 97.73 ms versus 61.89 ms forward). The paper's explanation is that the low-rank branch is computed as separate sequential GPU kernels rather than fused into the base matrix multiplication. The proposed remedy is selective non-adaptive fine-tuning: freeze the lower $L-K$ layers and apply PaCA only to the upper $K$ layers, using $2r$ columns per layer so the trainable parameter count matches a full-model LoRA of rank $r$. On LLaMA2-7B with MMLU, Three Quarters PaCA ($K=24$) scores 52.02 average accuracy in 5:15:44.61, versus LoRA's 52.15 in 7:17:01.79. The paper's own table also shows the layer choice is decisive: tuning only the upper 16 layers gains just 1.2 points over no tuning, while tuning the upper 24 matches LoRA.

Load-bearing premise

The result stands on the premise that the most task-critical parameters of a 32-layer Transformer all live in the upper 24 layers, and that this layer set can be chosen without first measuring both K=16 and K=24 outcomes; the paper's own table shows Upper Half PaCA gains only 1.2 MMLU points over no tuning, so the 2-hour speedup depends entirely on picking K=24.

Editorial extensions

If this is right

  • A practitioner who freezes the lower 8 layers of LLaMA2-7B and tunes the upper 24 with PaCA can expect MMLU accuracy within about 0.1 points of LoRA while saving roughly 2 hours of training time on a single GPU.
  • LoRA's speed should be benchmarked by wall-clock time on the target hardware, not inferred from parameter counts, because the adapter's extra kernels can erase its theoretical FLOP savings.
  • The more layers are frozen, the larger the speed gain, since backpropagation through those layers is skipped entirely in addition to their forward pass.
  • To preserve the parameter budget while freezing layers, the remaining layers must update proportionally more columns; PaCA's mask density provides that knob.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The upper-layer priority is probably task-dependent: MMLU is knowledge-heavy, so the same K=24 choice may not transfer to tasks that rely on lower-layer syntactic or lexical representations, and a validation-based K would be the natural fix.
  • If LoRA implementations fuse the low-rank branch into the base GEMM, the measured slowdown could shrink or vanish; the paper's bottleneck claim describes LoRA as typically implemented rather than a fundamental limit of low-rank updates.
  • The reported 2-hour saving on one epoch suggests larger cumulative savings over longer training runs, but the paper does not report energy or memory measurements, so those remain open quantities.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that LoRA, despite its theoretical parameter savings, does not consistently speed up LLM fine-tuning on GPUs because the low-rank adapter matrices are processed as sequential kernels. It proposes a selective non-adaptive fine-tuning scheme that freezes the lower L-K layers and applies PaCA only to the upper K layers, and reports experiments on LLaMA2-7B with the MMLU benchmark. The main result in Table 2 is that 'Three Quarters PaCA' (K=24) achieves 52.02 MMLU accuracy versus LoRA's 52.15, while reducing wall-clock training time by about 2 hours. The paper also presents per-layer complexity formulas for full fine-tuning and LoRA and a short ablation over K=16 and K=24.

Significance. If the central claim were established, the paper would give practitioners a practical recipe for cheaper fine-tuning: freeze the lower layers of a Transformer and apply a partial-connection update to the upper layers. The paper identifies a real and often overlooked issue—parameter-efficient methods do not automatically reduce wall-clock time on GPU hardware—and the proposed method is simple and measurable. However, the current evidence is not sufficient to support the advertised speed/accuracy trade-off: the comparison rests on single unreplicated runs, the layer-selection parameter K is chosen after observing the results, and the parameter-count matching is inexact. The paper would be more valuable with multi-seed evaluations, a principled layer-selection rule, and a clear timing methodology; as it stands, the empirical support is too thin to justify the general conclusion.

major comments (3)
  1. [Section 4, Table 2] The central claim that Three Quarters PaCA 'achieves performance comparable to both LoRA and full PaCA' is based on a single run with no error bars or repeated seeds; the 0.13-point MMLU gap between Three Quarters PaCA (52.02) and LoRA (52.15) is well within typical run-to-run variation, and no statistical significance is reported. In addition, the wall-clock times are presented without any timing methodology, warmup, repetition counts, or variance, so the reported 2-hour speedup is not established as reproducible.
  2. [Section 3, Table 2] The choice K=24 is post hoc and unvalidated. The paper's own data show a cliff: Upper Half PaCA (K=16) reaches only 47.10 MMLU, barely above no tuning (45.9), whereas Three Quarters PaCA (K=24) reaches 52.02, close to LoRA's 52.15. Since no selection criterion for K is provided and Section 5 delegates automated layer selection to future work, the advertised speedup is a property of a configuration chosen after seeing the results, not of a transferable method. MMLU is used both to select the configuration and to evaluate it, making the central claim circular in this respect.
  3. [Section 3, Table 3] The statement that the method doubles PaCA mask density 'to keep the total number of updated parameters identical to a full model LoRA of rank r' is inconsistent with the reported hyperparameters. For K=24 layers, matching the total parameter count of LoRA rank 8 over 32 layers would require a per-layer PaCA rank of 32*8*2/24 = 21.33, but Table 3 lists Rank 24, which gives about 12.5% more trainable parameters than full-model LoRA. Only the K=16 case (Rank 32) actually matches the claimed parameter count.
minor comments (6)
  1. [Title/Abstract] The title contains a typo: 'LoRA Is Slower Than Y ou Think' should read 'LoRA Is Slower Than You Think'.
  2. [Section 2.2] PaCA is introduced as learning a binary mask M, but the experiments appear to use a fixed column-selection scheme; please clarify whether the mask is learned or prescribed and how this affects the comparison.
  3. [Table 3] The learning rate entries '3e-4/1e-4' for Upper Half PaCA and Three Quarters PaCA are unexplained; please specify which layers or parameter groups receive which learning rate.
  4. [Tables 1 and 2] The speed measurements in Table 1 use an A100 GPU while the main experiments in Table 2 use an A6000 GPU, and no implementation details (library, kernel fusion, or whether LoRA weights are merged) are given; this makes it difficult to assess how the speed claim transfers across hardware and setups.
  5. [Section 4] Training for only one epoch on MMLU is a very short schedule, and the paper does not discuss whether the observed performance and timing trends persist with longer training or on other benchmarks.
  6. [Section 3] The phrase 'selective non-adaptive fine-tuning' is confusing because PaCA does update weights; a term such as 'partial-layer fine-tuning' would be clearer.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims are empirical benchmark comparisons against external baselines, with no equation defined in terms of the result it purports to prove.

full rationale

The central claims—that LoRA is not consistently faster and that Three Quarters PaCA matches LoRA accuracy while reducing training time—are direct observations from Tables 1 and 2, not derivations from fitted quantities. PaCA is cited from other authors (Woo et al., reference [14]), and the present paper introduces no parameter that is fit to a target and then renamed as a prediction. The freeze-lower-layers-and-apply-PaCA-to-upper-K recipe is an experimental configuration choice, not an equation whose output is equal to its input by construction. The concern that K=24 may have been selected after seeing results, and the absence of error bars on the 0.13-point MMLU gap, are real validity and reproducibility issues, but they are not circularity: the paper does not claim to derive the speedup or accuracy from an assumption that already contains them. No self-citation chain is load-bearing. Accordingly, the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No physical or architectural entities are introduced. The named method 'selective non-adaptive fine-tuning' is a configuration of existing PaCA plus layer freezing, not a new entity.

free parameters (2)
  • number of trainable upper layers K = 16 and 24 out of 32
    Chosen by hand in Section 3; the paper's favorable result depends on selecting K=24 after K=16 underperformed by 5 points.
  • PaCA rank and mask density per selected layer = 24 columns for Three Quarters PaCA, 32 for Upper Half PaCA, 16 for full PaCA
    Set in Table 3 to roughly match LoRA rank 8 parameter counts, but Three Quarters uses 1152 layer-column units versus 1024 for LoRA, so the comparison is not strictly parameter-matched.
assumptions (4)
  • domain assumption GPU execution is efficient only for large, few kernels; sequential small adapter kernels create a bottleneck.
    Stated in Section 1 as the cause of LoRA slowness, but no kernel profiling or ablation is provided to isolate this mechanism.
  • domain assumption The most task-critical parameters reside in the upper K transformer layers, so the lower layers can be frozen.
    Assumed in Section 3 without per-layer importance analysis; the paper's own Upper Half PaCA result (47.10 accuracy) casts doubt on this assumption.
  • domain assumption MMLU average accuracy after one epoch and a single run is a sufficient measure for comparing fine-tuning methods.
    Used in Section 4; no repeated runs, confidence intervals, or additional tasks are reported.
  • domain assumption The FLOP counts in Section 2 predict wall-clock time on A100 and A6000 hardware.
    The complexity analysis ignores kernel launch, memory bandwidth, and fused implementations; the paper's own Table 1 contradicts the FLOP ranking, so this assumption is internally inconsistent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LoRA Is Slower Than You Think." pith.science (2026). https://pith.science/paper/KECBCWRV

@misc{pith2026250708833,
  author       = {Pith},
  title        = {Pith review of: LoRA Is Slower Than You Think},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KECBCWRV}},
  note         = {Machine review of arXiv:2507.08833}
}
read the original abstract

Low-Rank Adaptation (LoRA) is one of the most widely used techniques for fine-tuning large language models (LLMs). By introducing a small number of trainable low-rank weight matrices, LoRA substantially reduces the number of parameters that need to be updated, offering significant advantages in memory consumption and computational efficiency compared to full fine-tuning. However, we observed that LoRA does not consistently provide speed improvements across all model architectures and training setups. Motivated by this inconsistency, we conduct a comprehensive analysis of LoRA's performance and investigate the underlying factors limiting its speedup. Based on our findings, we propose several methods for more efficient fine-tuning of LLMs. We empirically evaluate these methods and compare them to LoRA, demonstrating that our approach achieves comparable or superior performance while delivering more consistent training speed improvements. Our work offers valuable insights and practical guidelines for practitioners seeking to optimize LLM fine-tuning under resource constraints.

Figures

Figures reproduced from arXiv: 2507.08833 by the authors.

Figure 1
Figure 1. The Partial Connection Adaptation (PaCA) architecture [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Selective non-adaptive fine-tuning: only the top [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 7 canonical work pages

  1. [1]

    Run LoRA Run: Faster and Lighter LoRA Implementations

    Daria Cherniuk, Aleksandr Mikhalev, and Ivan Oseledets. Run lora run: Faster and lighter lora implementations. arXiv preprint arXiv:2312.03415, 2023

    work page Pith review arXiv 2023

  2. [2]

    Nvidia a100 tensor core gpu: Performance and innovation

    Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. Nvidia a100 tensor core gpu: Performance and innovation. IEEE Micro, 41(2):29–35, 2021

    2021

  3. [3]

    Accelerate gpu concurrent kernel execution by mitigating memory pipeline stalls

    Hongwen Dai, Zhen Lin, Chao Li, Chen Zhao, Fei Wang, Nanning Zheng, and Huiyang Zhou. Accelerate gpu concurrent kernel execution by mitigating memory pipeline stalls. In 2018 IEEE international symposium on high performance computer architecture (HPCA), pages 208–220. IEEE, 2018

    work page 2018

  4. [4]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088– 10115, 2023

    2023

  5. [5]

    Microsecond-scale preemption for concurrent{GPU-accelerated}{DNN} inferences

    Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. Microsecond-scale preemption for concurrent{GPU-accelerated}{DNN} inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 539–558, 2022

    work page 2022

  6. [6]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    arXiv 2009

  7. [7]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019

    2019

  8. [8]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1 (2):3, 2022

    2022

Show all 17 references
  1. [9]

    Fine-tuning transformers efficiently: A survey on lora and its impact

    Muchen Huan and Jianhong Shun. Fine-tuning transformers efficiently: A survey on lora and its impact. 2025

    2025

  2. [10]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  3. [11]

    Learning representations by back-propagating errors

    David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(6088):533–536, 1986

    1986

  4. [12]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 5

    2023 arXiv

  5. [13]

    Non-intrusive adaptation: Input- centric parameter-efficient fine-tuning for versatile multimodal modeling

    Yaqing Wang, Jialin Wu, Tanmaya Dabral, Jiageng Zhang, Geoff Brown, Chun-Ta Lu, Fred- erick Liu, Yi Liang, Bo Pang, Michael Bendersky, et al. Non-intrusive adaptation: Input- centric parameter-efficient fine-tuning for versatile multimodal modeling. arXiv preprint arXiv:2310.1...

    arXiv

  6. [14]

    Paca: Partial connection adaptation for efficient fine-tuning

    Sunghyeon Woo, Sol Namkung, Sunwoo Lee, Inho Jeong, Beomseok Kim, and Dongsuk Jeon. Paca: Partial connection adaptation for efficient fine-tuning. arXiv preprint arXiv:2503.01905, 2025

    2025 arXiv

  7. [15]

    Accurate and convenient energy measurements for gpus: A detailed study of nvidia gpu’s built-in power sensor

    Zeyu Yang, Karel Adamek, and Wesley Armour. Accurate and convenient energy measurements for gpus: A detailed study of nvidia gpu’s built-in power sensor. In SC24: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–17. IEEE, 2024

    2024

  8. [16]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

    Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199, 2021

    2021

  9. [17]

    A commentary of gpt-3 in mit technology review 2021

    Min Zhang and Juntao Li. A commentary of gpt-3 in mit technology review 2021. Fundamental Research, 1(6):831–833, 2021. 6

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.