REVIEW 4 major objections 5 minor 15 references
From LLMs to Edge: Parameter-Efficient Fine-Tuning on Edge Devices
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read On depthwise-separable convolutional models, PEFT memory savings shrink to about half the LLM advantage, while adapter updates still cut FLOPs by up to 95%.
desk verdict Useful FLOPs benchmark for PEFT on edge CNNs, but the headline memory-efficiency claim rests on a sum-of-peaks upper bound that overstates the true peak. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing tools are the paper's profiling extensions: a memory tracker that attributes peak memory to parameters, gradients, activations, optimizer state, and temporary buffers, and a FLOPs counter that separates forward, backward, and optimizer steps. These reveal the mechanism behind the headline result: in depthwise-separable convolution layers, filter grouping cuts forward-pass FLOPs by a factor of the input channels, but the input gradient still requires full computation, creating a large backward-to-forward FLOP asymmetry and leaving activations as the dominant memory cost; PEFT only removes gradient and optimizer memory, which is a small share in such models.
What would settle it
Run the paper's own profiler on a small language model and a MobileNet under identical batch size, sequence length, and memory-accounting rules; if the LLM's PEFT memory reduction turns out to be comparable to, rather than about double, the CNN's, the paper's central 'half as memory-efficient' comparison collapses. A simpler check would be measuring actual on-device peak memory for one update step to see whether the profiler totals hold in deployment.
Extended reading notes
Core claim
The paper's central claim is that PEFT efficiency is not a property of the method alone but of the interaction between the method and the convolution type. For depthwise-separable convolutions, activation memory—not gradients or optimizer state—dominates the peak memory of a forward-backward step, so the memory PEFT saves is a small slice of the total; the paper quantifies this as about half the memory efficiency reported for LLMs. For standard convolutions, gradient and optimizer memory dominate, and LoRA and DoRA recover most of their LLM-class memory savings. On the compute side, the backward pass of a depthwise-separable layer is about 20 times more expensive than its forward pass under full fine-tuning, and adapter methods compress this ratio to about 1.2:1, yielding up to 95% FLOPs reduction. Across tasks, GaLore is the most accurate and consistent but uses more memory and FLOPs; LoRA gives the best balance; DoRA's extra memory buys no accuracy; and rank effects are task-dependent, with lower ranks sometimes better.
Load-bearing premise
The claim that PEFT is only half as memory-efficient on these CNNs as on LLMs is measured against literature-reported LLM numbers from earlier work, so the ratio could change substantially under identical profiling settings and memory accounting.
Editorial extensions
If this is right
- On ResNet-style standard CNNs, LoRA and DoRA reproduce the memory reductions reported for LLMs, making them the natural choice when memory is the binding constraint.
- On MobileNet-style depthwise models, PEFT should be chosen for FLOPs rather than memory, because activation memory will dominate regardless of method.
- GaLore's usual LLM advantage over LoRA—lower memory via optimizer-state compression—does not transfer to these CNNs, where LoRA uses less optimizer memory.
- DoRA adds 29% to 58% training memory overhead on the models tested and never beats LoRA in accuracy, so for edge CNNs it is dominated by LoRA.
- Rank tuning has no monotonic benefit: at high pre-adaptation accuracy higher ranks can hurt GaLore, while at low pre-adaptation accuracy low-rank adapters can trail GaLore by up to 50%.
Reading between the lines
- Because activations are the bottleneck on depthwise models, combining PEFT with activation-reducing techniques—gradient checkpointing, selective layer freezing, or reversible blocks—should recover larger memory savings than adjusting PEFT rank; this is directly testable with the same profiler.
- The measured 20:1 backward-to-forward FLOP ratio points to the input-gradient computation in depthwise layers as the cost to attack; a backward-pass approximation that exploits channel grouping could cut update cost for all fine-tuning methods, not just adapters.
- The 'half as memory-efficient' figure is a cross-paper comparison with LLM numbers from earlier work; re-running the same profiler on a small transformer under identical memory accounting would put the factor on firmer ground and likely make it model-family specific.
- The finding that DoRA gains nothing over LoRA on these CNNs, despite gains reported for LLMs, suggests adapter decomposition benefits transfer poorly to compact vision models; extending the benchmark to lightweight transformers would show whether this is about convolution or about model scale.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper benchmarks parameter-efficient fine-tuning (PEFT) methods—LoRA, DoRA, GaLore, head-only fine-tuning with batch normalization (BN+H), and full fine-tuning (FFT)—on MobileNetV2, MobileNetV3, and ResNet-18 for distribution-shift and unseen-class tasks. It extends PyTorch's FLOPs counter and memory tracker to profile forward, backward, and optimizer steps for a single 224x224 image, and reports peak memory, FLOPs, and accuracy after 10 epochs with early stopping. The central claims are that PEFT memory savings are substantially smaller on depthwise-separable CNN architectures than on LLMs (roughly 'half as memory-efficient'), while adapter-based methods can reduce update FLOPs by up to 95% compared with full fine-tuning.
Significance. If the memory-related claims withstand correction, the paper provides a useful empirical reference for selecting PEFT methods for edge-deployed CNNs, a setting that is underrepresented relative to LLM-centric PEFT studies. The work has clear practical value: it covers standard architectures and tasks, includes rank sweeps, reports both memory and FLOPs for the profiled configurations, and makes the code publicly available. The observation that GaLore is more robust than LoRA on challenging corruptions for MobileNets, while being more expensive, is a concrete and potentially actionable finding. However, the headline memory-efficiency comparison rests on a nonstandard memory aggregation and an uncontrolled comparison with literature-reported LLM numbers, so the quantitative strength of the main claim is not yet established.
major comments (4)
- [Section 3.1, Memory] The profiler 'records the peak memory usage of each group regardless of when it occurs, with total memory computed as their sum.' This is not the true peak memory of a training step, because the five memory groups peak at different phases: activations during forward/backward, gradients during backward, and optimizer state at optimizer.step. Summing per-group maxima overestimates max_t sum_g alloc_g(t), and the overestimate is method-dependent because PEFT methods shift the relative contribution of each group. Since Figure 1 and the abstract's 'only half as memory-efficient' claim are computed from this upper bound, the memory numbers should be recomputed as a true max-over-time peak and the figures and conclusions updated accordingly.
- [Section 4, Memory and Figure 1] The comparison of CNN memory reductions with LLM reductions from Hu et al. (2021) and Zhao et al. (2024) is uncontrolled: the LLM numbers come from different papers, likely with different batch sizes, sequence lengths, profilers, and memory-accounting conventions. The 'half as memory-efficient' ratio is therefore not a well-defined measurement. Please either perform a controlled re-benchmark of an LLM under the same profiler and accounting, or explicitly reframe the statement as a rough cross-literature comparison with the necessary caveats and a precise definition of 'half.'
- [Section 4, Accuracy and Performance, Figures 3 and 4] Accuracy results are reported from a single run with early stopping after 10 epochs, without error bars, confidence intervals, or multiple seeds. Claims such as 'accuracy variations of up to 20%' for LoRA/DoRA on MobileNets and 'GaLore shows the most consistent accuracy' depend on differences that could be within run-to-run noise. Please add repeated-seed experiments and report mean and standard deviation, or otherwise demonstrate that the observed accuracy gaps are stable.
- [Abstract and Section 4, FLOPs] The abstract claims adapter-based PEFT methods 'can reduce floating point operations (FLOPs) during model updates by up to 95%,' but the text reports an 80% reduction for MobileNetV3 and a 57% reduction for ResNet-18. The paper does not identify which configuration yields the 95% figure. Please either point to the exact setting in Figure 2 or Figure 5 that produces 95%, or correct the claim to match the reported measurements.
minor comments (5)
- [Section 5, Related Work] The citation 'Frankle and et al.' is malformed and should be 'Frankle et al.'; please check the reference list for similar formatting issues.
- [Figure 3 caption] The caption says 'four different models' but the profiling and accuracy plots cover three architectures (MobileNetV2, MobileNetV3, ResNet-18). Please clarify whether the fourth model refers to a CIFAR-10-pretrained variant or correct the caption.
- [Section 2, DoRA description] The formula for DoRA's weight update is typeset ambiguously: 'W' = m V + ∆V / V + ∆V' should read W' = m (V + ∆V) / ||V + ∆V||_c. Please fix the notation.
- [Section 4, Summary] The sentence 'DoRA introduces a memory overhead compared to FFT' is inconsistent with Figure 1, where DoRA has lower peak memory than FFT for the profiled models. The intended comparison appears to be against LoRA, not FFT; please clarify.
- [Appendix, Table 2] The table lists hyperparameters but does not state the number of training epochs used for the rank sweep in Figure 4; the text says 5 epochs, while the main accuracy results use early stopping after 10 epochs. Please make the training protocol for each figure explicit.
Circularity Check
No significant circularity: the headline claims are direct measurements against public baselines and externally defined PEFT methods.
full rationale
The paper's central claims are empirical benchmarks. The memory-efficiency and FLOPs reductions (abstract; Section 4 Memory/FLOPs) are measured with a modified PyTorch profiler on MobileNetV2, MobileNetV3, and ResNet-18, and accuracy results are evaluated on public benchmarks (ImageNet, CIFAR-10-C, VWW) against standard implementations of LoRA, DoRA, and GaLore from external libraries (Hugging Face PEFT; GaLore pre-release). No parameter is fitted and then reported as a prediction: hyperparameters are fixed to values from external publications (Appendix Table 2), and no derived quantity is defined in terms of the target claim. The comparison of PEFT memory efficiency on CNNs to LLM numbers from Hu et al. (2021) and Zhao et al. (2024) is an uncontrolled external comparison, and the Section 3.1 accounting choice of summing per-group peak maxima is a measurement-validity concern, but neither reduces a result to its inputs by construction. The only self-citations (Corti et al. 2023; Saukh et al. 2023) appear in background statements about edge constraints and parameterized subspaces; they are not load-bearing for the benchmark results. No uniqueness theorem, ansatz, or prior-work conclusion is imported from the authors' own papers to force a conclusion. The paper's Limitations section acknowledges missing on-device profiling and hyperparameter tuning, which is a scope caveat, not circularity. I find no circular step that satisfies the requirement of exhibiting a specific reduction of a claimed result to its own inputs.
Assumptions & free parameters
free parameters (4)
- LoRA rank r =
4
- DoRA rank r =
4
- GaLore rank r and scale =
r=4, scale=0.25
- Early stopping patience =
10 epochs without validation improvement
assumptions (5)
- domain assumption Memory tracker group peaks can be summed to yield valid total peak memory
- domain assumption FLOP counts from operand shapes are a valid proxy for update cost
- domain assumption LLM memory-efficiency numbers from cited papers are directly comparable
- domain assumption Fixed hyperparameters from source papers provide a fair comparison
- domain assumption ImageNet-pretrained CNN weights are representative of edge deployments
Cite this review
Pith. "Pith review of From LLMs to Edge: Parameter-Efficient Fine-Tuning on Edge Devices." pith.science (2026). https://pith.science/paper/U2KPSTDY
@misc{pith2026250723536,
author = {Pith},
title = {Pith review of: From LLMs to Edge: Parameter-Efficient Fine-Tuning on Edge Devices},
year = {2026},
howpublished = {\url{https://pith.science/paper/U2KPSTDY}},
note = {Machine review of arXiv:2507.23536}
}
read the original abstract
Parameter-efficient fine-tuning (PEFT) methods reduce the computational costs of updating deep learning models by minimizing the number of additional parameters used to adapt a model to a down- stream task. While extensively researched in large language models (LLMs), their application to smaller models used on edge devices, such as convolutional neural networks, remains underexplored. This paper benchmarks and analyzes popular PEFT methods on convolutional architectures typically deployed in resource-constrained edge environments. We evaluate LoRA, DoRA, and GaLore for updating standard and depthwise convolutional architectures to handle distribution shifts and accommodate unseen classes. We utilize recently proposed PyTorch profilers to compare the updated model performance and computational costs of these PEFT methods with traditional fine-tuning approaches. With resource efficiency in mind, we investigate their update behavior across different rank dimensions. We find that the evaluated PEFT methods are only half as memory-efficient when applied to depthwise-separable convolution architectures, compared to their efficiency with LLMs. Conversely, when targeting convolu- tional architectures optimized for edge deployment, adapter-based PEFT methods can reduce floating point operations (FLOPs) during model updates by up to 95%. These insights offer valuable guidance for selecting PEFT methods based on hardware constraints, performance requirements, and application needs. Our code is online.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893,
arXiv 1907
-
[7]
D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261,
arXiv 1903
-
[8]
URL https://epoch.ai/blog/backward-forward-FLOP-ratio. A. Howard, M. Zhu, B. Chen, D. Kalenichenko, et al. Mobilenets: Efficient cnns for mobile vision applications. arXiv preprint arXiv:1704.04861,
-
[9]
E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, et al. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685,
- [11]
-
[13]
D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv:2006.10726,
arXiv 2006
-
[14]
URL https://arxiv.org/abs/2403.03507. Z. Zhou, X. Chen, E. Li, andet al.. Edge intelligence: Paving the last mile of artificial intelligence with edge computing. Proceedings of the IEEE, 107(8):1738–1762,
-
[15]
Baseline hyperparameters for the analyzed PEFT methods, consistent with Hu et al
11 Appendix Table 2:Hyperparameters. Baseline hyperparameters for the analyzed PEFT methods, consistent with Hu et al. (2021); Liu et al. (2024); Zhao et al. (2024). Parameter Value Description rLoRA 4 Rank of the low-rank adapter. αLoRA 4 Influence of the adapter result scaled byα/r. rDoRA 4 Rank of the low-rank adapter. αDoRA 4 Influence of the adapter ...
work page 2021
Show all 15 references
-
[2016]
Z. Han, C. Gao, J. Liu, J. Zhang, and S. Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey.arXiv preprint arXiv:2403.14608,
-
[2017]
Chowdhery, P
A. Chowdhery, P. Warden, J. Shlens, A. Howard, and R. Rhodes. Visual wake words dataset.arXiv preprint arXiv:1906.05721,
1906 arXiv
-
[2018]
Saukh, D
O. Saukh, D. Wang, X. He, and L. Thiele. Representing input transformations by low-dimensional parameter subspaces. arXiv:2305.13536,
-
[2019]
Corti, B
F. Corti, B. Maag, J. Schauer, U. Pferschy, and O. Saukh. REDS: Resource-efficient deep subnetworks for dynamic resource constraints.arXiv preprint arXiv:2311.13349,
-
[2021]
Frankle andet al
9 J. Frankle andet al.. Training batchnorm and only batchnorm: On the expressive power of random features in cnns. arXiv:2003.00152,
2003 arXiv
-
[2023]
URL https://arxiv.org/abs/2308.06522. P. Busto and G. Juergen. Open set domain adaptation. InICCV, pages 754–763,
-
[2024]
S. Niu, C. Miao, G. Chen, P. Wu, and P. Zhao. Test-time model adaptation with only forward passes.arXiv preprint arXiv:2404.01650,
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.