REVIEW 3 major objections 6 minor 2 cited by
Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read This paper claims that activation maps are stable enough across training steps that a single warm-started subspace iteration can replace per-step HOSVD, cutting activation memory up to 120.09x and training FLOPs up to 1.86x while keeping…
desk verdict A genuinely useful idea for activation-memory reduction, but the headline memory numbers count only stored low-rank factors and may not reflect real peak memory; deserves a revision, not a desk reject. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is ASI's warm-started single subspace iteration applied per mode of the four-dimensional activation tensor. At the first step the right factor is a random Gaussian matrix; at later steps it is reused from the previous training step before one matrix multiplication and one Gram-Schmidt orthogonalization (Algorithm 1). This produces factors $U$ and a core $S$ that replace the stored activation map, with memory cost equal to the core plus the factors. Rank selection is a separate offline stage: for each explained-variance threshold the method computes the Frobenius-norm difference between the true weight gradient and the gradient computed from the compressed activation (its activation perplexity), then a backtracking search over the resulting perplexity matrix picks ranks that satisfy the memory budget. This machinery makes the per-step compression overhead $O(2abr + r^3)$ per mode instead of the much larger HOSVD cost, which is what allows total training FLOPs to drop below vanilla training.
What would settle it
Run ASI with a fixed rank budget and double the learning rate or switch downstream data halfway through fine-tuning; if the warm-started subspace cannot track the faster drift, the gap between ASI and vanilla training accuracy should widen beyond the roughly one-point difference reported for stable schedules. A more direct check is to measure the principal-angle distance between activation subspaces in consecutive steps and test whether one subspace iteration keeps the gradient reconstruction error under the threshold implied by the chosen perplexity values.
Extended reading notes
Core claim
The paper claims that activation maps change only marginally across consecutive training steps, so a single subspace iteration with a warm start can track the dominant subspace accurately enough for gradient computation. Given optimal ranks selected offline, ASI decomposes each activation tensor by performing one orthogonalized subspace step per mode and stores only the resulting low-rank factors; gradients are then computed in the compressed space. The paper's experiments show that with a pre-selected budget, ASI matches the accuracy of HOSVDε and approaches vanilla training, while consuming up to 120.09x less activation memory than vanilla and up to 1.86x fewer total FLOPs; on a Raspberry Pi 5 it is about 91x faster per training step than HOSVDε and 1.56x faster than vanilla. The paper also reports extending the method to fine-tuning TinyLlama 1B, where five layers trained with rank 20 give roughly 2500x activation-memory reduction and a near-halving of training FLOPs at a small accuracy cost.
Load-bearing premise
The method assumes activation maps change only a little from one training step to the next, so the previous step's projection subspace remains a good warm start for a single subspace iteration; if the subspace drifts quickly—early in training, under large learning rates, or under distribution shift—the approximation would degrade and accuracy would suffer.
Editorial extensions
If this is right
- With ASI, fine-tuning four layers of a compact CNN stays within roughly one accuracy point of vanilla training while using tens to hundreds of times less activation memory, so deeper fine-tuning fits in a small fixed budget.
- Because the backward pass runs in the compressed space, total training FLOPs can be below vanilla training (up to 1.86x fewer), reversing the usual memory-for-compute trade-off in activation compression.
- A memory budget can be set before training starts, which matters for deployment: the user knows the peak activation memory in advance, unlike variance-threshold methods whose memory use varies during training.
- On-device latency improves dramatically: on a Raspberry Pi 5, ASI trains roughly 91x faster per iteration than HOSVDε and 1.56x faster than vanilla training on the same MCUNet task.
- The mechanism is not limited to convolution: fine-tuning five layers of a 1B language model with rank 20 cuts activation memory by about 2500x and FLOPs by about 1.9x while losing about 1.5 accuracy points, suggesting ASI transfers to attention-based architectures.
Reading between the lines
- My inference: the same stability argument suggests ASI could run the warm-started iteration only every $k$ steps instead of every step, potentially cutting overhead further; the paper does not test this schedule.
- My inference: because the perplexity measure is defined on gradients, it could be reused to reallocate ranks dynamically if the subspace does drift—for example after a learning-rate warm-up or a dataset switch—although the paper fixes ranks once before training.
- My inference: the LLM result hints that attention output activations also have stable dominant subspaces; a direct test would compare ASI with standard low-rank adapters on a longer fine-tuning run, which the paper does not include.
- My inference: replacing the backtracking rank search with a greedy or submodular allocation would let the method scale to hundreds of fine-tuned layers, addressing the brute-force limitation the paper itself notes in its appendix.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ASI (Activation Subspace Iteration), a method to reduce activation memory and training FLOPs for on-device fine-tuning of deep convolutional networks. The approach replaces per-step HOSVD decomposition of activation tensors with a single subspace iteration that is warm-started from the previous step, and it selects per-layer ranks once before training using a gradient-based perplexity measure under a memory budget. Experiments on MCUNet, ResNet, MobileNetV2, SwinT, and segmentation models claim up to a 120.09x activation-memory reduction and up to a 1.86x FLOP reduction relative to vanilla training, with a real-time deployment study on a Raspberry Pi 5.
Significance. If the claims hold, this is a practically relevant contribution to on-device learning: it addresses the activation-memory bottleneck without the per-step HOSVD overhead, which had limited prior low-rank compression approaches. The paper provides an extensive experimental evaluation across many architectures and datasets, includes real hardware measurements on a Raspberry Pi 5, and makes code available. The warm-start ablation is a useful control. The central ideas—offline rank selection via a gradient-perplexity proxy and single-step subspace tracking—are plausible and could benefit the community beyond this specific implementation. However, as detailed below, the memory-accounting methodology and the lack of derivation for the FLOP formulas need to be addressed before the quantitative claims can be fully credited; at present the measured benefits are partially overstated, though the overall approach remains defensible.
major comments (3)
- [Sec. 4.3, Eq. (5), Algorithm 1] The headline memory reduction (up to 120.09x, Fig. 4) is computed from Eq. (5), which counts only the storage of the low-rank core tensor plus factor matrices. But Algorithm 1 takes the full activation tensor A_i^(t) as input and performs four mode-wise subspace iterations; the full tensor must exist in memory before compression. In a sequential network, once A_i is produced, it coexists with previously stored compressed activations of other layers until the mode products finish, so the true peak activation memory is at least max_i size(full A_i) plus the sum of the low-rank tensors, not the sum of the low-rank tensors alone. For the MCUNet/Pets 2-layer case (Table 2), vanilla activation memory is 27.56 MB and the reported ASI memory is 0.14 MB; if even one full activation map of a fine-tuned layer is a substantial fraction of 27.56 MB, the realized peak-memory reduction is far less than 196x. The 'Result Logging' paragraph states that ASI and vanilla memory are reported as constants, confirming that the numbers are formula-based rather than measured high-water marks. The authors should either provide a fused/streaming compression procedure that avoids materializing the full activation tensor, or revise the memory claims to report the true peak memory during forward/backward passes.
- [Appendix A.3, Eqs. (14)-(18)] The computational-savings claims (up to 1.86x fewer FLOPs in Sec. 4.3 and the on-device speedups in Sec. 4.4) rest entirely on the closed-form FLOP expressions OASI, CASI, Ovanilla, and Cvanilla. These formulas are stated without derivation, and several terms are not self-evident (for example, the r^3 term in Eq. (14) and the individual terms in Eq. (15) for CASI). Since the paper's speedups are sensitive to these counts, the authors should provide a step-by-step derivation or a citation to a complete derivation, and ideally validate the counts with profiling tools or hardware measurements.
- [Sec. 3.4 and Sec. 4.2] The core premise of ASI is that a single subspace iteration with a warm start can track the dominant activation subspace because activation maps change only marginally between consecutive training steps. The paper justifies this by citing the Lipschitz continuity of activation functions and the small-update assumption on weights, but it does not empirically study how the subspace drifts during training, especially under large learning rates or distribution shifts in the fine-tuning data. The ablation in Sec. 4.2 shows that warm-start improves accuracy on CIFAR-10 on average, but it does not identify regimes where the single iteration might fail. Since the method's accuracy guarantees depend on this tracking assumption, an analysis of subspace drift (e.g., measuring subspace distance between consecutive steps) or experiments with more aggressive learning rates / earlier training would materially strengthen the evaluation.
minor comments (6)
- [Abstract and Sec. 4.3] The paper claims 'up to 120.09x' memory reduction, but Table 2 shows MCUNet/Pets with 2 layers at 27.56 MB versus 0.14 MB, which is about 196.9x; please clarify the exact setting that yields the 120.09x figure or correct the claim.
- [Appendix A.3, Eq. (14)] The notation 'd' = D\{d}' is unclear; it should mean the product of all dimensions except d, and should be written as d' = prod_{d'' in D, d'' != d} d'' for clarity.
- [Appendix A.2] There are typos: 'yeilds' should be 'yields' and 'analitically' should be 'analytically'.
- [Appendix B.1 and references] The Pets dataset is cited as Zhang et al. (2022) with the title '0/1 deep neural networks via block coordinate descent', which does not correspond to the Oxford Pets dataset used in the experiments; please use the appropriate reference (e.g., Parkhi et al., 2012).
- [Sec. 4.1, memory budget] The text says the peak memory consumption of HOSVD is used as the memory budget for ASI, but Sec. 4.3 then says ASI never reaches that threshold because HOSVD memory is not uniform; please clarify whether the budget is the reported peak across training or the value at a specific epoch.
- [General] All reported accuracies are single runs without variance or significance testing; adding multiple seeds and reporting mean/std would make the accuracy comparisons more robust.
Circularity Check
No significant circularity: ASI is an empirical method evaluated on external benchmarks, and its claimed savings are measured outputs rather than fitted inputs.
full rationale
I find no circular step in the paper's derivation chain. The central contribution is an empirical algorithm: rank selection uses a gradient-difference perplexity proxy, but ASI's accuracy is then evaluated on held-out data (ImageNet, CIFAR, Pets, segmentation, BoolQ), so the reported results are not fed back into the method. The warm-start subspace iteration is an explicit stability assumption borrowed from PowerSGD and Lipschitz-continuity results, not a self-imported uniqueness theorem. Self-citations to Nguyen et al. (2024) supply the HOSVD baseline and low-rank gradient formulas, but ASI's own accuracy and FLOP measurements are produced independently by the experiments in Tables 1-4, so these citations are not load-bearing for the main claim. The skeptic's memory-accounting objection is a measurement concern rather than circularity: Eq. (5) and the 'peak activation memory' axis in Fig. 4 count only the stored low-rank representation, while Algorithm 1 must materialize the full activation tensor as input. This could overstate the numerical saving, but it does not make the central claim equivalent to its inputs by definition. Appendix C candidly notes the backtracking search cost, which is a scalability limitation, not a circular step.
Assumptions & free parameters
free parameters (4)
- Per-layer/mode ranks r_i,m selected by perplexity backtracking =
reported per experiment, e.g., MCUNet 4-layer settings
- Memory budget B =
set to HOSVDepsilon's peak memory in main experiments
- Explained-variance threshold set E =
E = {0.4, 0.5, 0.6, 0.7, 0.8, 0.9}
- LLM compression rank =
20
assumptions (4)
- domain assumption Activation maps Ai are stable over a small number of training steps, so a warm-started subspace iteration tracks their dominant subspace.
- domain assumption The perplexity Pi,j, defined as the Frobenius norm of the difference between true and low-rank gradients, is a valid proxy for final accuracy loss due to compression.
- standard math Subspace iteration with warm start yields a low-rank approximation comparable to HOSVD.
- standard math Backpropagation requires storing activation maps during the forward pass.
Cite this review
Pith. "Pith review of Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning." pith.science (2026). https://pith.science/paper/GUBF6KXY
@misc{pith2026250505086,
author = {Pith},
title = {Pith review of: Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/GUBF6KXY}},
note = {Machine review of arXiv:2505.05086}
}
abstract
On-device learning has emerged as a promising direction for AI development, particularly because of its potential to reduce latency issues and mitigate privacy risks associated with device-server communication, while improving energy efficiency. Despite these advantages, significant memory and computational constraints still represent major challenges for its deployment. Drawing on previous studies on low-rank decomposition methods that address activation memory bottlenecks in backpropagation, we propose a novel shortcut approach as an alternative. Our analysis and experiments demonstrate that our method can reduce activation memory usage, even up to $120.09\times$ compared to vanilla training, while also reducing overall training FLOPs up to $1.86\times$ when evaluated on traditional benchmarks.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 2 Pith papers
-
Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMs
A training-free compression pipeline that updates calibration activations layer by layer and re-allocates ranks iteratively improves compressed LLM accuracy by up to a few points on five zero-shot tasks.
-
CosmosAlign: Adapting a World Foundation Model for Generative Traffic Video Forecasting
CosmosAlign adapts Cosmos3-Nano with two-stage LoRA, medoid sample selection, and motion-adaptive blending, achieving first place (76.49) on the AI City Challenge 2026 Track 5 traffic video forecasting benchmark.
Reference graph
Works this paper leans on
-
[4]
Performance comparision between vanilla training and ASI when fine-tuning TinyLlama 1B with BoolQ dataset. #Layers Vanilla training ASI (rank=20) Acc↑ Mem (MB)↓ TFLOPs↓ Acc↑ Mem (MB)↓ TFLOPs↓ 1 65.94 1408 3.02 64.69 0.51 1.68 2 66.37 1920 6.04 64.81 0.74 3.33 3 66.91 2432 9.07 65.00 0.98 4.98 4 67.44 3840 12.09 66.31 1.49 6.66 5 67.78 4352 15.11 66.34 1.7...
work page 1920
-
[6]
Federated learning for mobile keyboard prediction
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., and Ramage, D. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604,
-
[8]
J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W
Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,
-
[9]
Alora: Allocating low-rank adaptation for fine-tuning large lan- guage models
Liu, Z., Lyn, J., Zhu, W., Tian, X., and Graham, Y . Alora: Allocating low-rank adaptation for fine-tuning large lan- guage models. arXiv preprint arXiv:2403.16187,
-
[11]
Pau, D. P. and Aymone, F. M. Suitability of forward-forward and pepita learning to mlcommons-tiny benchmarks. In 2023 IEEE International Conference on Omni-layer In- telligent Systems (COINS), pp. 1–6. IEEE,
work page 2023
-
[12]
C., Greenewald, K., Lee, K., and Manso, G
Thompson, N. C., Greenewald, K., Lee, K., and Manso, G. F. The computational limits of deep learning. arXiv preprint arXiv:2007.05558, 10,
arXiv 2007
-
[13]
Valipour, M., Rezagholizadeh, M., Kobyzev, I., and Ghodsi, A. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558,
-
[14]
The caltech-ucsd birds-200-2011 dataset,
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd birds-200-2011 dataset,
work page 2011
Show all 22 references
-
[15]
Asvd: Activation-aware singular value decomposition for compressing large language models
Yuan, Z., Shang, Y ., Song, Y ., Wu, Q., Yan, Y ., and Sun, G. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821,
-
[16]
Y ., and Xiu, N
Zhang, H., Zhou, S., Li, G. Y ., and Xiu, N. 0/1 deep neural networks via block coordinate descent. arXiv preprint arXiv:2206.09379,
-
[17]
Tinyllama: An open-source small language model
Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385,
-
[21]
Perplexity as a function of the explained variance threshold ε for the last layers of MCUNet. Fig. 6 illustrates the change in perplexity of the last four layers of MCUNet when evaluated with different explained variance thresholds ε. As expected, higher values of ε result in ...
2024
-
[64]
The learning rate increases linearly over four warm-up epochs, reaching 0.005, and then follows a cosine annealing decay schedule
The loss function is cross-entropy. The learning rate increases linearly over four warm-up epochs, reaching 0.005, and then follows a cosine annealing decay schedule. Other Datasets. The models are first pretrained on ImageNet before being fine-tuned on a completely different ...
2023
-
[2016]
The forward-forward algorithm: Some prelim- inary investigations
Hinton, G. The forward-forward algorithm: Some prelim- inary investigations. arXiv preprint arXiv:2212.13345,
-
[2017]
Boolq: Exploring the surprising difficulty of natural yes/no questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044,
1905 arXiv
-
[2018]
Warm start
and the “tiny” incremental nature of parameter updates during optimization. By leveraging the previous approximation, we effectively smooth the sequence of activation maps across iterations, reducing the variance of the low-rank approximation compared to the case without reuse...
2024
-
[2019]
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition,
2009
-
[2020]
Car- bon emissions and large neural network training
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.- M., Rothchild, D., So, D., Texier, M., and Dean, J. Car- bon emissions and large neural network training. arXiv preprint arXiv:2104.10350,
-
[2021]
Sparse low-rank adaptation of pre-trained lan- guage models
Ding, N., Lv, X., Wang, Q., Chen, Y ., Zhou, B., Liu, Z., and Sun, M. Sparse low-rank adaptation of pre-trained lan- guage models. arXiv preprint arXiv:2311.11696,
-
[2022]
Rethinking atrous convolution for semantic image seg- mentation
Chen, L.-C., Papandreou, G., Schroff, F., and Adam, H. Rethinking atrous convolution for semantic image seg- mentation. arXiv preprint arXiv:1706.05587,
-
[2023]
M., Hauth, A., Millican, K., et al
Gemini Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805,
-
[2024]
Adalora: Adaptive budget allocation for parameter-efficient fine- tuning
Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y ., Chen, W., and Zhao, T. Adalora: Adaptive budget allocation for parameter-efficient fine- tuning. arXiv preprint arXiv:2303.10512,
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.