Pith. sign in

REVIEW 4 major objections 5 minor 12 references

InstantFT: An FPGA-Based Runtime Subsecond Fine-tuning of CNN Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An FPGA-based fine-tuning method adapts a small CNN in 0.36 seconds, matching low-rank accuracy.

desk verdict Real subsecond fine-tuning on FPGA, but the forward cache only pays off under batch replay, and the 17.4x figure is against the CPU, not LoRA. read the letter →

arxiv 2506.06505 v1 pith:NBTYT5KM submitted 2025-06-06 cs.LG cs.AR

Keisuke Sugiura , Hiroki Matsutani This is my paper
classification cs.LGcs.AR
keywords FPGACNNfine-tuninglow-rankadaptationforwardcacheconceptdrifton-devicelearningNF4quantizationIoT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

InstantFT sets out to close the gap between how fast a small CNN can run inference and how fast it can adapt. It restructures low-rank fine-tuning so the frozen body of the network is forwarded once per input and its intermediate activations are cached, while the only trainable pieces are low-rank adapters that jump directly to the output layer. The paper reports that on an embedded FPGA this brings 10 epochs of fine-tuning down to 0.36 seconds, 17.4x faster than running the same method on the board's CPU, with accuracy comparable to full low-rank adaptation on rotated MNIST, rotated Fashion-MNIST, and SVHN. If the result holds, deployed vision models on resource-limited devices can be re-fitted to concept drift within a second rather than minutes.

What carries the argument

Two mechanisms carry the argument. First, every LoRA adapter connects directly to the final layer, so backpropagation only computes gradients with respect to the small adapter matrices and never propagates activation gradients through the frozen network. Second, the Forward Cache stores the frozen network's intermediate activations and unadapted logits in 4-bit NF4 after the first epoch, so later epochs reuse them instead of recomputing the full forward pass. The output identity $x_L = \hat{x}_L + \sum_i B_{i,L} A_{i,L} x_i$ is what separates the once-per-input frozen computation from the per-epoch adapter computation, and the direct-to-final-layer wiring is what lets the FPGA hardware run all adapter computations in parallel.

What would settle it

Run the same fine-tuning task on the same FPGA with the Forward Cache disabled, or with a stream of 1024 unique samples that are never shown twice; if the wall-clock time grows by roughly the reported 5.1x cache factor and 10 epochs no longer complete in 0.36 seconds, the subsecond result is an artifact of replay rather than of the adapter design.

Watch

Extended reading notes

Core claim

The paper's central claim is that fine-tuning need not pay the cost of the frozen network in every epoch. InstantFT writes the network output as $x_L = \hat{x}_L + \sum_{i=0}^{L-1} B_{i,L} A_{i,L} x_i$, where $\hat{x}_L$ is the output of the untouched pre-trained network and each term is a low-rank adapter from an intermediate activation directly to the last layer. Because $\hat{x}_L$ and the intermediate activations are input-dependent but not parameter-dependent during fine-tuning, they can be computed once, quantized to 4-bit NF4, and cached; later epochs need only the cheap adapter paths and the adapter-side gradients. The authors report that this design, implemented on an FPGA, fine-tunes a LeNet-5-like CNN for 10 epochs in 0.36 seconds, 17.4x faster than the CPU-backed version, and 16.3x more energy-efficient, while staying within a few points of LoRA-All accuracy (91.8% on rotated MNIST at 90 degrees, 77.9% on rotated Fashion-MNIST, and 79.0% on SVHN under the hardest settings).

Load-bearing premise

The headline speed rests on a fixed, replayable dataset: the frozen network is run once per input and the results are cached, so anything that makes inputs unique or non-repeating removes the main accelerator.

Editorial extensions

If this is right

  • Fine-tuning a small CNN can complete in roughly the time of a few inference passes, so a deployed model can be updated between user interactions or sensor readings.
  • Because accuracy matches LoRA-All while backward FLOPs are far lower, the speed does not come from restricting adaptation to the last layer; adapters distributed across layers remain trainable.
  • The NF4 quantized cache brings the replay buffer for 1024 samples to about 1 MB on the rotation benchmarks, meaning the cached approach fits in the DRAM of small embedded boards.
  • With 16.3x better energy-efficiency than the CPU implementation, repeated on-device adaptation becomes plausible for battery-powered IoT nodes.
  • The FPGA implementation reaches subsecond fine-tuning even when including data-transfer and kernel-invocation overheads, which is the number that matters for real deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the deployment is a true one-pass stream rather than a replayed dataset, the Forward Cache saves nothing for unseen inputs: every new sample still requires the full frozen-network forward pass, so only the backward-pass savings remain and the 0.36-second figure is tied to the replay setting.
  • Cache size grows linearly with dataset size and model depth; the paper's own numbers (1.02 MB for 1024 rotated samples, 72.89 MB for SVHN's 73k samples) suggest that larger or higher-resolution inputs would need stronger quantization or cache eviction to keep the subsecond claim.
  • A direct experiment would run the same FPGA core with the cache disabled; from the paper's component breakdown (roughly 5.1x from the cache and 4.8x from the backward-pass redesign), the wall-clock time would grow to a few seconds, separating algorithmic savings from hardware acceleration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents InstantFT, an FPGA-based parameter-efficient fine-tuning method for small CNNs on an embedded platform (Xilinx Kria KV260). The method combines LoRA-style adapters that connect each intermediate layer directly to the final layer with a Forward Cache that stores and reuses the frozen network's intermediate activations across epochs, using NF4 quantization to reduce cache size. On a LeNet-5-like model for RotMNIST, RotFMNIST, and SVHN, reported results show accuracy comparable to LoRA-All (91.8/77.9/79.0% vs 91.2/75.1/81.7% in the hardest settings), and the FPGA implementation fine-tunes a 1024-sample batch for 10 epochs in 0.36s. The paper claims this is 17.4x faster than the CPU counterpart and 16.3x more energy-efficient, motivating 'on-the-fly adaptation of CNNs to non-stationary data distributions.'

Significance. If the reported measurements are reproducible, the paper makes a useful engineering contribution: subsecond batch fine-tuning of a small CNN on an embedded FPGA at accuracy close to full LoRA fine-tuning is demonstrated. The hardware implementation details are concrete, the FLOP and memory arithmetic in Table 1 is internally consistent, and the accuracy comparison against LoRA-All on two of the three datasets supports the claim of comparable accuracy. However, the scope is narrower than the framing suggests: the speedup relies on a replayable fixed dataset, and no streaming or single-pass evaluation is provided. The significance is therefore real but conditional on the batch-adaptation use case.

major comments (4)
  1. [Abstract; Section 5.3] The abstract states that InstantFT 'fine-tunes a pre-trained CNN 17.4x faster than existing Low-Rank Adaptation (LoRA)-based approaches,' but Section 5.3 reports the 17.4x figure as the speedup of the FPGA implementation over the CPU counterpart of InstantFT (0.36s vs the CPU implementation). The speedup over LoRA-All reported in the body is 4.8x on RotMNIST, measured on the ARM CPU, which is a different comparison. Because the 17.4x number is the headline quantitative claim, the abstract should either be corrected to say '17.4x faster than the CPU implementation' or the paper should provide an FPGA-versus-FPGA LoRA baseline.
  2. [Section 3.1; Section 5.1] The Forward Cache is load-bearing for the subsecond claim. Section 3.1 states that the cache stores intermediate results 'for a fixed dataset,' and Section 5.1 fine-tunes on 1024 fixed samples replayed for 10 epochs. The abstract and introduction invoke 'on-the-fly adaptation ... to non-stationary data distributions,' which implies a streaming setting. In a one-pass stream, each new input is encountered once, so the cache is never reused; the full forward pass of the frozen network (33.7-40.7x the forward FLOPs of the adapters, Table 1) is incurred for every sample, and the 0.36s figure is not attainable. The paper should either explicitly scope the claims to batch fine-tuning on a collected sample set or evaluate a streaming protocol.
  3. [Section 5.5; Section 5.6] The forward cache size grows with dataset size and activation dimensions. Section 5.5 reports 1.02 MB and 72.89 MB after NF4 quantization for the two evaluated settings, and Section 5.6 concludes that 'larger-scale networks can be fine-tuned on KV260' based on low LUT/DSP utilization. That conclusion ignores the cache: a larger model or a larger replay buffer will increase cache memory and DRAM bandwidth requirements, and the 4GB DDR4 capacity may become the binding constraint. A quantitative discussion of this trade-off is needed before making the scalability claim.
  4. [Section 5.2] The accuracy claim 'comparable or even better than LoRA-All' is only partially supported. In the hardest SVHN setting, InstantFT reaches 79.0% vs LoRA-All's 81.7%, a 2.7-point gap, and its accuracy drop of 9.4% is larger than LoRA-All's 6.7%. The paper should explicitly report this case as one where InstantFT is worse than LoRA-All, and the 'comparable' claim should be qualified accordingly.
minor comments (5)
  1. [Section 3.1, Eq. (1)] The update is written as B_{i,L}A_{i,L}x_i in Eq. (1), but the text defines W_{i,L}=A_{i,L}B_{i,L}; this is inconsistent with the matrix dimensions given in Section 2 (A in R^{r x d}, B in R^{d x r}). Please correct the order in the definition.
  2. [Section 5.2] The paper reports the average of ten runs but does not provide standard deviations or error bars in Fig. 7a; given the small accuracy differences (e.g., 91.8 vs 91.2), error bars would help assess whether the differences are significant.
  3. [Section 5.3] The sentence 'InstantFT performs fine-tuning with only a 17% increase in execution time compared to the inference-only case' does not specify whether this is measured on CPU or FPGA, or with or without cache quantization; please clarify the measurement conditions.
  4. [Introduction and References] The relationship to the authors' earlier Skip2-LoRA [3] is not discussed; since both methods target lightweight on-device fine-tuning, the paper should state the differences explicitly in the introduction or related work.
  5. [Section 5.1; Section 5.3] The SVHN fine-tuning dataset size is not stated clearly in the setup; Section 5.3 refers to '73257/1024 images' in passing. Please state the exact number of fine-tuning samples used for the SVHN case.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the speed and accuracy claims are backed by external benchmarks and measured ablations, and the self-citations are background only.

full rationale

The central claims of InstantFT are empirical measurements, not derivations from a fitted input. The 0.36s fine-tuning time on Kria KV260 is a measured number for a concrete FPGA implementation, and the 17.4x figure in Section 5.3 is explicitly a comparison against the same method running on the ARM CPU, not a quantity that is true by construction. Accuracy is compared against externally defined baselines (LoRA-All, LoRA-Last, FT-All, FT-Last, FT-Bias) on MNIST, Fashion-MNIST, and SVHN, so the results are not self-referential. The Forward Cache is a memoization optimization: Section 3.1 states that intermediate results are computed once and reused, and Section 5.3 quantifies its contribution as a 5.1x speedup via ablation; this is a measured efficiency gain, not a fitted parameter renamed as a prediction. The self-citations [2], [3], and [8] are used only to point to prior work on on-device PEFT and to justify that ELMs are limited to small models; none of these references supplies the load-bearing premise of the paper, since the benchmark numbers stand on the measurements reported here. The abstract's phrase "17.4x faster than existing Low-Rank Adaptation (LoRA)-based approaches" is a reporting inconsistency, because the 17.4x figure in Section 5.3 compares InstantFT on FPGA to InstantFT on CPU, not to LoRA; however, this is a correctness or framing issue, not circularity. The limitation stated in Section 6 that evaluation is on small-scale networks, and the observation that the Forward Cache assumes a fixed, replayable dataset, are scope concerns that do not make any prediction equivalent to its inputs by definition. Therefore, no circular step is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical systems contribution, so the ledger is about its premises rather than fitted constants. The headline result (0.36s fine-tuning, 4.8x over LoRA-All on the same CPU, comparable accuracy) is measured, not derived, and no parameter is fitted to produce it. What the claim rests on: the frozen-backbone PEFT premise, the fixed-batch replay assumption that powers the forward cache, standard LoRA gradient equations, NF4 quantization taken from QLoRA, and hand-chosen hyperparameters (learning rate, rank, batch size, epochs, fixed-point formats) without sensitivity studies. The most consequential premise is the fixed-batch replay assumption: in a streaming concept-drift setting the cache loses most of its value.

free parameters (4)
  • Learning rate eta = 0.1 (RotMNIST/RotFMNIST), 0.025 (SVHN)
    Hand-selected per dataset (Section 5.1); no sensitivity study is reported, so the accuracy comparison could shift with eta.
  • LoRA rank r = 4
    Chosen without a sweep (Section 4); adapter capacity depends on it, though the speedup claims are largely independent of r.
  • Epochs and batch size = 10 epochs, batch size 20
    Fixed protocol in Section 5.1; the forward-cache speedup grows with epoch count because the base forward pass is amortized over more passes.
  • Fixed-point and cache quantization formats = Q8.16 activations, Q4.12 parameters/gradients, NF4 cache
    Implementation choices (Sections 3.2 and 4.1); the reported accuracy costs are 0.09-0.4% for NF4 and up to 3.2% for the full FPGA path, with no format-alternative study.
assumptions (4)
  • domain assumption The pre-trained base network weights are frozen during fine-tuning.
    Standard PEFT premise (Section 3.1) that makes the forward cache valid; if the base network were updated, cached features would go stale.
  • domain assumption The fine-tuning dataset is fixed and fully available at runtime, with each input replayed over multiple epochs.
    Section 3.1: 'For a fixed dataset, the intermediate results for all samples are computed and cached in the first epoch.' The headline speedup depends on this; streaming one-pass data would not benefit from the cache.
  • standard math LoRA gradient equations (Eq. 3) and NF4 quantization (from QLoRA, ref [5]) are correct and applicable.
    The backward-pass equations are standard low-rank-adapter differentiation; NF4's suitability for normally distributed activations is asserted and checked only empirically.
  • domain assumption Fixed-point arithmetic (Q8.16/Q4.12) and lookup-table softmax preserve accuracy within the reported margins.
    Empirically the FPGA stays within 3.2% of the software version (Section 5.2); no formal error bound is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of InstantFT: An FPGA-Based Runtime Subsecond Fine-tuning of CNN Models." pith.science (2026). https://pith.science/paper/NBTYT5KM

@misc{pith2026250606505,
  author       = {Pith},
  title        = {Pith review of: InstantFT: An FPGA-Based Runtime Subsecond Fine-tuning of CNN Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NBTYT5KM}},
  note         = {Machine review of arXiv:2506.06505}
}
read the original abstract

Training deep neural networks (DNNs) requires significantly more computation and memory than inference, making runtime adaptation of DNNs challenging on resource-limited IoT platforms. We propose InstantFT, an FPGA-based method for ultra-fast CNN fine-tuning on IoT devices, by optimizing the forward and backward computations in parameter-efficient fine-tuning (PEFT). Experiments on datasets with concept drift demonstrate that InstantFT fine-tunes a pre-trained CNN 17.4x faster than existing Low-Rank Adaptation (LoRA)-based approaches, while achieving comparable accuracy. Our FPGA-based InstantFT reduces the fine-tuning time to just 0.36s and improves energy-efficiency by 16.3x, enabling on-the-fly adaptation of CNNs to non-stationary data distributions.

Figures

Figures reproduced from arXiv: 2506.06505 by the authors.

Figure 1
Figure 1. Fine-tuning time vs. accuracy of InstantFT and baselines (Rotated Fashion-MNIST, 75deg). [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Baseline fine-tuning methods and InstantFT. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Flow of fine-tuning with InstantFT. last layer using the activation x L, resulting in significantly reduced FLOPs and memory costs for backpropagation, at the cost of lower fine-tuning capability ( [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: InstantFT for LeNet-5-like model. Conv-MP Conv-MP FC FC FC Params ～ or Quant / Dequant LConv LConv LFC LFC LFC Params BPLConv BPLConv BPLFC BPLFC BPLFC ～ or Softmax CELoss AXI Interconnect S S S M 4GB 64-bit DDR4 Quad-core ARM Cortex-A53 (1.33GHz) HP0 HP1 HP2 HPM0 M M …
Figure 5
Figure 5. Figure 5: InstantFT implemented on Xilinx Kria KV260. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Block diagram of InstantFT core. 4.1 InstantFT on FPGA We design a dedicated InstantFT core for embedded FPGAs. The core ( [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Evaluation of InstantFT and baselines. Vitis 2024.1 to implement the core in C/C++ and generate an FPGA bitstream. The core along with two AXI IPs (an interrupt controller and interconnect) are clocked at 200MHz and implemented on PL (Programmable Logic). The core has …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages

  1. [1]

    TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning

    Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning. InProceedings of the International Conference on Neural Information Processing Systems (NeurIPS), pages 11285–11297, December 2020

    work page 2020

  2. [2]

    Addressing Gap between Training Data and Deployed Environment by On-Device Learning.IEEE Micro, 43(6):66–73, Nov/Dec 2023

    Kazuki Sunaga, Masaaki Kondo, and Hiroki Matsutani. Addressing Gap between Training Data and Deployed Environment by On-Device Learning.IEEE Micro, 43(6):66–73, Nov/Dec 2023

    work page 2023

  3. [3]

    Skip2-LoRA: A Lightweight On- device DNN Fine-tuning Method for Low-cost Edge Devices

    Hiroki Matsutani, Masaaki Kondo, Kazuki Sunaga, and Radu Marculescu. Skip2-LoRA: A Lightweight On- device DNN Fine-tuning Method for Low-cost Edge Devices. InProceedings of the Asia and South Pacific Design Automation Conference (ASP-DAC), pages 51–57, January 2025

    work page 2025

  4. [4]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. InInternational Conference on Learning Representations (ICLR), January 2022

    work page 2022

  5. [5]

    QLoRA: Efficient Finetuning of Quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. InProceedings of the International Conference on Neural Information Processing Systems (NeurIPS), pages 10088–10115, December 2023

    work page 2023

  6. [6]

    Saratchandran, and N

    Nan ying Liang, Guang bin Huang, P. Saratchandran, and N. Sundararajan. A Fast and Accurate Online Sequen- tial Learning Algorithm for Feedforward Networks.IEEE Transactions on Neural Networks, 17(6):1411–1423, November 2006

    work page 2006

  7. [7]

    Hero III, and Pramod K

    Sijia Liu, Pin-Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O. Hero III, and Pramod K. Varshney. A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning: Principals, Recent Advances, and Applications.IEEE Signal Processing Magazine, 37(5):43–54, September 2020

    work page 2020

  8. [8]

    A Neural Network-Based On-device Learning Anomaly Detector for Edge Devices.IEEE Transactions on Computers, 69(7):1027–1044, July 2020

    Mineto Tsukada, Masaaki Kondo, and Hiroki Matsutani. A Neural Network-Based On-device Learning Anomaly Detector for Edge Devices.IEEE Transactions on Computers, 69(7):1027–1044, July 2020

    work page 2020

Show all 12 references
  1. [9]

    Poor Man’s Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach

    Yequan Zhao, Hai Li, Ian Young, and Zheng Zhang. Poor Man’s Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach. arXiv preprint arXiv:2411.05873, November 2024

    2024 arXiv

  2. [10]

    Haoyu Ren, Darko Anicic, and Thomas A. Runkler. TinyOL: TinyML with Online-Learning on Microcontrollers. InProceedings of the International Joint Conference on Neural Networks (IJCNN), pages 1–8, July 2021

    2021

  3. [11]

    Gradient-Based Learning Applied to Document Recognition.Proceedings of the IEEE, 86(11):2278–2324, November 1998

    Yann Lecun, L’eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-Based Learning Applied to Document Recognition.Proceedings of the IEEE, 86(11):2278–2324, November 1998

    1998

  4. [12]

    Fast Inference of Deep Neural Networks in FPGAs for Particle Physics.Journal of Instrumentation, 13(7):P07027, July 2018

    Javier Duarte, Song Han, Philip Harris, Sergo Jindariani, Edward Kreinar, Benjamin Kreis, Jennifer Ngadiuba, Maurizio Pierini, Ryan Rivera, Nhan Tran, and Zhenbin Wu. Fast Inference of Deep Neural Networks in FPGAs for Particle Physics.Journal of Instrumentation, 13(7):P07027,...

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.