REVIEW 3 major objections 7 minor 27 references
FINN-GL: Generalized Mixed-Precision Extensions for FPGA-Accelerated LSTMs
T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The FINN framework, extended with ONNX Scan-based LSTM support and new compiler transformations, compiles a mixed-precision quantized ConvLSTM into an FPGA IP that runs at 4.3 ms with 77.40% macro F1 on FI-2010, within 0.5 points of…
desk verdict Useful FINN extension for mixed-precision LSTM deployment, with a credible case study, but the recurrent-state equivalence claim needs harder evidence before I'd rely on the generated hardware. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is FINN's multithreshold operator, which approximates monotonically increasing activations such as $\tanh$ and $\sigma$ by comparing inputs against a set of thresholds and returning the count of thresholds exceeded. The flow converts the QCDQ graph ($\text{QuantizeLinear}+\text{Clip}+\text{DequantizeLinear}$) into a QONNX $\text{Quant}$ node and then into FINN-ONNX as $\text{Multithreshold}+\text{Mul}+\text{Add}$. New or adapted transformations (MoveScalarMulPastMatMul, MoveLinearPastEltwiseMul, AbsorbMulIntoMultithreshold, AbsorbSignBiasIntoMultithreshold, RoundAndClipThresholds) move scalar floating-point scale and bias factors across the graph until they can be absorbed into threshold values, eliminating floating-point arithmetic entirely. The ONNX Scan operator—a container that executes a body graph iteratively while updating state variables—encapsulates the per-timestep recurrence, so the streamlined body is compiled once into HLS blocks and run in a standard for-loop bounded by the sequence length.
What would settle it
Take a trained QLSTM of any supported mixed-precision configuration, compile it through the flow, run the generated IP on a fixed input sequence, and compare the IP's output at every timestep with the Brevitas model's outputs; a single mismatch falsifies the claimed functional equivalence. A more targeted check is to compile an extreme configuration such as INT2 activations and observe whether recurrent-state errors accumulate beyond the rounding tolerance of RoundAndClipThresholds.
Extended reading notes
Core claim
The central claim is that a quantized LSTM's recurrent computation can be expressed as an ONNX Scan body and then pushed through the QCDQ-to-QONNX-to-FINN-ONNX conversion chain into threshold-based integer-only hardware blocks, with the trained model's behavior preserved. The paper demonstrates this on a ConvLSTM for mid-price stock prediction: after quantizing to W8A6, the model compiles to a 150 MHz IP that reproduces the Brevitas-trained outputs, processes batch-1 in 4.3 ms, and scores macro F1 77.40% on FI-2010, comparable to floating-point DeepLOB at 77.88% and above several published baselines. The mixed-precision capability comes from controlling the bit-widths of all eleven internal quantizers of the LSTM layer, which the standard ONNX LSTM node does not expose.
Load-bearing premise
Everything hinges on the claim that replacing sigmoid and tanh with threshold-based approximations, and folding floating-point scale and bias into the thresholds, is exactly semantically equivalent to the original Brevitas QCDQ computation at every recurrent step; if any mixed-precision configuration breaks that equivalence, the FPGA would no longer reproduce the trained model's outputs.
Editorial extensions
If this is right
- LSTM layers can be dropped into the standard FINN pipeline alongside convolutional and dense layers, so hybrid models such as ConvLSTMs compile to hardware automatically.
- All eleven internal quantizers of an LSTM layer become individually selectable, enabling systematic mixed-precision exploration of accuracy against hardware cost.
- At 4.3 ms per batch, the compiled IP processes FI-2010 data far faster than the roughly 192 ms average interval between market events, leaving a large real-time margin.
- The integer-only streamlined graph keeps resource use modest on an edge FPGA (49.6% LUTs, 12.8% FFs, 15.2% DSPs), leaving headroom to unroll or replicate LSTM cells for further latency reduction.
- Because the final graph contains no floating-point operators, the approach avoids the resource overhead of flows that retain floating-point blocks, at the cost of the threshold-approximation error.
Reading between the lines
- The paper validates the flow on a single W8A6 quantized configuration; whether the semantic equivalence of the threshold approximations holds across the full mixed-precision space remains an open empirical question, since error accumulation over long sequences could vary with bit-width choices.
- The same Scan-based ONNX representation should extend naturally to other recurrent cells such as GRUs, because the Scan body is just an ONNX subgraph; the paper only exercises the LSTM case.
- The reported 4.3 ms latency excludes data-movement overhead from the AXI-DMA subsystem; end-to-end latency will be higher, though the 192 ms event interval suggests the margin remains comfortable.
- The accuracy comparison, while close, was obtained on a model that omits the inception blocks of DeepLOB; the paper's own experiments attribute no significant loss to this omission, but the claim of being 'comparable to DeepLOB' should be read against that architectural difference.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents FINN-GL, an extension of the FINN toolchain for deploying quantized LSTM networks on FPGAs. The authors model a quantized LSTM using the ONNX Scan operator, convert the Brevitas QCDQ graph to QONNX and then to FINN-ONNX, and introduce or modify streamlining transformations that absorb floating-point scale and bias operations into thresholding operators. A custom HLS backend maps the streamlined graph to finn-hlslib blocks. They validate the flow with a W8A6 quantized ConvLSTM for the FI-2010 limit-order-book dataset, targeting a ZCU104/XCZU7EV device, reporting 4.3 ms batch-1 latency, resource utilization of about 49% LUTs, 13% FFs, and 15% DSPs, and macro F1 of 77.40% versus 77.88% for floating-point DeepLOB. The central claim is that the extended FINN flow provides a generalizable, mixed-precision FPGA deployment path for LSTMs that preserves the trained model's behavior.
Significance. If the central claim is established, this is a useful contribution to FPGA-based ML deployment. The work leverages an established open-source ecosystem (FINN, Brevitas, QONNX) and extends it at a sensible abstraction level: the Scan-based representation exposes all internal quantizers, and the compiler transformations map recurrent computations onto reusable HLS blocks. The case study provides concrete latency, resource, and accuracy numbers, and the authors state that their implementations are open sourced. The main gaps are in verification: the semantic equivalence of the recurrent streamlining transformations and the equivalence between the trained model and the generated hardware are asserted rather than demonstrated, and the accuracy comparison rests on single-run numbers without statistical error bars.
major comments (3)
- [III.B, Eqs. (7)-(8), Table I] The streamlining transformations that absorb floating-point scale and bias into MultiThreshold nodes are not shown to be semantics-preserving in a recurrent graph. In the LSTM equations (1)-(6), affine operations interact with elementwise multiplications (e.g., C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t and h_t = o_t ⊙ tanh(C_t)), so moving scalar or affine factors past elementwise multiplications and into thresholds requires careful reconciliation of state scales at every timestep. Moreover, RoundAndClipThresholds can change outputs for inputs near threshold boundaries, and in a feedback loop such perturbations can accumulate. The paper asserts that the graph is "functionally verified" (Section III.B) and claims a "one-to-one mapping" (Section IV.C.1), but it provides neither a proof nor numerical evidence. I ask for a concrete equivalence test: compare hidden states, cell states, and outputs of the original QCDQ graph with the streamlined FINN-ONNX graph over all timesteps for each supported mixed-precision configuration, reporting bit-exact or bounded-error results and a discussion of threshold-boundary cases.
- [IV.B.2, Tables III-IV] The accuracy comparison is not statistically grounded. All reported metrics appear to come from a single training and evaluation run; there are no error bars, confidence intervals, or multiple seeds. The macro F1 difference between Q-ConvLSTM (77.40%) and DeepLOB (77.88%) is small and may be within run-to-run variation, yet Section IV.C.2 claims "matching (or bettering)" and quantifies gains over baselines as percentages. Please report mean and standard deviation over at least several seeds for the floating-point and quantized models, and state the significance test used. In addition, the statements "accuracy drop of ≈1.5%" and "1% higher accuracy" in Section IV.B.2 need a defined metric and repeatability.
- [IV.C.1] The hardware deployment section does not report any direct comparison between the outputs of the generated IP and the trained software model. The "one-to-one mapping" is the load-bearing assertion of the paper, but the only evidence cited is agreement between HLS cosimulation latency and board-level latency. Please include a measured accuracy or output-comparison experiment on the FPGA, for example running the test set through the IP and reporting the resulting F1, or comparing IP outputs to software outputs sample by sample, and state the data widths and rounding modes used in the HLS kernels.
minor comments (7)
- [Table I] The transformation name "AbsorbsignbiasintoMultithrehsold()" is misspelled; it should be "AbsorbSignBiasIntoMultiThreshold()".
- [III.B] The phrase "the result shown in eq8" should be "the result shown in Eq. (8)".
- [IV.B.2] The text says the z-score normalized features were quantized to INT8, while the model is later described as W8A6; clarify whether input quantization is separate from the activation bit-widths and report the input precision used in the hardware implementation.
- [Abstract and IV.C.2] The claim that the quantized model "matches (or betters)" state-of-the-art models is stronger than Table IV supports, since the macro F1 of Q-ConvLSTM is slightly below DeepLOB and C(TABL); please qualify the claim.
- [III.A and IV.C.1] The paper refers to open-source repositories (qonnx and FINN-GLSTM-Hw) but provides no URLs or version hashes; please include them for reproducibility.
- [Eq. (8)] The mapping "Tanh + Quant -> Multithreshold + Mul + Add" is not fully explained; please clarify how sigmoid and tanh thresholds are generated and what the Mul and Add nodes represent after the transformation.
- [References] Reference [25] is listed as "A. Brevitas"; please provide the actual author or tutorial title for the QuantLSTM documentation.
Circularity Check
No significant circularity: the paper's claims rest on external benchmarks (FI-2010, XCZU7EV latency) and on functional verification against Brevitas QuantLSTM; the recurrent-streamlining gap is an unsupported-equivalence risk, not a circular reduction.
full rationale
The central results are measured, not derived from fitted inputs: macro F1 77.40% is evaluated on the held-out two days of FI-2010 and compared against independently reproduced DeepLOB (77.88%); batch-1 latency 4.3 ms is measured from the synthesized IP on XCZU7EV. No accuracy or latency number is obtained by fitting a parameter and then 'predicting' a related quantity. The QCDQ-to-QONNX-to-FINN-ONNX conversions in Eqs. (7)-(8) are compiler rewrite rules defined by the QONNX/FINN operator semantics; they are not empirical predictions, and the paper does not present them as derived from the target numbers. The paper cites FINN, Brevitas, and QONNX, all with overlapping author groups, but these are open-source toolchains with external user bases and the relevant comparison (against Brevitas QuantLSTM) is a functional check against an independent reference implementation of the same layer, not a self-referential derivation. The skeptical concern that threshold-based streamlining may not preserve recurrent state semantics across iterations is a correctness/verification gap: the paper asserts 'functional verification' and 'one-to-one mapping' (Section IV.C.1) without publishing the equivalence proof or per-timestep numerical traces. That is an evidence deficiency, not a circular argument, because the hardware output is not defined to equal the streamlined graph by construction; it is an empirical claim that could fail. No circular step can be quoted, so the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- Activation bit-width (A6) =
6 bits
- Weight bit-width (W8) =
8 bits
- LSTM hidden units =
64
- Input sequence length =
25
assumptions (3)
- domain assumption Quantized sigmoid and tanh activations can be exactly represented by multithreshold operators with appropriate integer thresholds.
- domain assumption The ONNX Scan operator body graph exactly replicates the LSTM recurrence semantics of the Brevitas QuantLSTM layer.
- domain assumption The streamlining transformations, including the new ones described in Table I, preserve functional equivalence after absorbing floating-point operations into thresholds.
Cite this review
Pith. "Pith review of FINN-GL: Generalized Mixed-Precision Extensions for FPGA-Accelerated LSTMs." pith.science (2026). https://pith.science/paper/J5XZZZ2F
@misc{pith2026250620810,
author = {Pith},
title = {Pith review of: FINN-GL: Generalized Mixed-Precision Extensions for FPGA-Accelerated LSTMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/J5XZZZ2F}},
note = {Machine review of arXiv:2506.20810}
}
read the original abstract
Recurrent neural networks (RNNs), particularly LSTMs, are effective for time-series tasks like sentiment analysis and short-term stock prediction. However, their computational complexity poses challenges for real-time deployment in resource constrained environments. While FPGAs offer a promising platform for energy-efficient AI acceleration, existing tools mainly target feed-forward networks, and LSTM acceleration typically requires full custom implementation. In this paper, we address this gap by leveraging the open-source and extensible FINN framework to enable the generalized deployment of LSTMs on FPGAs. Specifically, we leverage the Scan operator from the Open Neural Network Exchange (ONNX) specification to model the recurrent nature of LSTM computations, enabling support for mixed quantisation within them and functional verification of LSTM-based models. Furthermore, we introduce custom transformations within the FINN compiler to map the quantised ONNX computation graph to hardware blocks from the HLS kernel library of the FINN compiler and Vitis HLS. We validate the proposed tool-flow by training a quantised ConvLSTM model for a mid-price stock prediction task using the widely used dataset and generating a corresponding hardware IP of the model using our flow, targeting the XCZU7EV device. We show that the generated quantised ConvLSTM accelerator through our flow achieves a balance between performance (latency) and resource consumption, while matching (or bettering) inference accuracy of state-of-the-art models with reduced precision. We believe that the generalisable nature of the proposed flow will pave the way for resource-efficient RNN accelerator designs on FPGAs.
Figures
Reference graph
Works this paper leans on
-
[1]
Deeplob: Deep convolutional neural networks for limit order books,
Z. Zhang, S. Zohren, and S. Roberts, “Deeplob: Deep convolutional neural networks for limit order books,”IEEE Transactions on Signal Processing, vol. 67, no. 11, pp. 3001–3012, 2019
work page 2019
-
[2]
FINN-R: An end-to-end deep- learning framework for fast exploration of quantized neural networks,
M. Blott, T. B. Preußer, N. J. Fraser, G. Gambardella, K. O’brien, Y . Umuroglu, M. Leeser, and K. Vissers, “FINN-R: An end-to-end deep- learning framework for fast exploration of quantized neural networks,” ACM Transactions on Reconfigurable Technology and Systems (TRETS), vol. 11, no. 3, pp. 1–23, 2018
2018
-
[3]
Fast inference of deep neural networks in FPGAs for particle physics,
J. Duarteet al., “Fast inference of deep neural networks in FPGAs for particle physics,”JINST, vol. 13, no. 07, p. P07027, 2018
2018
-
[4]
Near real-time wind speed forecast model with bidirectional LSTM networks,
L. P. Joseph, R. C. Deo, R. Prasad, S. Salcedo-Sanz, N. Raj, and J. Soar, “Near real-time wind speed forecast model with bidirectional LSTM networks,”Renewable Energy, vol. 204, pp. 39–58, 2023
work page 2023
- [5]
-
[6]
V . Rybalkin, A. Pappalardo, M. M. Ghaffar, G. Gambardella, N. Wehn, and M. Blott, “FINN-L: Library extensions and design trade-off analysis for variable precision LSTM networks on FPGAs,” in2018 28th international conference on field programmable logic and applications (FPL), pp. 89–897, IEEE, 2018
work page 2018
-
[7]
Elastic-df: Scaling perfor- mance of dnn inference in fpga clouds through automatic partitioning,
T. Alonso, L. Petrica, M. Ruiz, J. Petri-Koenig, Y . Umuroglu, I. Stame- los, E. Koromilas, M. Blott, and K. Vissers, “Elastic-df: Scaling perfor- mance of dnn inference in fpga clouds through automatic partitioning,” ACM Trans. Reconfigurable Technol. Syst., vol. 15, Dec. 2021
work page 2021
-
[8]
Streamlined Deployment for Quantized Neural Networks
Y . Umuroglu and M. Jahre, “Streamlined deployment for quantized neural networks,”arXiv preprint arXiv:1709.04060, 2017
work page Pith review arXiv 2017
Show all 27 references
-
[9]
The use of recurrent neural networks in continuous speech recognition,
T. Robinson, M. Hochberg, and S. Renals, “The use of recurrent neural networks in continuous speech recognition,” inAutomatic Speech and Speaker Recognition: Advanced Topics, pp. 233–258, Springer, 1996
1996
-
[10]
Ultra-low latency recurrent neural network inference on FPGAs for physics applications with hls4ml,
E. E. Khoda, D. Rankin, R. T. de Lima, P. Harris, S. Hauck, S.-C. Hsu, M. Kagan, V . Loncar, C. Paikara, R. Rao,et al., “Ultra-low latency recurrent neural network inference on FPGAs for physics applications with hls4ml,”Machine Learning: Science and Technology, vol. 4, no. 2,...
2023
-
[11]
Mapping large lstms to fpgas with weight reuse,
Z. Que, Y . Zhu, H. Fan, J. Meng, X. Niu, and W. Luk, “Mapping large lstms to fpgas with weight reuse,”Journal of Signal Processing Systems, vol. 92, pp. 965–979, 2020
2020
-
[12]
F-LSTM: FPGA-based heterogeneous computing framework for deploying LSTM-based algorithms,
Liang, Bushun and Wang, Siye and Huang, Yeqin and Liu, Yiling and Ma, Linpeng, “F-LSTM: FPGA-based heterogeneous computing framework for deploying LSTM-based algorithms,”Electronics, vol. 12, no. 5, p. 1139, 2023
2023
-
[13]
An Efficient Sparse LSTM Accelerator on Embedded FPGAs with Bandwidth-Oriented Pruning,
S. Li, S. Zhu, X. Luo, T. Luo, and W. Liu, “An Efficient Sparse LSTM Accelerator on Embedded FPGAs with Bandwidth-Oriented Pruning,” in2023 33rd International Conference on Field-Programmable Logic and Applications (FPL), pp. 42–48, 2023
2023
-
[14]
C-LSTM: Enabling efficient LSTM using structured compression tech- niques on FPGAs,
S. Wang, Z. Li, C. Ding, B. Yuan, Q. Qiu, Y . Wang, and Y . Liang, “C-LSTM: Enabling efficient LSTM using structured compression tech- niques on FPGAs,” inProceedings of the 2018 ACM/SIGDA Inter- national Symposium on Field-Programmable Gate Arrays, pp. 11–20, 2018
2018
-
[15]
Mapping multiple LSTM models on FPGAs,
S. Ribes, P. Trancoso, I. Sourdis, and C.-S. Bouganis, “Mapping multiple LSTM models on FPGAs,” in2020 International Conference on Field- Programmable Technology (ICFPT), pp. 1–9, IEEE, 2020
2020
-
[16]
Streaming Overlay Architecture for Lightweight LSTM Computation on FPGA SoCs,
L. Ioannou and S. A. Fahmy, “Streaming Overlay Architecture for Lightweight LSTM Computation on FPGA SoCs,”ACM Transactions on Reconfigurable Technology and Systems, vol. 16, no. 1, pp. 1–26, 2022
2022
-
[17]
https://www.investopedia.com/terms/l/limitorderbook.asp,
Investopedia, “https://www.investopedia.com/terms/l/limitorderbook.asp,” 2022
2022
-
[18]
Using deep learning to detect price change indications in financial markets,
Tsantekidis, Avraam and Passalis, Nikolaos and Tefas, Anastasios and Kanniainen, Juho and Gabbouj, Moncef and Iosifidis, Alexandros, “Using deep learning to detect price change indications in financial markets,” in25th European Signal Processing Conference (EUSIPCO), pp. 2511–...
2017
-
[19]
Benchmark dataset for mid-price forecasting of limit order book data with machine learning methods,
A. Ntakaris, M. Magris, J. Kanniainen, M. Gabbouj, and A. Iosifidis, “Benchmark dataset for mid-price forecasting of limit order book data with machine learning methods,”Journal of Forecasting, vol. 37, no. 8, pp. 852–866, 2018
2018
-
[20]
Temporal attention-augmented bilinear network for financial time-series data analysis,
Tran, Dat Thanh and Iosifidis, Alexandros and Kanniainen, Juho and Gabbouj, Moncef, “Temporal attention-augmented bilinear network for financial time-series data analysis,”IEEE transactions on neural net- works and learning systems, vol. 30, no. 5, pp. 1407–1418, 2018
2018
-
[21]
Forecasting stock prices from the limit order book using convolutional neural networks,
Tsantekidis, Avraam and Passalis, Nikolaos and Tefas, Anastasios and Kanniainen, Juho and Gabbouj, Moncef and Iosifidis, Alexandros, “Forecasting stock prices from the limit order book using convolutional neural networks,” in2017 IEEE 19th conference on business informatics (C...
2017
-
[22]
Using deep learning for price prediction by exploiting stationary limit order book features,
A. Tsantekidis, N. Passalis, A. Tefas, J. Kanniainen, M. Gabbouj, and A. Iosifidis, “Using deep learning for price prediction by exploiting stationary limit order book features,”Applied Soft Computing, vol. 93, p. 106401, 2020
2020
-
[23]
https://github.com/onnx/onnx/blob/main/docs/operators.md#scan,
ONNX, “https://github.com/onnx/onnx/blob/main/docs/operators.md#scan,” 2018
2018
-
[24]
Xilinx/brevitas,
A. Pappalardo, “Xilinx/brevitas,” 2021
2021
-
[25]
Quantlstm - brevitas tutorials,
A. Brevitas, “Quantlstm - brevitas tutorials,” 2024
2024
-
[26]
Qonnx: Repre- senting arbitrary-precision quantized neural networks,
A. Pappalardo, Y . Umuroglu, M. Blott, J. Mitrevski, B. Hawks, N. Tran, V . Loncar, S. Summers, H. Borras, J. Muhizi,et al., “Qonnx: Repre- senting arbitrary-precision quantized neural networks,”arXiv preprint arXiv:2206.07527, 2022
2022 arXiv
-
[27]
Temporal bag-of-features learning for predicting mid price movements using high frequency limit order book data,
N. Passalis, A. Tefas, J. Kanniainen, M. Gabbouj, and A. Iosifidis, “Temporal bag-of-features learning for predicting mid price movements using high frequency limit order book data,”IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 4, no. 6, pp. 774– 785, 2018
2018
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.