{"id":"11fde056-7c7c-42a4-b6a2-44ba650996a2","arxiv_id":"2506.20810","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FINN-GL adds ONNX Scan based LSTM support to the FINN compiler, enabling mixed-precision quantized LSTM layers to be synthesized into FPGA accelerators, demonstrated on a ConvLSTM for FI-2010 stock prediction.","lead":"This paper extends the FINN FPGA compiler framework to support mixed-precision LSTM networks by representing recurrence with the ONNX Scan operator and adding hardware mapping transformations. The authors validate the flow by deploying a quantized ConvLSTM for stock price trend prediction on a Xilinx ZCU104 FPGA, achieving 4.3 ms latency and accuracy close to floating-point models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Recurrent-state semantic equivalence of the threshold-based streamlining transformations is asserted but not demonstrated; generated hardware may not reproduce trained model states.","rationale":"The reader identified the equivalence between the QCDQ graph and the streamlined FINN-ONNX graph as the weakest assumption, and I agree: that is where the correctness of the central claim is least secure. My stress-test adds specificity about why the recurrent setting matters: algebraic simplification of affine operations past elementwise multiplications and state feedback is not automatically sound, and the paper gives no proof or exhaustive check. This does not establish that the claim is false; FINN-style thresholding and streamlining are well suited to feed-forward quantized networks and may extend soundly to LSTMs, but the burden is on demonstrating state-consistent equivalence across iterations. The paper's one-to-one mapping assertion is unsubstantiated by the presented experiments, which report aggregate F1 metrics rather than per-state numerical comparisons. The absence of code artifacts and error bars further weakens reproducibility, but the state-equivalence issue is the load-bearing concern. Since the reader already issued CONDITIONAL with moderate confidence, my analysis does not move the verdict; it reinforces the conditions under which the paper should be accepted.","tokens_in":13529,"tokens_out":5147,"duration_ms":59156,"concrete_test":"In the released qonnx/FINN-GLSTM-Hw flow, add and run a bit-exact differential check: for T=25 random input windows and several initial states, compute h_t and C_t at every timestep under (a) the original Brevitas QuantLSTM, (b) the QCDQ Scan ONNX graph, (c) the final streamlined FINN-ONNX graph, and (d) HLS/cosimulation output. Require exact match for W8A6 and for at least one mixed-precision setting, reporting the maximum absolute difference per timestep. If any non-negligible mismatch appears before the final output, the one-to-one mapping claim fails; if all match, the equivalence concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on semantic equivalence between the trained Brevitas QuantLSTM QCDQ graph and the streamlined FINN-ONNX graph produced by the transformations in Section III.B and Table I. In a recurrent graph, affine scale/bias operations interact with elementwise multiplications and state feedback: C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t and h_t = o_t ⊙ tanh(C_t). Moving a scalar/affine operation past an elementwise multiplication is not unconditionally valid: for example, s·(A⊙B) = (s·A)⊙B only if the scaling is applied to one operand, and when the two operands carry different scales, the state scales and threshold values must be reconciled consistently at every iteration. The paper states that the graph is 'functionally verified' and asserts a 'one-to-one mapping' in Section IV.C.1, but it provides neither a proof nor numerical evidence that the transformations preserve h_t and C_t across all timesteps and all mixed-precision assignments. RoundAndClipThresholds can also alter equality when an intermediate value lies near a threshold boundary, and in a feedback loop such errors propagate and accumulate rather than remaining independent per timestep. Thus the strongest claim—that the generated hardware reproduces the trained model—rests on the least documented part of the paper. This is a correctness risk, not merely a missing code artifact: if the equivalence fails for some mixed-precision configuration, the generated IP would silently implement a different model.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents FINN-GL, an extension of the FINN toolchain for deploying quantized LSTM networks on FPGAs. The authors model a quantized LSTM using the ONNX Scan operator, convert the Brevitas QCDQ graph to QONNX and then to FINN-ONNX, and introduce or modify streamlining transformations that absorb floating-point scale and bias operations into thresholding operators. A custom HLS backend maps the streamlined graph to finn-hlslib blocks. They validate the flow with a W8A6 quantized ConvLSTM for the FI-2010 limit-order-book dataset, targeting a ZCU104/XCZU7EV device, reporting 4.3 ms batch-1 latency, resource utilization of about 49% LUTs, 13% FFs, and 15% DSPs, and macro F1 of 77.40% versus 77.88% for floating-point DeepLOB. The central claim is that the extended FINN flow provides a generalizable, mixed-precision FPGA deployment path for LSTMs that preserves the trained model's behavior.","tokens_in":13845,"tokens_out":4988,"duration_ms":55798,"significance":"If the central claim is established, this is a useful contribution to FPGA-based ML deployment. The work leverages an established open-source ecosystem (FINN, Brevitas, QONNX) and extends it at a sensible abstraction level: the Scan-based representation exposes all internal quantizers, and the compiler transformations map recurrent computations onto reusable HLS blocks. The case study provides concrete latency, resource, and accuracy numbers, and the authors state that their implementations are open sourced. The main gaps are in verification: the semantic equivalence of the recurrent streamlining transformations and the equivalence between the trained model and the generated hardware are asserted rather than demonstrated, and the accuracy comparison rests on single-run numbers without statistical error bars.","major_comments":[{"comment":"The streamlining transformations that absorb floating-point scale and bias into MultiThreshold nodes are not shown to be semantics-preserving in a recurrent graph. In the LSTM equations (1)-(6), affine operations interact with elementwise multiplications (e.g., C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t and h_t = o_t ⊙ tanh(C_t)), so moving scalar or affine factors past elementwise multiplications and into thresholds requires careful reconciliation of state scales at every timestep. Moreover, RoundAndClipThresholds can change outputs for inputs near threshold boundaries, and in a feedback loop such perturbations can accumulate. The paper asserts that the graph is \"functionally verified\" (Section III.B) and claims a \"one-to-one mapping\" (Section IV.C.1), but it provides neither a proof nor numerical evidence. I ask for a concrete equivalence test: compare hidden states, cell states, and outputs of the original QCDQ graph with the streamlined FINN-ONNX graph over all timesteps for each supported mixed-precision configuration, reporting bit-exact or bounded-error results and a discussion of threshold-boundary cases.","section":"III.B, Eqs. (7)-(8), Table I"},{"comment":"The accuracy comparison is not statistically grounded. All reported metrics appear to come from a single training and evaluation run; there are no error bars, confidence intervals, or multiple seeds. The macro F1 difference between Q-ConvLSTM (77.40%) and DeepLOB (77.88%) is small and may be within run-to-run variation, yet Section IV.C.2 claims \"matching (or bettering)\" and quantifies gains over baselines as percentages. Please report mean and standard deviation over at least several seeds for the floating-point and quantized models, and state the significance test used. In addition, the statements \"accuracy drop of ≈1.5%\" and \"1% higher accuracy\" in Section IV.B.2 need a defined metric and repeatability.","section":"IV.B.2, Tables III-IV"},{"comment":"The hardware deployment section does not report any direct comparison between the outputs of the generated IP and the trained software model. The \"one-to-one mapping\" is the load-bearing assertion of the paper, but the only evidence cited is agreement between HLS cosimulation latency and board-level latency. Please include a measured accuracy or output-comparison experiment on the FPGA, for example running the test set through the IP and reporting the resulting F1, or comparing IP outputs to software outputs sample by sample, and state the data widths and rounding modes used in the HLS kernels.","section":"IV.C.1"}],"minor_comments":[{"comment":"The transformation name \"AbsorbsignbiasintoMultithrehsold()\" is misspelled; it should be \"AbsorbSignBiasIntoMultiThreshold()\".","section":"Table I"},{"comment":"The phrase \"the result shown in eq8\" should be \"the result shown in Eq. (8)\".","section":"III.B"},{"comment":"The text says the z-score normalized features were quantized to INT8, while the model is later described as W8A6; clarify whether input quantization is separate from the activation bit-widths and report the input precision used in the hardware implementation.","section":"IV.B.2"},{"comment":"The claim that the quantized model \"matches (or betters)\" state-of-the-art models is stronger than Table IV supports, since the macro F1 of Q-ConvLSTM is slightly below DeepLOB and C(TABL); please qualify the claim.","section":"Abstract and IV.C.2"},{"comment":"The paper refers to open-source repositories (qonnx and FINN-GLSTM-Hw) but provides no URLs or version hashes; please include them for reproducibility.","section":"III.A and IV.C.1"},{"comment":"The mapping \"Tanh + Quant -> Multithreshold + Mul + Add\" is not fully explained; please clarify how sigmoid and tanh thresholds are generated and what the Mul and Add nodes represent after the transformation.","section":"Eq. (8)"},{"comment":"Reference [25] is listed as \"A. Brevitas\"; please provide the actual author or tutorial title for the QuantLSTM documentation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a systems and tools contribution; the verification gap around recurrent-streamlining equivalence is the key correctness risk. If the authors can supply the numerical equivalence test and a hardware-software output comparison, the contribution would be solid. The statistical weakness in the accuracy evaluation is fixable but requires additional training runs. Fit with the journal scope appears acceptable if the tool-flow claims are strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a genuine engineering contribution to the FPGA ML tooling space, not a science breakthrough. It does three solid things: models quantized LSTM layers in ONNX via Scan, adds compiler transformations to streamline the quantized recurrent graph into integer-only FINN-ONNX, and validates the flow on a quantized ConvLSTM for FI-2010 with 4.3 ms latency and macro F1 77.40% versus DeepLOB's 77.88%. The comparison against hls4ml is fair, and the resource numbers are plausible. Credit where due: the Scan-based representation gives fine-grained control over the 11 internal quantizers, which hls4ml doesn't offer, and the authors open-sourced parts in qonnx and FINN-GLSTM-Hw.\n\nThe soft spots are real but mostly fixable. The biggest one is the semantic equivalence of the streamlining transformations inside the recurrent loop. Section III.B asserts the QCDQ-to-QONNX-to-FINN-ONNX conversion preserves function, and Section IV.C.1 claims a one-to-one mapping between the trained Brevitas model and the hardware, but the paper shows no bit-exact comparison, no numerical evidence across timesteps, and no analysis of threshold-rounding in the feedback path. The stress-test concern about state scaling interacting with elementwise multiplications is legitimate; the transformations may be correct, but the paper doesn't demonstrate it. That's a correctness risk, not just a missing code artifact. Also missing: error bars on the accuracy numbers (the 1% improvement over floating point from fine-tuning is probably noise), and no direct link to the claimed open-source repos. The 'generalized' in the title overstates things: one LSTM case study, not a full family of RNNs.\n\nThe citation pattern is fine. FINN, Brevitas, and qonnx are the authors' own frameworks, but they're open-source with external users, and the LSTM representation is benchmarked against an external dataset. No red flag there.\n\nWho is this for? Anyone working on FPGA acceleration of recurrent models, especially in the FINN ecosystem, or considering hls4ml versus FINN for LSTM deployment. The paper deserves a serious referee: the core idea is sound, the case study is concrete, and the gaps are addressable in revision. I'd send it to review, requesting the equivalence evidence and code as conditions.","headline":"Useful FINN extension for mixed-precision LSTM deployment, with a credible case study, but the recurrent-state equivalence claim needs harder evidence before I'd rely on the generated hardware.","tokens_in":14366,"tokens_out":2129,"would_cite":true,"duration_ms":22576,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The FINN framework, extended with ONNX Scan-based LSTM support and new compiler transformations, compiles a mixed-precision quantized ConvLSTM into an FPGA IP that runs at 4.3 ms with 77.40% macro F1 on FI-2010, within 0.5 points of…","keywords":["Brevitas","Field Programmable Gate Arrays","FINN","HFT","ONNX","Quantised LSTMs","RNNs","mixed-precision quantization"],"falsifier":"Take a trained QLSTM of any supported mixed-precision configuration, compile it through the flow, run the generated IP on a fixed input sequence, and compare the IP's output at every timestep with the Brevitas model's outputs; a single mismatch falsifies the claimed functional equivalence. A more targeted check is to compile an extreme configuration such as INT2 activations and observe whether recurrent-state errors accumulate beyond the rounding tolerance of RoundAndClipThresholds.","tokens_in":13378,"feed_emoji":"📈","tokens_out":11398,"duration_ms":113776,"temperature":0.7,"pith_summary":"This paper claims that the FINN framework, historically limited to feed-forward quantized networks, can be generalized to deploy recurrent LSTMs on FPGAs without hand-built per-model hardware. The authors represent the LSTM's hidden-state loop with ONNX's Scan operator, expose all eleven internal quantizers so mixed precision becomes possible, and add compiler transformations that absorb floating-point scale and bias into threshold-based activations, producing an integer-only computation graph. They validate the flow by training a W8A6 quantized ConvLSTM on the FI-2010 limit-order-book dataset and compiling it to an XCZU7EV FPGA IP. The resulting accelerator runs a batch in 4.3 ms and achieves a macro F1 of 77.40%, nearly matching the 77.88% of the floating-point DeepLOB baseline while using under half the LUTs and 15% of DSPs. If correct, this offers a reusable, resource-efficient path for RNN deployment on edge FPGAs rather than one-off custom implementations.","feed_headline":"Quantized ConvLSTM: 4.3 ms latency, 77.4% F1 on FPGA","feed_subtitle":"A general FINN-based flow deploys recurrent layers without bespoke hardware, holding DeepLOB-level accuracy.","key_machinery":"The load-bearing mechanism is FINN's multithreshold operator, which approximates monotonically increasing activations such as $\\tanh$ and $\\sigma$ by comparing inputs against a set of thresholds and returning the count of thresholds exceeded. The flow converts the QCDQ graph ($\\text{QuantizeLinear}+\\text{Clip}+\\text{DequantizeLinear}$) into a QONNX $\\text{Quant}$ node and then into FINN-ONNX as $\\text{Multithreshold}+\\text{Mul}+\\text{Add}$. New or adapted transformations (MoveScalarMulPastMatMul, MoveLinearPastEltwiseMul, AbsorbMulIntoMultithreshold, AbsorbSignBiasIntoMultithreshold, RoundAndClipThresholds) move scalar floating-point scale and bias factors across the graph until they can be absorbed into threshold values, eliminating floating-point arithmetic entirely. The ONNX Scan operator—a container that executes a body graph iteratively while updating state variables—encapsulates the per-timestep recurrence, so the streamlined body is compiled once into HLS blocks and run in a standard for-loop bounded by the sequence length.","core_discovery":"The central claim is that a quantized LSTM's recurrent computation can be expressed as an ONNX Scan body and then pushed through the QCDQ-to-QONNX-to-FINN-ONNX conversion chain into threshold-based integer-only hardware blocks, with the trained model's behavior preserved. The paper demonstrates this on a ConvLSTM for mid-price stock prediction: after quantizing to W8A6, the model compiles to a 150 MHz IP that reproduces the Brevitas-trained outputs, processes batch-1 in 4.3 ms, and scores macro F1 77.40% on FI-2010, comparable to floating-point DeepLOB at 77.88% and above several published baselines. The mixed-precision capability comes from controlling the bit-widths of all eleven internal quantizers of the LSTM layer, which the standard ONNX LSTM node does not expose.","pith_inferences":["The paper validates the flow on a single W8A6 quantized configuration; whether the semantic equivalence of the threshold approximations holds across the full mixed-precision space remains an open empirical question, since error accumulation over long sequences could vary with bit-width choices.","The same Scan-based ONNX representation should extend naturally to other recurrent cells such as GRUs, because the Scan body is just an ONNX subgraph; the paper only exercises the LSTM case.","The reported 4.3 ms latency excludes data-movement overhead from the AXI-DMA subsystem; end-to-end latency will be higher, though the 192 ms event interval suggests the margin remains comfortable.","The accuracy comparison, while close, was obtained on a model that omits the inception blocks of DeepLOB; the paper's own experiments attribute no significant loss to this omission, but the claim of being 'comparable to DeepLOB' should be read against that architectural difference."],"forward_implications":["LSTM layers can be dropped into the standard FINN pipeline alongside convolutional and dense layers, so hybrid models such as ConvLSTMs compile to hardware automatically.","All eleven internal quantizers of an LSTM layer become individually selectable, enabling systematic mixed-precision exploration of accuracy against hardware cost.","At 4.3 ms per batch, the compiled IP processes FI-2010 data far faster than the roughly 192 ms average interval between market events, leaving a large real-time margin.","The integer-only streamlined graph keeps resource use modest on an edge FPGA (49.6% LUTs, 12.8% FFs, 15.2% DSPs), leaving headroom to unroll or replicate LSTM cells for further latency reduction.","Because the final graph contains no floating-point operators, the approach avoids the resource overhead of flows that retain floating-point blocks, at the cost of the threshold-approximation error."],"supporting_citations":[{"why":"Supplies the DeepLOB ConvLSTM architecture and the FI-2010 training pipeline that the case study replicates as its accuracy baseline.","marker":"[1]"},{"why":"Defines the FINN quantized-NN framework and HLS kernel library that this work extends to recurrent networks.","marker":"[2]"},{"why":"FINN-L, the earlier custom LSTM extension for FINN, motivates the need for a generalized integrated flow.","marker":"[6]"},{"why":"Provides the streamlining technique of moving floating-point operations into thresholding, which the new transformations adapt for LSTM graphs.","marker":"[8]"},{"why":"Describes the hls4ml LSTM flow, the main comparative approach, highlighting the trade-off between resource efficiency and latency.","marker":"[10]"},{"why":"Gives the FI-2010 limit-order-book dataset used for training and evaluation.","marker":"[19]"},{"why":"Documents the ONNX Scan operator that models the recurrent hidden-state loop.","marker":"[23]"},{"why":"Identifies Brevitas, the training framework whose QuantLSTM QCDQ graph the ONNX representation must match.","marker":"[24]"},{"why":"Specifies the eleven internal quantizers of QuantLSTM, the granularity that mixed-precision support depends on.","marker":"[25]"},{"why":"Introduces the QONNX format with Quant nodes that enables the QCDQ-to-FINN-ONNX conversion chain.","marker":"[26]"}],"fun_headline_variants":["General FPGA LSTM: 4.3 ms, 77.4 F1 via FINN","Mixed-precision LSTM on FPGA: 4.3 ms, 77.4 F1","No bespoke RTL: FINN deploys quantized ConvLSTM","ONNX Scan + FINN: FPGA LSTM at 4.3 ms, 77% F1","LSTM on FPGA: W8A6, 150 MHz, 4.3 ms, 77.4 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything hinges on the claim that replacing sigmoid and tanh with threshold-based approximations, and folding floating-point scale and bias into the thresholds, is exactly semantically equivalent to the original Brevitas QCDQ computation at every recurrent step; if any mixed-precision configuration breaks that equivalence, the FPGA would no longer reproduce the trained model's outputs.","fun_headline_variants_meta":{"raw":{"variants":["General FPGA LSTM: 4.3 ms, 77.4 F1 via FINN","Mixed-precision LSTM on FPGA: 4.3 ms, 77.4 F1","No bespoke RTL: FINN deploys quantized ConvLSTM","ONNX Scan + FINN: FPGA LSTM at 4.3 ms, 77% F1","LSTM on FPGA: W8A6, 150 MHz, 4.3 ms, 77.4 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000259,"raw_usage":{"total_tokens":1626,"prompt_tokens":1025,"completion_tokens":601,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":472}},"tokens_in":641,"tokens_out":601,"duration_ms":6980,"temperature":1.0,"reasoning_tokens":472,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:40:57.606063+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained QLSTM of any supported mixed-precision configuration, compile it through the flow, run the generated IP on a fixed input sequence, and compare the IP's output at every timestep with the Brevitas model's outputs; a single mismatch falsifies the claimed functional equivalence. A more targeted check is to compile an extreme configuration such as INT2 activations and observe whether recurrent-state errors accumulate beyond the rounding tolerance of RoundAndClipThresholds.","supporting_citations":[{"cited_title":"Deeplob: Deep convolutional neural networks for limit order books,","cited_arxiv_id":null,"evidence_quote":"Supplies the DeepLOB ConvLSTM architecture and the FI-2010 training pipeline that the case study replicates as its accuracy baseline."},{"cited_title":"FINN-L: Library extensions and design trade-off analysis for variable precision LSTM networks on FPGAs,","cited_arxiv_id":null,"evidence_quote":"FINN-L, the earlier custom LSTM extension for FINN, motivates the need for a generalized integrated flow."},{"cited_title":"Streamlined Deployment for Quantized Neural Networks","cited_arxiv_id":"1709.04060","evidence_quote":"Provides the streamlining technique of moving floating-point operations into thresholding, which the new transformations adapt for LSTM graphs."},{"cited_title":"Ultra-low latency recurrent neural network inference on FPGAs for physics applications with hls4ml,","cited_arxiv_id":null,"evidence_quote":"Describes the hls4ml LSTM flow, the main comparative approach, highlighting the trade-off between resource efficiency and latency."},{"cited_title":"Benchmark dataset for mid-price forecasting of limit order book data with machine learning methods,","cited_arxiv_id":null,"evidence_quote":"Gives the FI-2010 limit-order-book dataset used for training and evaluation."},{"cited_title":"https://github.com/onnx/onnx/blob/main/docs/operators.md#scan,","cited_arxiv_id":null,"evidence_quote":"Documents the ONNX Scan operator that models the recurrent hidden-state loop."},{"cited_title":"Xilinx/brevitas,","cited_arxiv_id":null,"evidence_quote":"Identifies Brevitas, the training framework whose QuantLSTM QCDQ graph the ONNX representation must match."},{"cited_title":"Quantlstm - brevitas tutorials,","cited_arxiv_id":null,"evidence_quote":"Specifies the eleven internal quantizers of QuantLSTM, the granularity that mixed-precision support depends on."}],"review_version":1}