REVIEW 3 major objections 3 minor 28 references
Hardware/Software Co-Design of RISC-V Extensions for Accelerating Sparse DNNs on FPGAs
T0 review · 3 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Two custom RISC-V instructions that hide sparsity metadata inside INT8 weights and shorten MAC cycles can accelerate pruned DNNs by up to 5x on small FPGAs, without accuracy loss.
desk verdict The SSSA lookahead encoding is a genuine contribution, but the headline speedup numbers for USSA and CSA are measured against a sequential baseline that is not the paper's own SIMD baseline, so the abstract's 3x/4x/5x claims are not substantiated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanisms are two custom functional units attached to a RISC-V soft-core through the custom instruction encoding. First, the lookahead skip mechanism: a 4-bit counter (0-15) is precomputed per non-zero block of four INT8 weights, recording how many following blocks are entirely zero; the counter's bits are appended to the LSBs of the four weights (restoring the sign bit), reducing effective weight precision from 8 to 7 bits. One instruction extracts these bits and increments the loop variable by (counter+1)*4, skipping zero blocks in the innermost convolution loop without any runtime metadata lookup. Second, the variable-cycle sequential MAC: four zero-comparators produce a case signal; control logic and two mux stages align the non-zero weights with their inputs; a sequential MAC then consumes exactly one cycle per non-zero weight, with one cycle for an all-zero block. The combined design simply merges both units. Together they let instruction-level sparsity skipping and cycle-counting replace data-dependent branches and compressed formats.
What would settle it
Measure the cycle count of a single MobileNetV2 convolutional layer on the same FPGA with the SIMD MAC baseline and with the combined accelerator at the sparsity levels reported; if the combined design does not approach 5x over that baseline, the headline claim depends on the choice of the slower sequential MAC comparator.
Extended reading notes
Core claim
The paper's central claim is that sparsity metadata and sparse computation can be folded into the instruction stream of a general-purpose RISC-V core at negligible hardware cost. For semi-structured sparsity, a pre-processing pass computes how many consecutive all-zero blocks follow each non-zero block, encodes that 0-15 count into the four LSBs of the block's four INT8 weights, and a custom instruction reads those bits and bumps the loop induction variable accordingly, while a companion instruction performs the usual 4-wide multiply-accumulate on the 7-bit effective weights. For unstructured sparsity, a variable-cycle sequential MAC compares four weights against zero in parallel, aligns the non-zero weight-input pairs through multiplexers, and takes one cycle per non-zero weight (one cycle even for an all-zero block), instead of the baseline's fixed four cycles per block. The combined design merges both mechanisms, and the paper reports up to 5x speedup on MobileNetV2, ResNet-56, DSCNN, and person-detection benchmarks, with accuracy essentially unchanged. The claimed payoff is that with a couple of instructions and a few percent extra LUTs and flip-flops, small FPGAs can run sparse TinyML models several times faster.
Load-bearing premise
The speedup numbers are relative to a baseline that always spends four clock cycles on each block of four weights; if the intended baseline is instead the faster SIMD multiply-accumulate described earlier in the paper, the reported gains would shrink and the unstructured design might not beat it.
Editorial extensions
If this is right
- If the measured speedups hold, the FPGA resource cost of sparse-DNN acceleration drops to a few percent of logic, making it feasible on very small, low-cost FPGAs.
- The lookahead encoding removes the need for compressed sparse formats and runtime index decoding for semi-structured models; pruning methods only need to produce 4-weight all-zero blocks.
- Because the custom instructions fit in the standard RISC-V custom opcode space, software changes are localized to the innermost loop, and no compiler or toolchain modifications are required.
- The paper's analytical models predict speedups grow with sparsity, so more aggressive pruning than tested should yield higher gains.
- The paper notes the same mechanisms extend to INT4/INT2 packed weights, where one 32-bit register holds 8 or 16 weights and the cycle savings multiply.
Reading between the lines
- Beyond the paper: the headline speedups are relative to a baseline sequential MAC that always takes four cycles per four-weight block; measured against the SIMD MAC baseline introduced in the paper's setup, the gains would be smaller, and the unstructured design could be slower when blocks are dense. The paper does not report this comparison.
- Beyond the paper: the one reserved bit per weight could alternatively carry longer skip counts or other metadata instead of a 4-bit counter; the accuracy headroom shown for 7-bit weights suggests such tradeoffs are worth testing, though quantization-sensitive models may not tolerate them.
- Beyond the paper: because each custom instruction passes only two 32-bit registers, the block size is capped at four (or eight with INT4) weights; scaling to wider SIMD or tensor operations would require a coprocessor interface, which the authors list as future work.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes RISC-V custom functional units for accelerating sparse DNN inference on a VexRiscv soft-core FPGA system. It describes three designs: a semi-structured sparsity accelerator (SSSA) that encodes lookahead skip counters in the low bits of four INT8 weights, an unstructured sparsity accelerator (USSA) based on a variable-cycle sequential multiply-accumulate unit, and a combined design (CSA) that integrates both mechanisms. The manuscript reports speedups of up to 3x, 4x, and 5x for USSA, SSSA, and CSA, respectively, with modest FPGA resource overheads, and presents INT8-vs-INT7 accuracy comparisons for three TinyML models. The authors state explicitly in Section IV-C that no pruned models were trained or accuracy-tested, and the performance comparisons are based on a baseline that differs from the paper's own SIMD MAC baseline.
Significance. The core hardware ideas are plausible and potentially useful: exploiting bit-level FPGA reconfigurability to encode sparse-block lookahead, and using a variable-cycle MAC to skip zero weights, require only small extra resources according to Table III. The analytical USSA cycle model in Section IV-D is derived cleanly from an explicit IID Bernoulli assumption with no fitted constants, and it yields falsifiable speedup-versus-sparsity predictions. The INT8-to-INT7 accuracy comparison in Table II is a useful sanity check. However, the headline speedup claims are currently not substantiated because they are measured against a four-cycle sequential MAC baseline rather than the cfu_simd_mac baseline defined in Section III-A, and the accuracy claims are not supported by any pruned-model evaluation. If the authors re-benchmark against the actual SIMD baseline and provide pruned-model accuracy results, the work could be a solid contribution to low-cost sparse DNN acceleration on small FPGAs.
major comments (3)
- [§III-A vs §III-C1, Figs. 8–10] The speedup experiments are grounded in a baseline that is not the one introduced as the starting point of the paper. Section III-A defines the CFU Playground SIMD MAC instruction (cfu_simd_mac) as the baseline for the convolutional kernel (Listing 1), but Section III-C1 introduces a different "baseline single sequential MAC unit" that always takes four cycles per four-weight block, and Figures 8 and 10 report speedups relative to that four-cycle unit. The reported "up to 3x/4x/5x" factors therefore do not establish acceleration over cfu_simd_mac; if cfu_simd_mac retires a four-weight block in one cycle, the variable-cycle USSA/CSA units would be slower than the actual baseline at all non-zero densities. The authors should provide direct cycle-count comparisons against cfu_simd_mac using identical weight blocks and loop structure, and restate the speedup claims accordingly.
- [§IV-C, §IV-G] The paper does not evaluate the accuracy of any pruned model. Section IV-C explicitly states, "We have not delved into training any pruned DNN model and optimizing for accuracy," and Section IV-G reports only INT8-vs-INT7 accuracy for unpruned models. Therefore the claims that the encoding scheme "does not compromise the DNN's performance" and that the co-design retains accuracy are unsupported. The authors should train or fine-tune models at the reported semi-structured and unstructured sparsity ratios, apply the weight encoding, and report accuracy per configuration, or explicitly limit the contributions to hardware speedup at matched accuracy.
- [§IV-D, §IV-E, §IV-F] The observed speedup curves in Figures 8–10 lack the measurement protocol needed to reproduce them: the manuscript does not report which layers or models were used, how cycle counts were obtained (timer, performance counter, or simulation), or the exact per-layer sparsity for each data point. A short methodology paragraph and a data table with cycle counts would strengthen the central performance claims and make the comparison with the analytical model verifiable.
minor comments (3)
- [Fig. 1 caption] The caption contains a typo: "resutling" should be "resulting."
- [§III-C2, §III-C3] The instruction name "usss vcmac" is inconsistent with the accelerator name USSA; please use a consistent spelling such as "ussa vcmac" or "us vcmac" throughout.
- [§IV-D] The sentence describing the zero-cycle ideal case is ambiguous: the text says "zero clock cycles are needed for a block of four zeros" in the ideal case, while Equation (co) assigns one cycle; this is explained later, but a one-sentence clarification at the first occurrence would help.
Circularity Check
No circularity found: analytical speedup equations follow from stated hardware behavior and IID assumptions, and self-citations are not load-bearing.
full rationale
The paper's central performance claims rest on analytical models that are derived from the described hardware, not from fitted parameters or self-referential definitions. For USSA, the average cycle counts ca and co are computed from a binomial distribution under an IID sparsity assumption, and speedups are defined as sa = 4/ca and so = 4/co; these are closed-form consequences of the variable-cycle MAC design and the one-cycle all-zero-block behavior. For SSSA, the analytical speedup is the ratio of total blocks to nonzero blocks, and the observed speedup is an independent cycle-count measurement. No equation in the paper is equivalent to its input by construction, and no fitted value is later renamed as a prediction. The authors do cite their own earlier pruning works ([24]-[26]) for the pruning methodology used to generate sparse models, but that methodology is not the load-bearing element of the hardware claim: any pruning method producing the same sparsity pattern would lead to the same cycle counts. The INT7 accuracy claim is supported by an independent measurement in Table II and by an external reference [28], so it is not circular. The reader's concern about comparing USSA/CSA against a four-cycle sequential MAC baseline rather than the SIMD MAC baseline is a benchmarking and correctness issue, not a circularity issue: the speedups are honestly defined with respect to the stated baseline, and no derivation reduces to its own output.
Assumptions & free parameters
free parameters (3)
- Skipped-block counter width =
4 bits (max skip 15)
- Weight block size =
4 weights per CFU word
- Reserved weight bits for encoding =
1 bit per weight (INT7)
assumptions (4)
- domain assumption Useful pruned models with 4:4 all-zero-block sparsity and arbitrary unstructured sparsity can be produced without unacceptable accuracy loss.
- domain assumption Weight zeros follow an IID Bernoulli distribution for the analytical USSA speedup model.
- domain assumption DNN weights are static at inference time, so offline lookahead encoding is valid.
- domain assumption Reducing weight precision from INT8 to INT7 does not degrade inference accuracy for the target applications.
Cite this review
Pith. "Pith review of Hardware/Software Co-Design of RISC-V Extensions for Accelerating Sparse DNNs on FPGAs." pith.science (2026). https://pith.science/paper/DQ32T3E5
@misc{pith2026250419659,
author = {Pith},
title = {Pith review of: Hardware/Software Co-Design of RISC-V Extensions for Accelerating Sparse DNNs on FPGAs},
year = {2026},
howpublished = {\url{https://pith.science/paper/DQ32T3E5}},
note = {Machine review of arXiv:2504.19659}
}
read the original abstract
The customizability of RISC-V makes it an attractive choice for accelerating deep neural networks (DNNs). It can be achieved through instruction set extensions and corresponding custom functional units. Yet, efficiently exploiting these opportunities requires a hardware/software co-design approach in which the DNN model, software, and hardware are designed together. In this paper, we propose novel RISC-V extensions for accelerating DNN models containing semi-structured and unstructured sparsity. While the idea of accelerating structured and unstructured pruning is not new, our novel design offers various advantages over other designs. To exploit semi-structured sparsity, we take advantage of the fine-grained (bit-level) configurability of FPGAs and suggest reserving a few bits in a block of DNN weights to encode the information about sparsity in the succeeding blocks. The proposed custom functional unit utilizes this information to skip computations. To exploit unstructured sparsity, we propose a variable cycle sequential multiply-and-accumulate unit that performs only as many multiplications as the non-zero weights. Our implementation of unstructured and semi-structured pruning accelerators can provide speedups of up to a factor of 3 and 4, respectively. We then propose a combined design that can accelerate both types of sparsities, providing speedups of up to a factor of 5. Our designs consume a small amount of additional FPGA resources such that the resulting co-designs enable the acceleration of DNNs even on small FPGAs. We benchmark our designs on standard TinyML applications such as keyword spotting, image classification, and person detection.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
The Internet of Things (IoT): Applications, In- vestments, and Challenges for Enterprises
I. Lee and K. Lee. “The Internet of Things (IoT): Applications, In- vestments, and Challenges for Enterprises”. In: Business Horizons 58.4 (2015), pp. 431–440. DOI: 10.1016/j.bushor.2015.03.008
-
[2]
R. Singh and S. S. Gill. “Edge AI: A Survey”. In: Internet of Things and Cyber-Physical Systems 3 (2023), pp. 71–92. DOI: 10.1016/j.iotcps.2023.02.004
-
[3]
TinyML: Tools, Applications, Challenges, and Future Research Directions
R. Kallimani, K. Pai, P. Raghuwanshi, S. Iyer, and O. L. A. L´opez. “TinyML: Tools, Applications, Challenges, and Future Research Directions”. In: Multimedia Tools and Applications 83 (2023), pp. 29015–29045. DOI: 10.1007/s11042-023-16740-9
-
[4]
A. Waterman and K. Asanovic. The RISC-V Instruction Set Manual. Volume I: User-Level ISA . 2017. URL: https : / / riscv. org/wp-content/uploads/2017/05/riscv-spec-v2.2.pdf
work page 2017
-
[5]
S. Ha and J. Teich, eds.Handbook of Hardware/Software Codesign. Springer, 2017. DOI: 10.1007/978-94-017-7267-9
-
[6]
CFU Playground: Want a Faster ML Processor? Do it Yourself!
S. Prakash, T. Callahan, J. Bushagour, C. Banbury, A. V . Green, P. Warden, T. Ansell, and V . J. Reddi. “CFU Playground: Want a Faster ML Processor? Do it Yourself!” In: Proceedings of the Conference on Design, Automation and Test in Europe (DATE) (Antwerp, Belgium). IEEE, Apr. 17–19, 2023. DOI: 10.23919/ DATE56975.2023.10137093
arXiv 2023
-
[8]
A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations
C. Hongrong, Z. Miao, and Q. Javen. “A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations”. In: The Computing Research Repository (CoRR) (2023). arXiv: 2308.06767 [cs.LG]
arXiv 2023
-
[9]
Hardware-Aware Evolutionary Explainable Filter Pruning for Convolutional Neural Networks
C. Heidorn, M. Sabih, N. Meyerh ¨ofer, C. Schinabeck, J. Teich, and F. Hannig. “Hardware-Aware Evolutionary Explainable Filter Pruning for Convolutional Neural Networks”. In: International Journal of Parallel Programming52 (Feb. 2024), pp. 40–58. DOI: 10.1007/s10766-024-00760-5
Show all 28 references
-
[10]
ESL Power and Performance Estimation for Heterogeneous MPSoCs using SystemC
M. Streub¨uhr, R. Rosales, R. Hasholzner, C. Haubelt, and J. Teich. “ESL Power and Performance Estimation for Heterogeneous MPSoCs using SystemC”. In: Proceedings of the Forum on Spec- ification and Design Languages (FDL) (Oldenburg, Germany). IEEE, Sept. 13–15, 2011, pp. 202–209
2011
-
[11]
What is the State of Neural Network Pruning?
D. Blalock, J. J. Gonzalez Ortiz, J. Frankle, and J. Guttag. “What is the State of Neural Network Pruning?” In: Proceedings of the Conference on Machine Learning and Systems (MLSys) (Austin, TX, USA). Mar. 2–4, 2020, pp. 129–146. URL: https: / / proceedings . mlsys . org / pap...
2020
-
[12]
Y . Saad. Iterative Methods for Sparse Linear Systems . Other Titles in Applied Mathematics. SIAM, 2003. DOI: 10.1137/1. 9780898718003
2003 doi
-
[13]
Accelerating Sparse Deep Neural Networks
A. Mishra, J. A. Latorre, J. Pool, D. Stosic, D. Stosic, G. Venkatesh, C. Yu, and P. Micikevicius. “Accelerating Sparse Deep Neural Networks”. In: The Computing Research Repository (CoRR) (2021). arXiv: 2104.08378 [cs.LG]
2021 arXiv
-
[14]
SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference
Z. Wang. “SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference”. In: Proceedings of the ACM International Conference on Parallel Architectures and Compilation Techniques (PACT)(Virtual Event). ACM, Oct. 3–7, 2020, pp. 31–42. DOI: 10.1145/3410463.3414654
2020
-
[15]
SNAP: An Efficient Sparse Neural Acceleration Processor for Unstructured Sparse Deep Neural Network Inference
J.-F. Zhang, C.-E. Lee, C. Liu, Y . S. Shao, S. W. Keckler, and Z. Zhang. “SNAP: An Efficient Sparse Neural Acceleration Processor for Unstructured Sparse Deep Neural Network Inference”. In: IEEE Journal of Solid-State Circuits 56.2 (2021), pp. 636–647. DOI: 10.1109/JSSC.2020.3043870
2021
-
[16]
DANNA: A Dimension-Aware Neural Net- work Accelerator for Unstructured Sparsity
X. Liu and H. Feng. “DANNA: A Dimension-Aware Neural Net- work Accelerator for Unstructured Sparsity”. In: Proceedings of the 5th International Conference on Communications, Information System and Computer Engineering (CISCE) (Guangzhou, China). IEEE, Apr. 14–16, 2023, pp. 306...
2023
-
[17]
IndexMAC: A Custom RISC-V Vector In- struction to Accelerate Structured-Sparse Matrix Multiplications
V . Titopoulos, K. Alexandridis, C. Peltekis, C. Nicopoulos, and G. Dimitrakopoulos. “IndexMAC: A Custom RISC-V Vector In- struction to Accelerate Structured-Sparse Matrix Multiplications”. In: Proceedings of the Conference on Design, Automation and Test in Europe (DATE)(Valen...
2024
-
[18]
VexRiscv Core
SpinalHDL. VexRiscv Core. 2023. URL: https : / / github. com / SpinalHDL/VexRiscv (visited on 06/14/2024)
2023
-
[19]
TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems
Mart´ın Abadi et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Software available from tensorflow.org
-
[20]
Learning Multiple Layers of Features from Tiny Images
A. Krizhevsky. “Learning Multiple Layers of Features from Tiny Images”. Master Thesis. University of Toronto, Canada, 2009
2009
-
[21]
Speech Commands: A Dataset for Limited-V ocabulary Speech Recognition
P. Warden. “Speech Commands: A Dataset for Limited-V ocabulary Speech Recognition”. In: The Computing Research Repository (CoRR) (2018). arXiv: 1804.03209 [cs.CL]
2018 arXiv
-
[22]
Visual Wake Words Dataset
A. Chowdhery, P. Warden, J. Shlens, A. G. Howard, and R. Rhodes. “Visual Wake Words Dataset”. In: The Computing Research Repository (CoRR) (2019). arXiv: 1906.05721 [cs.CV]
2019 arXiv
-
[23]
Microsoft COCO: Common Objects in Context
T.-Y . Lin et al. “Microsoft COCO: Common Objects in Context”. In: The Computing Research Repository (CoRR) (2014). arXiv: 1405.0312 [cs.CV]
2014 arXiv
-
[24]
Utilizing Explainable AI for Quantization and Pruning of Deep Neural Networks
M. Sabih, F. Hannig, and J. Teich. “Utilizing Explainable AI for Quantization and Pruning of Deep Neural Networks”. In: The Computing Research Repository (CoRR) (Aug. 20, 2020). arXiv: 2008.09072 [cs.CV]
2020 arXiv
-
[25]
DyFiP: Explainable AI-based Dynamic Filter Pruning of Convolutional Neural Networks
M. Sabih, F. Hannig, and J. Teich. “DyFiP: Explainable AI-based Dynamic Filter Pruning of Convolutional Neural Networks”. In: Proceedings of the 2nd European Workshop on Machine Learning and Systems (EuroMLSys) (Rennes, France). ACM, Apr. 5–8, 2022, pp. 109–115. DOI: 10.1145/3...
2022
-
[26]
MOSP: Multi- Objective Sensitivity Pruning of Deep Neural Networks
M. Sabih, A. Mishra, F. Hannig, and J. Teich. “MOSP: Multi- Objective Sensitivity Pruning of Deep Neural Networks”. In: Proceedings of the IEEE 13th International Green and Sustainable Computing Conference (IGSC) (Pittsburgh, PA, USA). IEEE, Oct. 24–25, 2022, pp. 1–8. DOI: 10 ...
2022
-
[27]
An Efficient Hardware Accelerator for Sparse Convolutional Neural Networks on FPGAs
L. Lu, J. Xie, R. Huang, J. Zhang, W. Lin, and Y . Liang. “An Efficient Hardware Accelerator for Sparse Convolutional Neural Networks on FPGAs”. In: Proceedings of the IEEE 27th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM) (San Diego, C...
2019 doi
-
[28]
EasyQuant: Post-training Quantization via Scale Optimization
D. Wu, Q. Tang, Y . Zhao, M. Zhang, Y . Fu, and D. Zhang. “EasyQuant: Post-training Quantization via Scale Optimization”. In: The Computing Research Repository (CoRR) (2020). arXiv: 2006.16669 [cs.CV]
2020 arXiv
-
[2015]
URL: https://www.tensorflow.org/
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.