REVIEW 5 major objections 6 minor 37 references
Towards An Approach to Identify Divergences in Hardware Designs for HPC Workloads
T0 review · 5 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read A bottom-up decomposition of FFT, GEMM, and QR into primitives, implemented in Verilog, Chisel, and HLS, reveals where the tools diverge in achievable frequency and resource use.
desk verdict A genuine methodology contribution—primitive-level decomposition plus automated achievable-frequency search—whose quantitative evidence is too uncontrolled and incomplete to support the tool-divergence claims as stated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the pairing of primitive-level decomposition with an automated achievable-frequency autotuner. The autotuner iteratively sets a target frequency, runs synthesis and implementation, reads the worst negative slack (WNS), and computes the achievable frequency as fa[i] = f[i] / (1 - WNS[i] * f[i]), then stochastically picks the next target frequency. This loop produces a frequency-resource frontier for each design at each decomposition level, turning tool comparison into a searchable design-space exploration rather than a single end-to-end benchmark.
What would settle it
A third-party replication in which each primitive and algorithm is implemented in Verilog, Chisel, and HLS by different engineers given equal time, targeting the same FPGA and using the same autotuner, should reproduce the main ordering: Verilog at the highest frequency, HLS at the highest resource cost for wide streaming. If the ordering flips or disappears under matched effort, the divergences are implementation artifacts rather than tool-level properties.
Extended reading notes
Core claim
The paper claims that a hierarchical decomposition of kernels into primitives—multiply-accumulate arrays, butterfly units, permutation units, pointwise operations, reduction trees, and AXPY operations—combined with an automated frequency-search loop, produces fairer and more granular comparisons among hardware design tools than the full-application benchmarks found in prior work. It reports that for small streaming widths all three implementations consume similar resources, with Verilog consistently reaching higher frequencies; as streaming width grows, HLS requires progressively more resources in compute-heavy primitives and in GEMM and FFT assemblies, while the QR factorization converges a
Load-bearing premise
The load-bearing premise is that the three hand-written designs are equally mature and comparable representatives of their tools; the paper states the implementations were made by hand with efficient Verilog as the focus, but it does not control for author effort or optimization quality, so the observed divergences could reflect implementer skill rather than tool capability.
Editorial extensions
If this is right
- Designers can attribute a performance or resource gap to a specific primitive rather than to the whole application, making optimization effort more targeted.
- Tool developers get a direct view of which constructs—such as permutation units or streaming interfaces—cause divergence in frequency or LUT usage.
- The same primitive decomposition can be applied to other kernels, since the building blocks (MAC, butterfly, permutation, pointwise, reduction, AXPY) are reusable across many HPC computations.
- Automated achievable-frequency search removes much of the manual timing-closure burden, making wide design-space exploration practical on clusters or single servers.
- Because comparisons happen at every level, the methodology is more likely to expose tool-specific strengths than a single end-to-end number.
Reading between the lines
- An extension beyond the paper: a controlled study that matches optimization effort across tools—or uses independently written designs from multiple engineers—would separate tool capability from implementer skill, which the present comparison does not do.
- The primitive set could evolve into a de facto microbenchmark suite for hardware generation tools, similar to compiler test suites, if the decomposition is standardized across more kernels and tools.
- The stochastic frequency-search loop could be paired with power, flip-flop, BRAM, or different FPGA families to test whether the observed divergence ordering holds beyond the single Alveo U280 target.
- The QR result, where HLS detects redundant dot products, hints that data-path reuse is a key divergence point; the paper reports it but does not generalize it into a design rule.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a bottom-up methodology for comparing hardware design flows (Verilog, Chisel, Vivado HLS) on HPC kernels. It decomposes GEMM, FFT, and QR factorization into primitive building blocks (MAC arrays, butterfly units, permutation units, pointwise operations, reduction trees), hand-implements each primitive and algorithm in the three environments, and uses an automated Vivado-based loop to search for a reported 'achievable frequency' and to record resource utilization. The authors report synthesizing approximately 1,500 configurations and present a representative subset in Figure 8, concluding that the methodology identifies divergence points in code generation, resource usage, and performance. The paper explicitly frames the main contribution as the methodology itself, with the concrete results serving as a proof of concept.
Significance. If the methodology were rigorously validated, it would be a useful complement to existing end-to-end tool comparisons by isolating where high-level tools diverge from hand-written RTL. The hierarchical decomposition into reusable primitives is sensible, and the automated exploration harness (Slurm/shell support, JSON configuration, interactive plotting) is a practical infrastructure contribution. The authors are candid that the autotuning constants are arbitrary and that the quantitative results are only a proof of concept. However, the current empirical support is too thin and uncontrolled to sustain the tool-level divergence claims: the comparison is only as informative as the three hand-written implementations being compared, and the reported metrics lack variance information and verification evidence.
major comments (5)
- [§III-B, §IV-B] The comparisons treat the three hand-written implementations as representative of their respective tools, but no control exists for implementation quality, author effort, or microarchitecture equivalence. Section III-B states: 'we have focused on efficient Verilog implementations, we provided a similar Chisel implementation and finally we have made use of the features offered by HLS.' There is no third-party baseline, no optimization-maturity metric, and no standardization of pipeline depth, operator latency, or interface protocol. The consistent pattern in Fig. 8 (Verilog at higher frequency, HLS at higher LUT count) could therefore reflect differential designer skill or different algorithmic mappings rather than intrinsic tool capability. This load-bearing issue undermines the 'fairer comparisons' and 'divergence points' claims. The authors should either add controls (e.g., a public ba
- [§IV-A, Fig. 8] The quantitative support is weakened by the absence of error bars or run-to-run variance. The autotuner in Eq. (12) injects stochastic offsets U(0, F) and U(-α, α), yet each reported point appears to come from a single run. Vivado synthesis and implementation also exhibit run-to-run variability, so the observed differences between tools and configurations may be within the noise of the stochastic search. Additionally, 'a representative subset' of the ~1,500 implementations is shown without defining the selection criteria. Please report multiple trials per configuration, provide variance statistics, and describe how the subset was chosen.
- [§IV-B, Fig. 8(f)] For the QR factorization, the results are explicitly limited to the smallest configuration: 'Due to delays in generating the larger Verilog and Chisel designs, we only report results for the smallest configuration.' Contributions list an 'in-depth analysis' for the QR algorithm, but the evidence is a single point per tool. This is insufficient to support any cross-configuration or tool-level conclusion for QR. The authors should either complete the larger QR runs or scale back the QR claims to a single-case demonstration.
- [§III-B, stopping criterion 3] The stability stopping rule is printed as min_{j∈[i-W,i]} f_a[j] / min_{j∈[i-W,i]} f_a[j] < R. As written, the numerator and denominator are identical, so the ratio is always 1 and the criterion is vacuous for any R ≤ 1 (and trivially always true for R > 1). The intended ratio is presumably max/min over the window. Because this rule is part of the automated method that generates all frequency results, the typo needs correction, and the chosen W and R values should be reported.
- [§IV-B] The paper asserts functional correctness ('our implementations function correctly across the three languages and varying streaming widths') but does not describe any testbench, simulation, or equivalence-checking methodology. Resource and frequency numbers for incorrect or partially correct designs would be meaningless. Please describe the verification flow: test vectors, comparison against a software model, coverage of streaming widths and algorithms, and whether all ~1,500 configurations were verified.
minor comments (6)
- [Eq. (11)] The units for f and WNS are not stated. The formula f_a = f/(1 - wns·f) is dimensionally sensitive: if f is in MHz and wns in ns, a factor of 1000 is missing. Specify the units and verify the formula dimensionally.
- [Fig. 8 caption] Typo: 'LU Ts' should be 'LUTs'.
- [§III-A (DFT)] The sentence 'we focus on the streaming Fourier transform []' contains an empty citation placeholder.
- [§V] Typo: 'HPC e kernels' should be 'HPC kernels'.
- [Fig. 2(a)] The SystemVerilog snippet shows 'generate' but not the matching 'endgenerate'; the ellipsis placement is confusing. Consider showing complete synthesizable code or indicating omitted lines more clearly.
- [§IV-A] The paper says 'approximately 1500 implementations' here and 'nearly 1,500 designs' in the conclusion; the exact count and the configuration grid (which parameters were swept, how many per benchmark) should be stated precisely.
Circularity Check
No significant circularity: the frequency search and decomposition comparisons are self-contained; the only self-citation is a non-load-bearing design reference.
full rationale
The paper's central derivation chain is methodological rather than predictive. The achievable-frequency update (Eq. 11) is the standard WNS-based timing correction, and the stochastic next-frequency rule (Eq. 12) is an explicitly heuristic search step whose constants are admitted to be 'chosen arbitrarily' — not fitted to the reported outcomes. The algorithmic decompositions (GEMM to AXPY+reduction, FFT to butterfly+twiddle+permute, QR to pointwise+reduction+AXPY) are derived from the displayed equations (Eqs. 7–10) and standard references [29,30,32,33,35–37]; no prediction is defined in terms of its own output. The only self-citation is [34], a prior streaming-FFT paper by overlapping authors, but it is used only as a recommended reference for details ('we recommend the reader peruse the following works [34]–[36]') and is not invoked to constrain or justify a result. The uncontrolled-implementation-quality concern raised by a skeptical reader is a threat to external validity of the tool comparison, not circularity: it concerns whether the hand-written Verilog/Chisel/HLS files are equally optimized, which no equation in the paper defines into existence. Accordingly no circular step meets the evidentiary bar.
Assumptions & free parameters
free parameters (5)
- alpha in stochastic frequency update
- F (maximum random offset)
- R (stability threshold)
- W (stability window)
- F_MAX (system frequency cap) =
not stated
assumptions (5)
- domain assumption Vivado's WNS report accurately predicts the maximum achievable clock frequency given by Eq. 11.
- domain assumption The hand-written Verilog, Chisel and HLS designs are comparable in optimization effort and algorithm structure, so differences measure tool capability.
- domain assumption A single Alveo U280 board and default Vivado 2024.1 strategies yield representative resource/frequency results.
- domain assumption The three kernel implementations are functionally correct.
- domain assumption The algorithm decompositions (Cooley-Tukey with streaming, column-scaling GEMM, Householder QR) map faithfully onto the chosen primitives.
Cite this review
Pith. "Pith review of Towards An Approach to Identify Divergences in Hardware Designs for HPC Workloads." pith.science (2026). https://pith.science/paper/BKVOA57P
@misc{pith2026250909774,
author = {Pith},
title = {Pith review of: Towards An Approach to Identify Divergences in Hardware Designs for HPC Workloads},
year = {2026},
howpublished = {\url{https://pith.science/paper/BKVOA57P}},
note = {Machine review of arXiv:2509.09774}
}
read the original abstract
Developing efficient hardware accelerators for mathematical kernels used in scientific applications and machine learning has traditionally been a labor-intensive task. These accelerators typically require low-level programming in Verilog or other hardware description languages, along with significant manual optimization effort. Recently, to alleviate this challenge, high-level hardware design tools like Chisel and High-Level Synthesis have emerged. However, as with any compiler, some of the generated hardware may be suboptimal compared to expert-crafted designs. Understanding where these inefficiencies arise is crucial, as it provides valuable insights for both users and tool developers. In this paper, we propose a methodology to hierarchically decompose mathematical kernels - such as Fourier transforms, matrix multiplication, and QR factorization - into a set of common building blocks or primitives. Then the primitives are implemented in the different programming environments, and the larger algorithms get assembled. Furthermore, we employ an automatic approach to investigate the achievable frequency and required resources. Performing this experimentation at each level will provide fairer comparisons between designs and offer guidance for both tool developers and hardware designers to adopt better practices.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[25]
An overview of today’s high-level synthesis tools.Des
Wim Meeus, Kristof Van Beeck, Toon Goedem ´e, Jan Meel, and Dirk Stroobandt. An overview of today’s high-level synthesis tools.Des. Autom. Embedded Syst., 16(3):31–51, September 2012
2012
-
[28]
H ¨am¨al¨ainen
Matti K ¨ayr¨a and Timo D. H ¨am¨al¨ainen. A survey on system-on-a-chip design using chisel hw construction language. InIECON 2021 – 47th Annual Conference of the IEEE Industrial Electronics Society, page 1–6. IEEE Press, 2021
2021
-
[1]
Prentice Hall Professional, 2003
Samir Palnitkar.Verilog HDL: a guide to digital design and synthesis, volume 1. Prentice Hall Professional, 2003
2003
-
[2]
Springer Science & Business Media, 1989
Roger Lipsett, Carl F Schaefer, and Cary Ussery.VHDL: Hardware description and design. Springer Science & Business Media, 1989
1989
-
[3]
Vivado design suite—vivadohls
Xilinx Inc. Vivado design suite—vivadohls
-
[4]
Anderson, Stephen Brown, and Tomasz Cza- jkowski
Andrew Canis, Jongsok Choi, Mark Aldham, Victor Zhang, Ahmed Kammoona, Jason H. Anderson, Stephen Brown, and Tomasz Cza- jkowski. Legup: high-level synthesis for fpga-based processor/accelera- tor systems. InProceedings of the 19th ACM/SIGDA International Sym- posium on Field Programmable Gate Arrays, FPGA ’11, page 33–36, New York, NY , USA, 2011. Associ...
2011
-
[5]
Bambu: A modular framework for the high level synthesis of memory-intensive applications
Christian Pilato and Fabrizio Ferrandi. Bambu: A modular framework for the high level synthesis of memory-intensive applications. In 2013 23rd International Conference on Field programmable Logic and Applications, pages 1–4, 2013
2013
-
[6]
Scalehls: A new scalable high-level synthesis framework on multi-level intermediate represen- tation
Hanchen Ye, Cong Hao, Jianyi Cheng, Hyunmin Jeong, Jack Huang, Stephen Neuendorffer, and Deming Chen. Scalehls: A new scalable high-level synthesis framework on multi-level intermediate represen- tation. In2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 741–755, 2022
2022
Show all 37 references
-
[7]
Chisel: Constructing Hardware in a Scala Embed- ded Language
UC Berkeley, EECS. Chisel: Constructing Hardware in a Scala Embed- ded Language. Software, 2022. Accessed on March 9, 2024
2022
-
[8]
R. Nikhil. Bluespec system verilog: efficient, correct rtl from high level specifications. InProceedings. Second ACM and IEEE International Conference on Formal Methods and Models for Co-Design, 2004. MEMOCODE ’04., pages 69–70, 2004
2004
-
[9]
Scala language specification
The Scala Team. Scala language specification. https://docs.scala-lang. org/sips/. Accessed: 2025-06-20
2025
-
[10]
Thundergp: Resource-efficient graph processing framework on fpgas with hls.ACM Trans
Xinyu Chen, Feng Cheng, Hongshi Tan, Yao Chen, Bingsheng He, Weng-Fai Wong, and Deming Chen. Thundergp: Resource-efficient graph processing framework on fpgas with hls.ACM Trans. Recon- figurable Technol. Syst., 15(4), December 2022
2022
-
[11]
Gahls: an optimized graph analytics based high level synthesis framework.Scientific Reports, 13(1), 12 2023
Yao Xiao, Shahin Nazarian, and Paul Bogdan. Gahls: an optimized graph analytics based high level synthesis framework.Scientific Reports, 13(1), 12 2023
2023
-
[12]
Peter Hofstee
Joost Hoozemans, Johan Peltenburg, Fabian Nonnemacher, Akos Had- nagy, Zaid Al-Ars, and H. Peter Hofstee. Fpga acceleration for big data analytics: Challenges and opportunities.IEEE Circuits and Systems Magazine, 21(2):30–47, 2021
2021
-
[13]
Energy-efficient acceleration of mapreduce applications using fpgas
Katayoun Neshatpour, Maria Malik, Avesta Sasan, Setareh Rafatirad, Tinoush Mohsenin, Hassan Ghasemzadeh, and Houman Homayoun. Energy-efficient acceleration of mapreduce applications using fpgas. Journal of Parallel and Distributed Computing, 119:1–17, 2018
2018
-
[14]
Hardware accelerated novel optical de novo assembly for large-scale genomes
Pingfan Meng, Matthew Jacobsen, Motoki Kimura, Vladimir Dergachev, Thomas Anantharaman, Michael Requa, and Ryan Kastner. Hardware accelerated novel optical de novo assembly for large-scale genomes. In 2014 24th International Conference on Field Programmable Logic and Applicati...
2014
-
[15]
Algorithm-hardware co-design for bqsr acceleration in genome analysis toolkit
Michael Lo, Zhenman Fang, Jie Wang, Peipei Zhou, Mau-Chung Frank Chang, and Jason Cong. Algorithm-hardware co-design for bqsr acceleration in genome analysis toolkit. In2020 IEEE 28th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), pages ...
2020
-
[16]
Hardware acceleration of long read pairwise overlapping in genome sequencing: A race between fpga and gpu
Licheng Guo, Jason Lau, Zhenyuan Ruan, Peng Wei, and Jason Cong. Hardware acceleration of long read pairwise overlapping in genome sequencing: A race between fpga and gpu. In2019 IEEE 27th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), p...
2019
-
[17]
Nothaft, Qijing Huang, Sagar Karandikar, Johnny Le, Andrew Lin, Howard Mao, Brendan Sweeney, Krste Asanovi ´c, David A
Lisa Wu, David Bruns-Smith, Frank A. Nothaft, Qijing Huang, Sagar Karandikar, Johnny Le, Andrew Lin, Howard Mao, Brendan Sweeney, Krste Asanovi ´c, David A. Patterson, and Anthony D. Joseph. Fpga accelerated indel realignment in the cloud. In2019 IEEE International Symposium o...
2019
-
[18]
Wright, and Samuel Williams
Tan Nguyen, Colin MacLean, Marco Siracusa, Douglas Doerfler, Nicholas J. Wright, and Samuel Williams. Fpga-based hpc accelerators: An evaluation on performance and energy efficiency.Concurrency and Computation: Practice and Experience, 34(20):e6570, 2022
2022
-
[19]
Accelerating face detection on programmable soc using c-based synthe- sis
Nitish Kumar Srivastava, Steve Dai, Rajit Manohar, and Zhiru Zhang. Accelerating face detection on programmable soc using c-based synthe- sis. InProceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, FPGA ’17, page 195–200, New York, NY , ...
2017
-
[20]
Fiplib: An image processing library for fpgas using high-level synthesis.Int
Paolo Palazzari, Marco Faltelli, and Francesco Iannone. Fiplib: An image processing library for fpgas using high-level synthesis.Int. J. Parallel Program., 53(2), February 2025
2025
-
[21]
Preußer, Nicholas J
Michaela Blott, Thomas B. Preußer, Nicholas J. Fraser, Giulio Gam- bardella, Kenneth O’brien, Yaman Umuroglu, Miriam Leeser, and Kees Vissers. Finn-r: An end-to-end deep-learning framework for fast ex- ploration of quantized neural networks.ACM Trans. Reconfigurable Technol. S...
2018
-
[22]
Systolic-cnn: An opencl- defined scalable run-time-flexible fpga accelerator architecture for ac- celerating convolutional neural network inference in cloud/edge com- puting
Akshay Dua, Yixing Li, and Fengbo Ren. Systolic-cnn: An opencl- defined scalable run-time-flexible fpga accelerator architecture for ac- celerating convolutional neural network inference in cloud/edge com- puting. In2020 IEEE 28th Annual International Symposium on Field- Progr...
2020
-
[23]
Duarte, S
J. Duarte, S. Han, P. Harris, S. Jindariani, E. Kreinar, B. Kreis, J. Nga- diuba, M. Pierini, R. Rivera, N. Tran, and Z. Wu. Fast inference of deep neural networks in fpgas for particle physics.Journal of Instrumentation, 13(07):P07027, jul 2018
2018
-
[24]
A scalable and efficient convolutional neural network accelerator using hls for a system-on-chip design.Microprocessors and Microsystems, 87:104363, 2021
Kim Bjerge, Jonathan Horsted Schougaard, and Daniel Ejnar Larsen. A scalable and efficient convolutional neural network accelerator using hls for a system-on-chip design.Microprocessors and Microsystems, 87:104363, 2021
2021
-
[26]
A survey and evalua- tion of fpga high-level synthesis tools.IEEE Transactions on Computer- Aided Design of Integrated Circuits and Systems, 35(10):1591–1604, 2016
Razvan Nane, Vlad-Mihai Sima, Christian Pilato, Jongsok Choi, Blair Fort, Andrew Canis, Yu Ting Chen, Hsuan Hsiao, Stephen Brown, Fab- rizio Ferrandi, Jason Anderson, and Koen Bertels. A survey and evalua- tion of fpga high-level synthesis tools.IEEE Transactions on Computer- ...
2016
-
[27]
High-level synthesis versus hardware construc- tion
Alexander Kamkin, Mikhail Chupilko, Mikhail Lebedev, Sergey Smolov, and Georgi Gaydadjiev. High-level synthesis versus hardware construc- tion. In2023 Design, Automation, Test in Europe Conference and Exhibition (DATE), pages 1–6, 2023
2023
-
[29]
An algorithm for the machine calculation of complex fourier series.Mathematics of Computation, 19(90):297– 301, 1965
James Cooley and John Tukey. An algorithm for the machine calculation of complex fourier series.Mathematics of Computation, 19(90):297– 301, 1965
1965
-
[30]
An adaptation of the fast fourier transform for parallel processing.Journal of the ACM (JACM), 15(2):252–264, 1968
Marshall C Pease. An adaptation of the fast fourier transform for parallel processing.Journal of the ACM (JACM), 15(2):252–264, 1968
1968
-
[31]
Computer gener- ation of streaming sorting networks
Marcela Zuluaga, Peter Milder, and Markus P ¨uschel. Computer gener- ation of streaming sorting networks. InProceedings of the 49th Annual Design Automation Conference, DAC ’12, page 1245–1253, New York, NY , USA, 2012. Association for Computing Machinery
2012
-
[32]
Milder, and James C
Markus P ¨uschel, Peter A. Milder, and James C. Hoe. Permuting streaming data using rams.J. ACM, 56(2), April 2009
2009
-
[33]
Jarvinen, P
T. Jarvinen, P. Salmela, H. Sorokin, and J. Takala. Stride permutation networks for array processors. InProceedings. 15th IEEE International Conference on Application-Specific Systems, Architectures and Proces- sors, 2004., pages 376–386, 2004
2004
-
[34]
Towards a Flexible Hardware Implementation for Mixed-Radix Fourier Transforms
Mario Vega, Xiaokun Yang, John Shalf, and Doru Thom Popovici. Towards a Flexible Hardware Implementation for Mixed-Radix Fourier Transforms. In2023 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–7, 2023
2023
-
[35]
Milder, Aliaksei Sandryhaila, Franz Franchetti, James C
Paolo D’Alberto, Peter A. Milder, Aliaksei Sandryhaila, Franz Franchetti, James C. Hoe, Jose M.F. Moura, Markus Puschel, and Jeremy R. Johnson. Generating fpga-accelerated dft libraries. In15th Annual IEEE Symposium on Field-Programmable Custom Computing Machines (FCCM 2007), ...
2007
-
[36]
Hoe, and Markus P ¨uschel
Peter Milder, Franz Franchetti, James C. Hoe, and Markus P ¨uschel. Computer generation of hardware for linear digital signal processing transforms.ACM Trans. Des. Autom. Electron. Syst., 17(2), April 2012
2012
-
[37]
Rafique, N
A. Rafique, N. Kapre, and G. A. Constantinides. Enhancing performance of tall-skinny QR factorization using FPGAs. InProceedings of the 22nd International Conference on Field Programmable Logic and Applications (FPL), pages 443–450, 2012. 9
2012
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.