REVIEW 5 major objections 6 minor 31 references
ForgeBench: A Machine Learning Benchmark Suite and Auto-Generation Framework for Next-Generation HLS Tools
T0 review · 5 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read ForgeBench auto-generates 6,000+ ML HLS designs to push high-level synthesis toward architecture reuse.
desk verdict Useful benchmark idea and a real gap, but the paper does not validate that its 6,000 generated designs work or that its modular cases compute what they claim; the main evidence table is malformed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a template-based design auto-generation pipeline: a Python generator reads JSON configuration files, selects from a library of HLS-oriented C/C++ kernel templates (GEMM, convolution, attention, normalization, activation, and helper units), and emits complete HLS-ready designs on which parallel synthesis and implementation are run. The modular benchmark suite is built around three identified reuse scopes: tiling (computing large operations as many iterations of a smaller shared tile), functional reuse (sharing the same function across different call sequences), and arithmetic reuse (recognizing mathematically equivalent operations with different hardware implementations). The minimal and maximal tiling choices define a resource–latency tradeoff: minimum tiling minimizes the shared module's size but requires more iterations, while maximum tiling fits many programs in one iteration at the cost of under-utilized hardware. These cases give future tools concrete targets for automatically identifying and instantiating shared modules.
What would settle it
Synthesize full GPT or LLaMA transformer blocks as single HLS designs, measure their resource utilization and latency, and compare the results against the performance predicted by benchmarking the individual kernel-level designs in the 6,000+ suite; if the kernel-level scores systematically mispredict full-model outcomes, the representativeness claim is falsified.
Extended reading notes
Core claim
ForgeBench is an open-source framework and benchmark suite for HLS tool development, centered on ML workloads. The auto-generation framework takes user-specified JSON configurations describing top-level interfaces, global memory, and sequences of ML module calls, then generates complete synthesizable C/C++ designs plus scripts for Vitis HLS synthesis and implementation. The suite contains 6,000+ designs: 1,920 GEMM variants, 2,304 DNN variants (convolution, batch normalization, activation), and 1,944 LLM variants (attention, dropout, normalization), produced by sweeping dimensions, loop orders, unroll factors, and functional options such as grouped attention and RoPE. The second suite contains modular HLS designs where shared computational modules, such as tiled GEMMs, common convolution blocks, and attention submodules, are manually extracted across up to three input programs. For each modular test case, resource utilization before and after modularization is reported on a Xilinx ZCU102 FPGA, showing substantial reductions in LUT and DSP usage when the shared modules are reused.
Load-bearing premise
The load-bearing premise is that parameter sweeps over four kernel templates (GEMM, convolution, attention, and normalization/activation) produce HLS designs representative enough of real ML workloads that improving tools on these designs transfers to full models such as GPT and LLaMA.
Editorial extensions
If this is right
- HLS tool developers gain a large, extensible ML benchmark corpus covering GEMM, DNN, and LLM kernels, with automated generation so new test cases can be created by editing JSON files.
- The modular test cases provide reference implementations for architecture-oriented tools, with measured resource savings quantifying the benefit of shared module extraction.
- The framework can be extended to support other HLS tool flows beyond Vitis HLS, and the generated designs can be integrated with existing HLS optimization frameworks that add advanced pragmas.
- The min/max tiling examples offer a concrete, small-scale setting for designing design-space exploration algorithms that trade off shared-module area against execution latency.
- If adopted, ForgeBench could shift HLS evaluation from single-program accelerator optimization toward multi-program architecture reuse, which the paper argues is the key to reducing hardware redundancy across models like GPT and LLaMA.
Reading between the lines
- The 6,000+ designs are kernel-level parameter sweeps, not full end-to-end model implementations; the benchmark's representativeness for complete models such as GPT-2 or LLaMA depends on the assumption that kernel-level diversity captures the architecture-level reuse opportunities these models exhibit.
- The modularized reference implementations are manually determined, so they represent an upper bound on what an automated modular HLS tool could achieve, rather than a feasible tool output; they are targets, not baselines from an algorithm.
- The paper's vision of using equivalence-graph (e-graph) methods to automatically discover shared modules could be tested directly against the provided modular test cases: an e-graph-based extractor should reproduce the manual module choices and the reported resource reductions.
- The min/max tiling examples suggest a natural quantitative benchmark for future tools: given the set of input programs, the tool must choose a tile size that optimizes a combined area–latency objective, with the GCD-based and HCM-based tilings as anchor points in the Pareto frontier.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. ForgeBench is an ML-focused benchmark suite and automatic HLS design-generation framework. It contributes (i) an extensible template library of GEMM, convolution, attention, activation, and helper kernels; (ii) a Python/JSON-based generator with a parallel Vitis HLS flow; (iii) a suite of 6,168 operator-level HLS test cases; and (iv) a modular benchmark suite in which shared computational modules are manually extracted and resource savings are reported in Table III. The paper argues that current HLS tools lack an architecture-oriented perspective and that ForgeBench will help develop next-generation tools that identify and reuse common hardware modules.
Significance. If the validation gaps are closed, ForgeBench fills a real gap: existing HLS benchmark suites such as MachSuite, Rosetta, and HLSFactory are not ML-centric and do not provide explicit modular-reuse targets. The paper's strengths are its open-source release, the configurable generation framework, and the concrete modular test-case taxonomy (tiling, functional, arithmetic reuse). These are useful artifacts for the HLS/ML hardware community. The modular resource-sharing test cases are the most distinctive contribution; they provide concrete targets that future architecture-oriented HLS tools must handle, which is a valuable falsifiable benchmark design.
major comments (5)
- [III-A, Table II] The central claim of 'over 6,000 ready-to-use ML HLS designs' rests on configuration counts (1,920 + 2,304 + 1,944 = 6,168), but the paper reports no synthesis success rate, no CSIM/CO-SIM pass rate, and no functional verification for any generated design. Since a JSON configuration file does not by itself guarantee a valid HLS program, this count is not evidence of usability. Please report the number of designs that pass HLS synthesis and co-simulation, the HLS tool version and target board, and an analysis of any excluded or failed designs.
- [Table III] Table III is internally inconsistent and cannot be independently checked. For example, the 'Tiled GEMM - Max' row provides only three resource pairs and no after-modularization or change columns; the 'Vec/Mtx Mult - MMV' row contains three pairs where four are expected; and the 'Vec/Mtx Mult - GEMM' row reports a change value without a 'before total'. Please reformat the table with consistent column alignment and verify the arithmetic in every row (for instance, the 'i-j-k Orders' row's before-total DSP value does not exactly equal three times the P1-P3 DSP values).
- [III-B] The central demonstration that modular implementations reduce resources requires that each modularized design compute the same function as its unmodularized counterpart. The paper never states that functional equivalence was verified (e.g., by CO-SIM) for the Tiled GEMM, Vec/Mtx Mult, DNN Blocks, or LLM test cases. Without this check, the resource reductions in Table III could in principle be achieved by designs that no longer implement the intended computation. Please add an equivalence-checking protocol and report per-test-case pass/fail results.
- [III-B, Fig. 2] The paper motivates modular architectures with a resource/latency tradeoff (Fig. 2) and includes Tiled GEMM-Min/Max test cases, but Table III reports only LUT/DSP utilization and explicitly excludes delay. Claiming that 'maximum tiling' comes 'at the expense of under-utilized hardware' without any latency or throughput measurement is unsupported, and the conclusion that architecture-oriented HLS is 'necessary' relies on a resource-only view. Please provide latency/throughput estimates under a consistent control architecture, or clearly state that the modular benchmark claims are resource-only and defer latency evaluation to future work.
- [II-C, Table I] The claim that JSON files 'each taking 1 PhD student around 1 hour to write' is anecdotal and not substantiated; more importantly, the 'representative ML HLS designs' are kernel-level operator variations, not the full models listed in Table I. The abstract and Section I should scope the representativeness claim to operator-level designs, or provide at least one end-to-end model implementation as evidence that the framework can generate complete models such as GPT or LLaMA.
minor comments (6)
- [II-B.1] There is a typo in 'ForgBench' on the line introducing the baseline GEMM operation; it should read 'ForgeBench'.
- [III-B, DNN Testsuite] The word 'testuite' appears in the DNN Testsuite paragraph and in Section III-A; it should be 'test suite'.
- [II-C, Fig. 2] Fig. 2's caption says 'Resource/Latency trade-off' but the figure and text do not report latency values; either add latency data or change the caption to 'resource trade-off'.
- [II-C] The term 'highest common multiple' is nonstandard; if the intended meaning is the least common multiple, the text should use 'least common multiple (LCM)'.
- [Author email and References] The email address in the header appears to have a typo ('haniqu.chen' instead of 'hanqiu.chen'), and reference [11]'s URL contains an unintended space ('language understanding paper.pdf').
- [Table I] The table uses 'LLaMa' in one row and 'LLaMA' elsewhere; please standardize the capitalization.
Circularity Check
No circular derivation: ForgeBench is a benchmark-generation artifact; test-case counts are configuration-sweep outputs, and modular reference outputs are manually defined benchmark ground truth rather than predictions.
full rationale
ForgeBench does not claim to derive a quantitative result from an input that already contains it. The 6,000+ test cases listed in Table II are the outputs of user- or script-specified configuration sweeps over the authors' own templates; the count is a construction artifact, not a fitted or predicted quantity, and 'representative' is an unvalidated label rather than a derived claim. The modular benchmark suite is built by manually choosing shared modules ('We then provide the ideal modularized implementation, determined manually. These are added to the benchmark suite and serve as a reference output for future modular HLS tools'), so the resource-utilization changes in Table III describe the hand-built reference designs themselves; this is benchmark definition, not circular reasoning, though it does mean the resource numbers are not independent evidence for the claimed necessity of architecture-oriented HLS. The only citations to author-affiliated prior work ([14] on DNN co-design for tiling and unrolling background, and [25] HLSFactory for interoperability) are not load-bearing for the central claims. The paper's own limitations — no reported CSIM/CO-SIM pass rates, no synthesis success verification, and the explicit disclaimer in Section III-B that delay and power are omitted because they 'have little meaning' without a control architecture — are validation and rigor gaps, not circularity. Consequently, no specific reduction of a claimed result to its own inputs can be exhibited, so the honest finding is no significant circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption Different ML models share substantial computational components such as attention and GEMM kernels.
- domain assumption Manually extracted shared modules represent the ideal output for future architecture-oriented HLS tools.
- domain assumption Parameter sweeps over operator templates yield HLS designs that are representative of full ML models.
Cite this review
Pith. "Pith review of ForgeBench: A Machine Learning Benchmark Suite and Auto-Generation Framework for Next-Generation HLS Tools." pith.science (2026). https://pith.science/paper/T2GGNSWA
@misc{pith2026250415185,
author = {Pith},
title = {Pith review of: ForgeBench: A Machine Learning Benchmark Suite and Auto-Generation Framework for Next-Generation HLS Tools},
year = {2026},
howpublished = {\url{https://pith.science/paper/T2GGNSWA}},
note = {Machine review of arXiv:2504.15185}
}
read the original abstract
Although High-Level Synthesis (HLS) has attracted considerable interest in hardware design, it has not yet become mainstream due to two primary challenges. First, current HLS hardware design benchmarks are outdated as they do not cover modern machine learning (ML) applications, preventing the rigorous development of HLS tools on ML-focused hardware design. Second, existing HLS tools are outdated because they predominantly target individual accelerator designs and lack an architecture-oriented perspective to support common hardware module extraction and reuse, limiting their adaptability and broader applicability. Motivated by these two limitations, we propose ForgeBench, an ML-focused benchmark suite with a hardware design auto-generation framework for next-generation HLS tools. In addition to the auto-generation framework, we provide two ready-to-use benchmark suites. The first contains over 6,000 representative ML HLS designs. We envision future HLS tools being architecture-oriented, capable of automatically identifying common computational modules across designs, and supporting flexible dataflow and control. Accordingly, the second benchmark suite includes ML HLS designs with possible resource sharing manually implemented to highlight the necessity of architecture-oriented design, ensuring it is future-HLS ready. ForgeBench is open-sourced at https://github.com/hchen799/ForgeBench .
Figures
Reference graph
Works this paper leans on
-
[1]
Machsuite: Benchmarks for accelerator design and customized architectures,
B. Reagenet al., “Machsuite: Benchmarks for accelerator design and customized architectures,” in2014 IEEE International Symposium on Workload Characterization (IISWC), 2014, pp. 110–119
work page 2014
-
[2]
Rosetta: A realistic high-level synthesis benchmark suite for software programmable fpgas,
Y . Zhouet al., “Rosetta: A realistic high-level synthesis benchmark suite for software programmable fpgas,” inProceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, ser. FPGA ’18. New York, NY , USA: Association for Computing Machinery, 2018, p. 269–278. [Online]. Available: https://doi.org/10.1145/3174243.3174255
arXiv 2018
-
[3]
Rodinia: A benchmark suite for heterogeneous computing,
S. Che, M. Boyer, J. Meng, D. Tarjan, J. W. Sheaffer, S.-H. Lee, and K. Skadron, “Rodinia: A benchmark suite for heterogeneous computing,” in2009 IEEE International Symposium on Workload Characterization (IISWC), 2009, pp. 44–54
2009
- [4]
-
[5]
Overgen: Improving fpga usability through domain- specific overlay generation,
S. Liuet al., “Overgen: Improving fpga usability through domain- specific overlay generation,” in2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO), 2022, pp. 35–56
work page 2022
-
[6]
L. Guoet al., “Tapa: A scalable task-parallel dataflow programming framework for modern fpgas with co-optimization of hls and physical design,”ACM Trans. Reconfigurable Technol. Syst., vol. 16, no. 4, Dec. 2023. [Online]. Available: https://doi.org/10.1145/3609335
doi:10.1145/3609335 2023
-
[7]
Heterocl: A multi-paradigm programming infrastructure for software-defined reconfigurable computing,
Y .-H. Laiet al., “Heterocl: A multi-paradigm programming infrastructure for software-defined reconfigurable computing,” in Proceedings of the 2019 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, ser. FPGA ’19. New York, NY , USA: Association for Computing Machinery, 2019, p. 242–251. [Online]. Available: https://doi.org/10.1145/32896...
arXiv 2019
-
[8]
Dsagen: Synthesizing programmable spatial accelerators,
J. Weng, S. Liu, V . Dadu, Z. Wang, P. Shah, and T. Nowatzki, “Dsagen: Synthesizing programmable spatial accelerators,” in2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 2020, pp. 268–281
work page 2020
Show all 31 references
-
[9]
Vitis hls,
Xilinx, “Vitis hls,” https://www.xilinx.com/products/design-tools/vitis/ vitis-hls.html
-
[10]
Llama: Open and efficient foundation language models,
H. Touvronet al., “Llama: Open and efficient foundation language models,” 2023. [Online]. Available: https://arxiv.org/abs/2302.13971
2023 arXiv
-
[11]
Improving language understanding by generative pre-training,
A. Radfordet al., “Improving language understanding by generative pre-training,”OpenAI Blog, 2018. [Online]. Available: https://cdn.openai.com/research-covers/language-unsupervised/ language understanding paper.pdf
2018
-
[12]
Optimizing fpga-based accelerator design for deep convolutional neural networks,
C. Zhanget al., “Optimizing fpga-based accelerator design for deep convolutional neural networks,” inProceedings of the 2015 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, ser. FPGA ’15. New York, NY , USA: Association for Computing Machinery, 2015, p. 16...
2015
-
[13]
Fast convolutional neural networks on fpgas with hls4ml,
T. Aarrestadet al., “Fast convolutional neural networks on fpgas with hls4ml,”Machine Learning: Science and Technology, vol. 2, no. 4, p. 045015, jul 2021. [Online]. Available: https://dx.doi.org/10.1088/2632-2153/ac0ea1
2021 doi
-
[14]
Fpga/dnn co-design: An efficient design methodology for iot intelligence on the edge,
C. Haoet al., “Fpga/dnn co-design: An efficient design methodology for iot intelligence on the edge,” inProceedings of the 56th Annual Design Automation Conference 2019, ser. DAC ’19. New York, NY , USA: Association for Computing Machinery, 2019. [Online]. Available: https://d...
2019
-
[15]
Scalehls: a scalable high-level synthesis framework with multi-level transformations and optimizations: invited,
H. Yeet al., “Scalehls: a scalable high-level synthesis framework with multi-level transformations and optimizations: invited,” inProceedings of the 59th ACM/IEEE Design Automation Conference, ser. DAC ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 1355...
2022
-
[16]
Deep residual learning for image recognition,
K. Heet al., “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778
2016
-
[17]
Very deep convolutional networks for large-scale image recognition,
K. Simonyanet al., “Very deep convolutional networks for large-scale image recognition,”arXiv preprint arXiv:1409.1556, 2014
2014 arXiv
-
[18]
Mobilenets: Efficient convolutional neural networks for mobile vision applications,
A. G. Howardet al., “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” 2017. [Online]. Available: https://arxiv.org/abs/1704.04861
2017 arXiv
-
[19]
Efficientnet: Rethinking model scaling for convolutional neural networks,
M. Tanet al., “Efficientnet: Rethinking model scaling for convolutional neural networks,”CoRR, vol. abs/1905.11946, 2019. [Online]. Available: http://arxiv.org/abs/1905.11946
1905 arXiv
-
[20]
Mobilenetv2: Inverted residuals and linear bottlenecks,
M. Sandleret al., “Mobilenetv2: Inverted residuals and linear bottlenecks,” 2019. [Online]. Available: https://arxiv.org/abs/1801.04381
2019 arXiv
-
[21]
Searching for mobilenetv3,
A. Howardet al., “Searching for mobilenetv3,” 2019. [Online]. Available: https://arxiv.org/abs/1905.02244
2019 arXiv
-
[22]
Mistral 7b,
A. Q. Jianget al., “Mistral 7b,” 2023. [Online]. Available: https://arxiv.org/abs/2310.06825
2023 arXiv
-
[23]
Gemma: Open models based on gemini research and technology,
G. Teamet al., “Gemma: Open models based on gemini research and technology,” 2024. [Online]. Available: https://arxiv.org/abs/2403.08295
2024 arXiv
-
[24]
Autodse: Enabling software programmers to design efficient fpga accelerators,
A. Sohrabizadehet al., “Autodse: Enabling software programmers to design efficient fpga accelerators,”ACM Trans. Des. Autom. Electron. Syst., vol. 27, no. 4, Feb. 2022. [Online]. Available: https://doi.org/10.1145/3494534
2022 doi
-
[25]
Hlsfactory: A framework empowering high-level synthesis datasets for machine learning and beyond,
S. Abi-Karamet al., “Hlsfactory: A framework empowering high-level synthesis datasets for machine learning and beyond,” in2024 ACM/IEEE 6th Symposium on Machine Learning for CAD (MLCAD), 2024, pp. 1–9
2024
-
[26]
Squeezenet: Alexnet-level accuracy with 50x fewer parameters and ¡0.5mb model size,
F. N. Iandolaet al., “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and ¡0.5mb model size,” 2016. [Online]. Available: https://arxiv.org/abs/1602.07360
2016 arXiv
-
[27]
Gemma 2: Improving open language models at a prac- tical size,
G. Teamet al., “Gemma 2: Improving open language models at a prac- tical size,” 2024. [Online]. Available: https://arxiv.org/abs/2408.00118
2024 arXiv
-
[28]
Griffin: Mixing gated linear recurrences with local attention for efficient language models,
S. Deet al., “Griffin: Mixing gated linear recurrences with local attention for efficient language models,” 2024. [Online]. Available: https://arxiv.org/abs/2402.19427
2024 arXiv
-
[29]
babble: Learning better abstractions with e-graphs and anti-unification,
D. Caoet al., “babble: Learning better abstractions with e-graphs and anti-unification,”Proc. ACM Program. Lang., vol. 7, no. POPL, Jan. 2023. [Online]. Available: https://doi.org/10.1145/3571207
2023 doi
-
[30]
Rover: Rtl optimization via verified e-graph rewriting,
S. Cowardet al., “Rover: Rtl optimization via verified e-graph rewriting,”Trans. Comp.-Aided Des. Integ. Cir. Sys., vol. 43, no. 12, p. 4687–4700, Dec. 2024. [Online]. Available: https://doi.org/10.1109/TCAD.2024.3410154
2024
-
[31]
Seer: Super-optimization explorer for high-level synthesis using e-graph rewriting,
J. Chenget al., “Seer: Super-optimization explorer for high-level synthesis using e-graph rewriting,” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Lan- guages and Operating Systems, Volume 2, ser. ASPLOS ’24. New York, NY , US...
2024
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.