{"id":"6ac25bb4-4314-456d-97e8-efed3305051b","arxiv_id":"2504.15185","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ForgeBench is an open-source framework and benchmark suite with over 6,000 generated ML-focused HLS designs and modularized designs that show resource savings from shared hardware modules.","lead":"ForgeBench provides a large open-source benchmark suite of ML-style HLS hardware designs, plus a framework that auto-generates them from JSON settings. A second set of modularized designs shows large resource savings when shared computational blocks are manually extracted, making the case for architecture-oriented HLS tools.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's 'ready-to-use' and resource-sharing claims rest on designs whose functional correctness and synthesizability are never reported, and Table III is internally inconsistent, undermining the central evidence.","rationale":"The reader's weakest assumption concerns representativeness of the 6,000+ parameter-swept designs. That is a real concern, but I judge the more load-bearing condition to be baseline functional correctness and synthesizability. A benchmark suite can be non-representative yet still useful as a stress test, but a suite whose designs fail to compile, fail simulation, or produce incorrect hardware is not 'ready-to-use' at all. The paper counts test cases from JSON configurations (Section III-A, Table II) without any evidence that the generated C/C++ passes HLS CSIM, CO-SIM, or synthesis. Similarly, the modular test cases in Section III-B are presented as manually shared implementations, but no equivalence checking or even mention of simulation results is given; the only quantitative evidence, Table III, is also internally inconsistent, with misaligned rows and missing entries. These issues are verifiable directly from the open-source artifact, and they are more fundamental than representativeness because they affect the validity of every generated design and modular test case. The reader's CONDITIONAL verdict already requires table corrections and validation; my concern reinforces that requirement rather than changing the verdict. I therefore set verdict_should_be to UNCHANGED, with the concrete test above as the decisive check. Agreement is partial because the reader identified representativeness while I identify correctness/synthesizability as the primary load-bearing gap, though both are validation-oriented and complementary.","tokens_in":10195,"tokens_out":4299,"duration_ms":41739,"concrete_test":"Obtain the open-source repository and run the provided Python generation scripts to recreate all 6,168 designs. Execute the Vitis HLS flow with CSIM and CO-SIM enabled on each design and report the pass/fail rates for synthesis and simulation. Then, for every modularized test case in Section III-B, run co-simulation on the original programs and the modularized design with identical input vectors and compare outputs bit-for-bit; any mismatch invalidates that test case. Finally, re-render Table III with one fully populated row per test case and independently verify each 'Change' percentage against the before/after totals; if the percentages do not reproduce, the quantitative resource-sharing evidence is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim to hold, every generated design must be a valid, synthesizable, functionally correct HLS program, and every modularized design must be functionally equivalent to its original programs. The paper never verifies this. Section III-A counts 6,168 test cases from configuration sweeps, but no CSIM/CO-SIM pass rate or synthesis success rate is reported; a configuration file does not guarantee a working HLS design. Section III-B presents Table III as evidence of resource-sharing benefit, but the table is visibly malformed—for instance, the 'Tiled GEMM - Max' row contains only three numeric entries where four are expected, and the 'Vec/Mtx Mult - MMV' row is missing a column—so the quantitative evidence cannot be independently checked. More importantly, the modularized implementations are asserted to share modules but are never checked for functional equivalence against the original programs. Without correctness validation, the 6,000+ count may include designs that fail HLS synthesis or produce incorrect hardware, and the modular test cases may achieve lower resource utilization only by no longer computing the intended functions. The paper also omits latency and throughput, stating that delay/power 'have little meaning' without a control architecture, yet uses resource reduction alone to argue for architecture-oriented HLS tools; a modularized design that is twice as slow cannot demonstrate the claimed necessity. These gaps directly affect both benchmark suites: the general suite's value depends on designs being runnable and correct, and the modular suite's value depends on the shared implementations being correct and beneficial.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ForgeBench is an ML-focused benchmark suite and automatic HLS design-generation framework. It contributes (i) an extensible template library of GEMM, convolution, attention, activation, and helper kernels; (ii) a Python/JSON-based generator with a parallel Vitis HLS flow; (iii) a suite of 6,168 operator-level HLS test cases; and (iv) a modular benchmark suite in which shared computational modules are manually extracted and resource savings are reported in Table III. The paper argues that current HLS tools lack an architecture-oriented perspective and that ForgeBench will help develop next-generation tools that identify and reuse common hardware modules.","tokens_in":10450,"tokens_out":5120,"duration_ms":43157,"significance":"If the validation gaps are closed, ForgeBench fills a real gap: existing HLS benchmark suites such as MachSuite, Rosetta, and HLSFactory are not ML-centric and do not provide explicit modular-reuse targets. The paper's strengths are its open-source release, the configurable generation framework, and the concrete modular test-case taxonomy (tiling, functional, arithmetic reuse). These are useful artifacts for the HLS/ML hardware community. The modular resource-sharing test cases are the most distinctive contribution; they provide concrete targets that future architecture-oriented HLS tools must handle, which is a valuable falsifiable benchmark design.","major_comments":[{"comment":"The central claim of 'over 6,000 ready-to-use ML HLS designs' rests on configuration counts (1,920 + 2,304 + 1,944 = 6,168), but the paper reports no synthesis success rate, no CSIM/CO-SIM pass rate, and no functional verification for any generated design. Since a JSON configuration file does not by itself guarantee a valid HLS program, this count is not evidence of usability. Please report the number of designs that pass HLS synthesis and co-simulation, the HLS tool version and target board, and an analysis of any excluded or failed designs.","section":"III-A, Table II"},{"comment":"Table III is internally inconsistent and cannot be independently checked. For example, the 'Tiled GEMM - Max' row provides only three resource pairs and no after-modularization or change columns; the 'Vec/Mtx Mult - MMV' row contains three pairs where four are expected; and the 'Vec/Mtx Mult - GEMM' row reports a change value without a 'before total'. Please reformat the table with consistent column alignment and verify the arithmetic in every row (for instance, the 'i-j-k Orders' row's before-total DSP value does not exactly equal three times the P1-P3 DSP values).","section":"Table III"},{"comment":"The central demonstration that modular implementations reduce resources requires that each modularized design compute the same function as its unmodularized counterpart. The paper never states that functional equivalence was verified (e.g., by CO-SIM) for the Tiled GEMM, Vec/Mtx Mult, DNN Blocks, or LLM test cases. Without this check, the resource reductions in Table III could in principle be achieved by designs that no longer implement the intended computation. Please add an equivalence-checking protocol and report per-test-case pass/fail results.","section":"III-B"},{"comment":"The paper motivates modular architectures with a resource/latency tradeoff (Fig. 2) and includes Tiled GEMM-Min/Max test cases, but Table III reports only LUT/DSP utilization and explicitly excludes delay. Claiming that 'maximum tiling' comes 'at the expense of under-utilized hardware' without any latency or throughput measurement is unsupported, and the conclusion that architecture-oriented HLS is 'necessary' relies on a resource-only view. Please provide latency/throughput estimates under a consistent control architecture, or clearly state that the modular benchmark claims are resource-only and defer latency evaluation to future work.","section":"III-B, Fig. 2"},{"comment":"The claim that JSON files 'each taking 1 PhD student around 1 hour to write' is anecdotal and not substantiated; more importantly, the 'representative ML HLS designs' are kernel-level operator variations, not the full models listed in Table I. The abstract and Section I should scope the representativeness claim to operator-level designs, or provide at least one end-to-end model implementation as evidence that the framework can generate complete models such as GPT or LLaMA.","section":"II-C, Table I"}],"minor_comments":[{"comment":"There is a typo in 'ForgBench' on the line introducing the baseline GEMM operation; it should read 'ForgeBench'.","section":"II-B.1"},{"comment":"The word 'testuite' appears in the DNN Testsuite paragraph and in Section III-A; it should be 'test suite'.","section":"III-B, DNN Testsuite"},{"comment":"Fig. 2's caption says 'Resource/Latency trade-off' but the figure and text do not report latency values; either add latency data or change the caption to 'resource trade-off'.","section":"II-C, Fig. 2"},{"comment":"The term 'highest common multiple' is nonstandard; if the intended meaning is the least common multiple, the text should use 'least common multiple (LCM)'.","section":"II-C"},{"comment":"The email address in the header appears to have a typo ('haniqu.chen' instead of 'hanqiu.chen'), and reference [11]'s URL contains an unintended space ('language understanding paper.pdf').","section":"Author email and References"},{"comment":"The table uses 'LLaMa' in one row and 'LLaMA' elsewhere; please standardize the capitalization.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a useful artifact and a clear vision, but the central evidence is not yet verifiable: Table III is malformed, and no functional/synthesis validation is reported for either benchmark suite. After the authors address the validation and table-integrity issues, the paper could be a solid contribution. The '1 hour' effort claim should be removed or substantiated, as it reads like marketing rather than evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: ForgeBench fills a real gap—an ML-oriented HLS benchmark generator plus modular test cases—and the open-sourced artifact is worth looking at. But the paper under-delivers on evidence: nothing reports synthesis success or functional correctness, and Table III isn't reliable as printed. It deserves peer review, not desk rejection, but it needs major revision.\n\nWhat is new: the four operator families (GEMM, convolution, attention, activation/normalization) are configured into 6,168 kernel variants, and there is a second suite of modularized designs targeting tiling, functional, and arithmetic reuse. The modular test case idea, with manually defined shared modules as reference outputs, is not in MachSuite, Rosetta, or HLSFactory. The framework is genuinely extensible and open source. That is a useful contribution.\n\nWhere it is soft: the 6,000 count is a count of configs, not validated designs. No CSIM/COSIM pass rates or synthesis success rates are reported, and there is no check that a generated design computes what it should. For the modular suite, the before-vs-after table is the core evidence, but it is malformed: the Tiled GEMM-Max row has three numbers where four columns are expected, Vec/Mtx Mult - MMV appears to be missing a column, and the GEMM row's change in DSP is negative. I couldn't independently check the resource savings, and that is the central quantitative claim. Also, omitting latency and throughput weakens the claim that resource sharing is a win; a modular design that is much slower would not show necessity. The one-hour effort claim in Fig. 1 is anecdotal, and \"representative ML designs\" oversells kernel-level sweeps as representing full models like GPT or LLaMA.\n\nCounterbalance: these are evidential weaknesses, not conceptual ones. The idea of architecture-oriented HLS benchmarks is sound, and the modular test cases are plausibly useful as long as they are validated. I'd want to see a revised version with a corrected table, correctness validation, and toned-down representativeness language.\n\nBottom line: this is for HLS tool builders and benchmark researchers. Send it to peer review with a request for major revision. I would not cite the current version's quantitative results, but I would consider the artifact after validation.","headline":"Useful benchmark idea and a real gap, but the paper does not validate that its 6,000 generated designs work or that its modular cases compute what they claim; the main evidence table is malformed.","tokens_in":11013,"tokens_out":4338,"would_cite":false,"duration_ms":34381,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ForgeBench auto-generates 6,000+ ML HLS designs to push high-level synthesis toward architecture reuse.","keywords":["high-level synthesis","machine learning benchmarks","FPGA","hardware reuse","design auto-generation","LLM accelerators","DNN accelerators","HLS tools"],"falsifier":"Synthesize full GPT or LLaMA transformer blocks as single HLS designs, measure their resource utilization and latency, and compare the results against the performance predicted by benchmarking the individual kernel-level designs in the 6,000+ suite; if the kernel-level scores systematically mispredict full-model outcomes, the representativeness claim is falsified.","tokens_in":10012,"feed_emoji":"🧩","tokens_out":3460,"duration_ms":32033,"temperature":0.7,"pith_summary":"ForgeBench claims that current HLS benchmarks are outdated for machine learning and that existing HLS tools ignore the resource-sharing opportunities across ML accelerators. It provides an auto-generation framework that turns JSON configuration files into HLS-ready C/C++ designs for common ML kernels, producing over 6,000 test cases across GEMM, DNN, and LLM domains. It also ships a second suite of modularized designs with manually implemented resource sharing, reporting large reductions in FPGA resource use when shared modules are extracted. The paper's central claim is that this combination of scale and modular reference targets is what next-generation, architecture-oriented HLS tools need to develop and be evaluated against.","feed_headline":"6,000 ML HLS designs aim to make synthesis architecture-aware","feed_subtitle":"The suite auto-generates GEMM, DNN, and LLM kernels and adds modular designs showing the payoff of sharing hardware.","key_machinery":"The central mechanism is a template-based design auto-generation pipeline: a Python generator reads JSON configuration files, selects from a library of HLS-oriented C/C++ kernel templates (GEMM, convolution, attention, normalization, activation, and helper units), and emits complete HLS-ready designs on which parallel synthesis and implementation are run. The modular benchmark suite is built around three identified reuse scopes: tiling (computing large operations as many iterations of a smaller shared tile), functional reuse (sharing the same function across different call sequences), and arithmetic reuse (recognizing mathematically equivalent operations with different hardware implementations). The minimal and maximal tiling choices define a resource–latency tradeoff: minimum tiling minimizes the shared module's size but requires more iterations, while maximum tiling fits many programs in one iteration at the cost of under-utilized hardware. These cases give future tools concrete targets for automatically identifying and instantiating shared modules.","core_discovery":"ForgeBench is an open-source framework and benchmark suite for HLS tool development, centered on ML workloads. The auto-generation framework takes user-specified JSON configurations describing top-level interfaces, global memory, and sequences of ML module calls, then generates complete synthesizable C/C++ designs plus scripts for Vitis HLS synthesis and implementation. The suite contains 6,000+ designs: 1,920 GEMM variants, 2,304 DNN variants (convolution, batch normalization, activation), and 1,944 LLM variants (attention, dropout, normalization), produced by sweeping dimensions, loop orders, unroll factors, and functional options such as grouped attention and RoPE. The second suite contains modular HLS designs where shared computational modules, such as tiled GEMMs, common convolution blocks, and attention submodules, are manually extracted across up to three input programs. For each modular test case, resource utilization before and after modularization is reported on a Xilinx ZCU102 FPGA, showing substantial reductions in LUT and DSP usage when the shared modules are reused.","pith_inferences":["The 6,000+ designs are kernel-level parameter sweeps, not full end-to-end model implementations; the benchmark's representativeness for complete models such as GPT-2 or LLaMA depends on the assumption that kernel-level diversity captures the architecture-level reuse opportunities these models exhibit.","The modularized reference implementations are manually determined, so they represent an upper bound on what an automated modular HLS tool could achieve, rather than a feasible tool output; they are targets, not baselines from an algorithm.","The paper's vision of using equivalence-graph (e-graph) methods to automatically discover shared modules could be tested directly against the provided modular test cases: an e-graph-based extractor should reproduce the manual module choices and the reported resource reductions.","The min/max tiling examples suggest a natural quantitative benchmark for future tools: given the set of input programs, the tool must choose a tile size that optimizes a combined area–latency objective, with the GCD-based and HCM-based tilings as anchor points in the Pareto frontier."],"forward_implications":["HLS tool developers gain a large, extensible ML benchmark corpus covering GEMM, DNN, and LLM kernels, with automated generation so new test cases can be created by editing JSON files.","The modular test cases provide reference implementations for architecture-oriented tools, with measured resource savings quantifying the benefit of shared module extraction.","The framework can be extended to support other HLS tool flows beyond Vitis HLS, and the generated designs can be integrated with existing HLS optimization frameworks that add advanced pragmas.","The min/max tiling examples offer a concrete, small-scale setting for designing design-space exploration algorithms that trade off shared-module area against execution latency.","If adopted, ForgeBench could shift HLS evaluation from single-program accelerator optimization toward multi-program architecture reuse, which the paper argues is the key to reducing hardware redundancy across models like GPT and LLaMA."],"supporting_citations":[{"why":"MachSuite is the existing accelerator benchmark suite that ForgeBench identifies as outdated and lacking ML coverage, serving as the comparison baseline for why a new benchmark is needed.","marker":"[1]"},{"why":"Rosetta is the existing HLS benchmark suite that ForgeBench argues does not cover modern ML applications, providing the motivation for an ML-focused HLS benchmark.","marker":"[2]"},{"why":"Vitis HLS is the commercial HLS tool flow the framework targets for synthesis and implementation, through which all resource utilization numbers are produced.","marker":"[9]"},{"why":"The LLaMA model supplies the grouped multi-head attention, RMSNorm, and RoPE operations that ForgeBench's LLM templates are built to represent.","marker":"[10]"},{"why":"The GPT model supplies the multi-head attention, LayerNorm, and dropout operations that anchor ForgeBench's LLM test suite and the GPT/LLaMA shared-module test case.","marker":"[11]"},{"why":"ResNet provides the specific 3x3 convolution blocks used in the DNN modular test cases, demonstrating cross-model reuse of identical convolution and residual blocks.","marker":"[16]"},{"why":"HLSFactory is cited as an existing HLS dataset framework that ForgeBench-generated designs can be integrated with, supporting the extensibility claim for incorporating the benchmarks into other toolchains.","marker":"[25]"}],"fun_headline_variants":["ForgeBench auto-generates 6000+ ML designs for HLS","ForgeBench shows module reuse cuts FPGA resources in HLS","Open-source ForgeBench brings ML benchmarks to HLS","ForgeBench: 6000+ ML designs to modernize HLS tools"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that parameter sweeps over four kernel templates (GEMM, convolution, attention, and normalization/activation) produce HLS designs representative enough of real ML workloads that improving tools on these designs transfers to full models such as GPT and LLaMA.","fun_headline_variants_meta":{"raw":{"variants":["ForgeBench auto-generates 6000+ ML designs for HLS","ForgeBench shows module reuse cuts FPGA resources in HLS","Open-source ForgeBench brings ML benchmarks to HLS","ForgeBench: 6000+ ML designs to modernize HLS tools"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002031,"raw_usage":{"total_tokens":7937,"prompt_tokens":991,"completion_tokens":6946,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":6869}},"tokens_in":607,"tokens_out":6946,"duration_ms":48058,"temperature":1.0,"reasoning_tokens":6869,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:30:38.586424+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Synthesize full GPT or LLaMA transformer blocks as single HLS designs, measure their resource utilization and latency, and compare the results against the performance predicted by benchmarking the individual kernel-level designs in the 6,000+ suite; if the kernel-level scores systematically mispredict full-model outcomes, the representativeness claim is falsified.","supporting_citations":[{"cited_title":"Machsuite: Benchmarks for accelerator design and customized architectures,","cited_arxiv_id":null,"evidence_quote":"MachSuite is the existing accelerator benchmark suite that ForgeBench identifies as outdated and lacking ML coverage, serving as the comparison baseline for why a new benchmark is needed."},{"cited_title":"Vitis hls,","cited_arxiv_id":null,"evidence_quote":"Vitis HLS is the commercial HLS tool flow the framework targets for synthesis and implementation, through which all resource utilization numbers are produced."},{"cited_title":"Improving language understanding by generative pre-training,","cited_arxiv_id":null,"evidence_quote":"The GPT model supplies the multi-head attention, LayerNorm, and dropout operations that anchor ForgeBench's LLM test suite and the GPT/LLaMA shared-module test case."},{"cited_title":"Deep residual learning for image recognition,","cited_arxiv_id":null,"evidence_quote":"ResNet provides the specific 3x3 convolution blocks used in the DNN modular test cases, demonstrating cross-model reuse of identical convolution and residual blocks."},{"cited_title":"Hlsfactory: A framework empowering high-level synthesis datasets for machine learning and beyond,","cited_arxiv_id":null,"evidence_quote":"HLSFactory is cited as an existing HLS dataset framework that ForgeBench-generated designs can be integrated with, supporting the extensibility claim for incorporating the benchmarks into other toolchains."}],"review_version":1}