{"id":"0258f79d-f42c-4b68-b8b6-46e26e9e954d","arxiv_id":"2509.09774","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A bottom-up primitive-decomposition framework with automated frequency search for comparing Verilog, Chisel, and Vivado HLS on FFT, GEMM, and QR hardware designs.","lead":"This paper proposes a structured way to compare hardware design tools by breaking math kernels like FFT, matrix multiplication, and QR factorization into small common building blocks, then testing each in Verilog, Chisel, and Vivado HLS. It adds an automated synthesis loop that searches for the highest clock speed each design can reach, and reports that hand-written Verilog tends to win on frequency and resources.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Uncontrolled implementation quality makes the reported Verilog-vs-Chisel/HLS divergences uninterpretable as tool comparisons.","rationale":"The reader identified Section III-B's lack of a control for implementation quality as the weakest assumption; I agree. This is the single most load-bearing issue because it determines whether Fig. 8's divergences can be interpreted as properties of Verilog, Chisel, and HLS at all. The paper's own conclusions are carefully hedged ('main value is not in which of these specific kernels performs better'), so the framework-level contribution survives the concern; however, the abstract's 'fairer comparisons' and 'identify divergence points' go beyond a framework and require controlled implementations. The degenerate stopping rule in Eq. 12 (criterion 3 compares min to itself) is a genuine mechanical defect that should also be corrected and the autotuning re-run, but it is secondary: even a perfect search would not fix the implementation-quality confound. A revision should ship the repository, add a second independent implementation for at least one primitive, and report run-to-run variation. With those additions the quantitative claims could be supported; without them, the empirical comparisons remain illustrative rather than conclusive.","tokens_in":12089,"tokens_out":6088,"duration_ms":71880,"concrete_test":"Select one primitive, e.g. the Array of MACs at streaming width 8, and ask an independent hardware engineer to implement the same RTL specification (fixed pipeline stages, fixed operator latencies, same AXI-Stream interface) in Chisel and in Vivado HLS, without knowledge of the authors' implementations. Rerun the automated frequency/resource search for all three versions under the same Vivado 2024.1/U280 flow. If the Verilog advantage and HLS resource overhead in Fig. 8(a) persist, the tool-level divergence is robust; if it shrinks, reverses, or varies by implementer, the reported divergences are dominated by implementation quality rather than tool capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-B states: 'we have focused on efficient Verilog implementations, we provided a similar Chisel implementation and finally we have made use of the features offered by HLS for all primitives and algorithms.' The central claim that the bottom-up framework yields 'fairer comparisons' and identifies 'divergence points in code generation' requires that the three hand-written designs be equivalent instantiations of the same microarchitecture. No such control exists: there is no independent or third-party implementation, no metric of effort or optimization maturity, and no standardization of pipeline depth, operator latency, or interface protocol across the three versions. The consistent pattern in Fig. 8—Verilog at higher frequency, HLS at higher resource cost—could therefore be caused by differential author effort or by different algorithmic mappings rather than by intrinsic tool capability. Because every quantitative conclusion in the paper is filtered through this uncontrolled comparison, the empirical support for the 'fairer comparisons' part of the central claim is not established. The framework itself may still be useful, but the current evidence does not support the tool-level divergences as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a bottom-up methodology for comparing hardware design flows (Verilog, Chisel, Vivado HLS) on HPC kernels. It decomposes GEMM, FFT, and QR factorization into primitive building blocks (MAC arrays, butterfly units, permutation units, pointwise operations, reduction trees), hand-implements each primitive and algorithm in the three environments, and uses an automated Vivado-based loop to search for a reported 'achievable frequency' and to record resource utilization. The authors report synthesizing approximately 1,500 configurations and present a representative subset in Figure 8, concluding that the methodology identifies divergence points in code generation, resource usage, and performance. The paper explicitly frames the main contribution as the methodology itself, with the concrete results serving as a proof of concept.","tokens_in":12330,"tokens_out":4211,"duration_ms":49806,"significance":"If the methodology were rigorously validated, it would be a useful complement to existing end-to-end tool comparisons by isolating where high-level tools diverge from hand-written RTL. The hierarchical decomposition into reusable primitives is sensible, and the automated exploration harness (Slurm/shell support, JSON configuration, interactive plotting) is a practical infrastructure contribution. The authors are candid that the autotuning constants are arbitrary and that the quantitative results are only a proof of concept. However, the current empirical support is too thin and uncontrolled to sustain the tool-level divergence claims: the comparison is only as informative as the three hand-written implementations being compared, and the reported metrics lack variance information and verification evidence.","major_comments":[{"comment":"The comparisons treat the three hand-written implementations as representative of their respective tools, but no control exists for implementation quality, author effort, or microarchitecture equivalence. Section III-B states: 'we have focused on efficient Verilog implementations, we provided a similar Chisel implementation and finally we have made use of the features offered by HLS.' There is no third-party baseline, no optimization-maturity metric, and no standardization of pipeline depth, operator latency, or interface protocol. The consistent pattern in Fig. 8 (Verilog at higher frequency, HLS at higher LUT count) could therefore reflect differential designer skill or different algorithmic mappings rather than intrinsic tool capability. This load-bearing issue undermines the 'fairer comparisons' and 'divergence points' claims. The authors should either add controls (e.g., a public ba","section":"§III-B, §IV-B"},{"comment":"The quantitative support is weakened by the absence of error bars or run-to-run variance. The autotuner in Eq. (12) injects stochastic offsets U(0, F) and U(-α, α), yet each reported point appears to come from a single run. Vivado synthesis and implementation also exhibit run-to-run variability, so the observed differences between tools and configurations may be within the noise of the stochastic search. Additionally, 'a representative subset' of the ~1,500 implementations is shown without defining the selection criteria. Please report multiple trials per configuration, provide variance statistics, and describe how the subset was chosen.","section":"§IV-A, Fig. 8"},{"comment":"For the QR factorization, the results are explicitly limited to the smallest configuration: 'Due to delays in generating the larger Verilog and Chisel designs, we only report results for the smallest configuration.' Contributions list an 'in-depth analysis' for the QR algorithm, but the evidence is a single point per tool. This is insufficient to support any cross-configuration or tool-level conclusion for QR. The authors should either complete the larger QR runs or scale back the QR claims to a single-case demonstration.","section":"§IV-B, Fig. 8(f)"},{"comment":"The stability stopping rule is printed as min_{j∈[i-W,i]} f_a[j] / min_{j∈[i-W,i]} f_a[j] < R. As written, the numerator and denominator are identical, so the ratio is always 1 and the criterion is vacuous for any R ≤ 1 (and trivially always true for R > 1). The intended ratio is presumably max/min over the window. Because this rule is part of the automated method that generates all frequency results, the typo needs correction, and the chosen W and R values should be reported.","section":"§III-B, stopping criterion 3"},{"comment":"The paper asserts functional correctness ('our implementations function correctly across the three languages and varying streaming widths') but does not describe any testbench, simulation, or equivalence-checking methodology. Resource and frequency numbers for incorrect or partially correct designs would be meaningless. Please describe the verification flow: test vectors, comparison against a software model, coverage of streaming widths and algorithms, and whether all ~1,500 configurations were verified.","section":"§IV-B"}],"minor_comments":[{"comment":"The units for f and WNS are not stated. The formula f_a = f/(1 - wns·f) is dimensionally sensitive: if f is in MHz and wns in ns, a factor of 1000 is missing. Specify the units and verify the formula dimensionally.","section":"Eq. (11)"},{"comment":"Typo: 'LU Ts' should be 'LUTs'.","section":"Fig. 8 caption"},{"comment":"The sentence 'we focus on the streaming Fourier transform []' contains an empty citation placeholder.","section":"§III-A (DFT)"},{"comment":"Typo: 'HPC e kernels' should be 'HPC kernels'.","section":"§V"},{"comment":"The SystemVerilog snippet shows 'generate' but not the matching 'endgenerate'; the ellipsis placement is confusing. Consider showing complete synthesizable code or indicating omitted lines more clearly.","section":"Fig. 2(a)"},{"comment":"The paper says 'approximately 1500 implementations' here and 'nearly 1,500 designs' in the conclusion; the exact count and the configuration grid (which parameters were swept, how many per benchmark) should be stated precisely.","section":"§IV-A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like an extended abstract or workshop paper. The decomposition idea and automation tool are useful, but the empirical evidence is not yet at the level required for a journal publication. The biggest risk is the uncontrolled comparison: without standardization of implementation quality or a third-party baseline, the central tool-divergence claims are not defensible. The proof-of-concept framing mitigates this somewhat, but the authors should either add controls or explicitly confine their conclusions to the specific hand-written designs. The missing verification description and the vacuous stopping-rule typo also need to be addressed before the methodology can be reproduced by others."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look, but read the results as a proof of concept, not as measurement. The decomposition of FFT, GEMM, and QR into MACs, butterflies, and permutation units is clean, and the automated WNS-driven frequency search is a sensible, reusable piece of tooling. That combination is genuinely new relative to the end-to-end comparisons in [25]–[28], and the paper is honest that the main value is the framework, not which language wins. Credit where due: the decomposition choices are reasonable, the automation story (Slurm/shell, CSV, interactive plots) is practical, and the reported scale (~1500 runs) suggests real engineering effort.\n\nThe soft spots are real and mostly the same ones you flagged. The biggest is the uncontrolled implementation quality. Section III-B literally says they focused on efficient Verilog, provided a similar Chisel implementation, and used HLS features. That asymmetry means the Verilog-versus-Chisel/HLS gaps in Figure 8 may reflect author effort or algorithmic mapping rather than tool capability. The stress-test note lands. It doesn't kill the methodology, but it does mean the 'fairer comparisons' claim is not currently supported by the data.\n\nThere are also mechanical defects that should be fixed before this is citable. The stability stopping rule in Section III-B has a typo or worse: it prints min(fa[j]) / min(fa[j]) < R, which is identically 1, so for any R≠1 the condition is either always true or always false. That needs correction. There is an empty citation ([] for the streaming Fourier transform), and the QR results cover only the smallest configuration. No error bars, no run-to-run variance, functional correctness asserted without a described testbench, and no artifact link despite the promise of a GitHub repo. Those are all addressable, but they currently bound how much quantitative weight the paper can carry.\n\nI don't think the central methodology is circular or broken; the frequency metric is defined from measured slack, and the arbitrary constants in the autotuner are a transparency issue, not a fitting trick. But the paper needs another pass: fix the stopping rule, ship the artifacts, add at least one third-party or independently-written implementation as a control, and either soften or re-scope the divergence claims. As-is, it is a solid workshop-to-conference paper, not a finished benchmark study.\n\nRecommendation: send to peer review. A serious referee can push on the implementation-quality control and the missing artifacts, and the methodology itself deserves discussion.","headline":"A genuine methodology contribution—primitive-level decomposition plus automated achievable-frequency search—whose quantitative evidence is too uncontrolled and incomplete to support the tool-divergence claims as stated.","tokens_in":12819,"tokens_out":1726,"would_cite":false,"duration_ms":21968,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A bottom-up decomposition of FFT, GEMM, and QR into primitives, implemented in Verilog, Chisel, and HLS, reveals where the tools diverge in achievable frequency and resource use.","keywords":["hardware design languages","high-level synthesis","Chisel","Verilog","FPGA","frequency autotuning","kernel decomposition","divergence analysis"],"falsifier":"A third-party replication in which each primitive and algorithm is implemented in Verilog, Chisel, and HLS by different engineers given equal time, targeting the same FPGA and using the same autotuner, should reproduce the main ordering: Verilog at the highest frequency, HLS at the highest resource cost for wide streaming. If the ordering flips or disappears under matched effort, the divergences are implementation artifacts rather than tool-level properties.","tokens_in":11998,"feed_emoji":"🧩","tokens_out":3821,"duration_ms":46766,"temperature":0.7,"pith_summary":"The paper is trying to establish that end-to-end benchmark comparisons of hardware design tools are too coarse, and that a better way is to decompose mathematical kernels into reusable primitives, implement each primitive in Verilog, Chisel, and Vivado HLS, and then automatically search for the highest achievable frequency at every level. If correct, designers can pinpoint whether an inefficiency comes from a single building block or from how blocks are assembled, and tool developers can see exactly where code generation diverges. The paper applies this methodology to FFT, GEMM, and QR factorization and reports concrete divergence patterns, but its central claim is the methodology itself, not a winner among the tools.","feed_headline":"Decompose kernels to expose real hardware-tool gaps","feed_subtitle":"Splitting FFT, GEMM, and QR into primitives reveals where Verilog, Chisel, and HLS differ in speed and resources.","key_machinery":"The central mechanism is the pairing of primitive-level decomposition with an automated achievable-frequency autotuner. The autotuner iteratively sets a target frequency, runs synthesis and implementation, reads the worst negative slack (WNS), and computes the achievable frequency as fa[i] = f[i] / (1 - WNS[i] * f[i]), then stochastically picks the next target frequency. This loop produces a frequency-resource frontier for each design at each decomposition level, turning tool comparison into a searchable design-space exploration rather than a single end-to-end benchmark.","core_discovery":"The paper claims that a hierarchical decomposition of kernels into primitives—multiply-accumulate arrays, butterfly units, permutation units, pointwise operations, reduction trees, and AXPY operations—combined with an automated frequency-search loop, produces fairer and more granular comparisons among hardware design tools than the full-application benchmarks found in prior work. It reports that for small streaming widths all three implementations consume similar resources, with Verilog consistently reaching higher frequencies; as streaming width grows, HLS requires progressively more resources in compute-heavy primitives and in GEMM and FFT assemblies, while the QR factorization converges a","pith_inferences":["An extension beyond the paper: a controlled study that matches optimization effort across tools—or uses independently written designs from multiple engineers—would separate tool capability from implementer skill, which the present comparison does not do.","The primitive set could evolve into a de facto microbenchmark suite for hardware generation tools, similar to compiler test suites, if the decomposition is standardized across more kernels and tools.","The stochastic frequency-search loop could be paired with power, flip-flop, BRAM, or different FPGA families to test whether the observed divergence ordering holds beyond the single Alveo U280 target.","The QR result, where HLS detects redundant dot products, hints that data-path reuse is a key divergence point; the paper reports it but does not generalize it into a design rule."],"forward_implications":["Designers can attribute a performance or resource gap to a specific primitive rather than to the whole application, making optimization effort more targeted.","Tool developers get a direct view of which constructs—such as permutation units or streaming interfaces—cause divergence in frequency or LUT usage.","The same primitive decomposition can be applied to other kernels, since the building blocks (MAC, butterfly, permutation, pointwise, reduction, AXPY) are reusable across many HPC computations.","Automated achievable-frequency search removes much of the manual timing-closure burden, making wide design-space exploration practical on clusters or single servers.","Because comparisons happen at every level, the methodology is more likely to expose tool-specific strengths than a single end-to-end number."],"fun_headline_variants":["Kernel primitives lay bare hardware-tool divergences","Decompose kernels to expose real hardware-tool gaps","Splitting math kernels uncovers design-tool tradeoffs","Frequency-resource analysis from kernel primitives","Tool divergence pinpointed via kernel decomposition"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the three hand-written designs are equally mature and comparable representatives of their tools; the paper states the implementations were made by hand with efficient Verilog as the focus, but it does not control for author effort or optimization quality, so the observed divergences could reflect implementer skill rather than tool capability.","fun_headline_variants_meta":{"raw":{"variants":["Kernel primitives lay bare hardware-tool divergences","Decompose kernels to expose real hardware-tool gaps","Splitting math kernels uncovers design-tool tradeoffs","Frequency-resource analysis from kernel primitives","Tool divergence pinpointed via kernel decomposition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000713,"raw_usage":{"total_tokens":3024,"prompt_tokens":705,"completion_tokens":2319,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":2244}},"tokens_in":449,"tokens_out":2319,"duration_ms":20114,"temperature":1.0,"reasoning_tokens":2244,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T18:43:47.754765+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A third-party replication in which each primitive and algorithm is implemented in Verilog, Chisel, and HLS by different engineers given equal time, targeting the same FPGA and using the same autotuner, should reproduce the main ordering: Verilog at the highest frequency, HLS at the highest resource cost for wide streaming. If the ordering flips or disappears under matched effort, the divergences are implementation artifacts rather than tool-level properties.","supporting_citations":[],"review_version":1}