{"id":"fef8f83c-3f0c-436a-b526-e5dad70d1485","arxiv_id":"2608.06791","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An expert-guided LLM framework translates C/C++ kernels into HLS FPGA designs, outperforming ChatHLS on PolyBench with a 4.24x geometric-mean speedup and valid RTL designs for all 14 attempted kernels.","lead":"HLSmith is a new AI framework that turns ordinary C/C++ programs into optimized FPGA hardware designs with more success than earlier LLM-based tools. A smart generalist might read it because it claims LLM agents can now produce working hardware accelerators with speedups, a task that usually needs a human hardware expert.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ChatHLS comparison may be confounded: HLSmith is given the gold reference HLS implementation, and the paper never states it is withheld from the LLM agents.","rationale":"The paper's central claim is comparative: HLSmith outperforms ChatHLS because of guarded recipes and a staged flow. For that claim to hold, both systems must receive the same inputs and neither may see the gold solution. Section III-A explicitly lists the reference HLS implementation as an input to HLSmith, and no later passage states that it is isolated from the LLM agents. If it is exposed, the system has an oracle that can be copied or adapted, which would inflate both the validity rate and the speedup while making the mechanism attribution unsupported. The reader's weakest assumption focused on the absence of independent RTL replay, which is a real reproducibility gap but not the only path to invalidation. The reference-input ambiguity is more fundamental: even with perfect replay, the comparison would be unfair if one system saw the solution. The within-model controls (expertise on/off, flash/multi-step) provide some evidence for the mechanism, so the paper is not fatally flawed, but the central claim cannot be accepted until the reference's role is clarified and the comparison is independently replayable. The appropriate verdict remains conditional, matching the reader's decision, though for a different primary reason.","tokens_in":19511,"tokens_out":12908,"duration_ms":100069,"concrete_test":"Examine the released HLSmith code and prompt templates (once the repository becomes available) to determine whether the kernel-specific reference HLS implementation is included in any LLM context, such as Translator or Optimizer prompts, few-shot examples, or validation feedback. If it is, rerun the 14-kernel ChatHLS comparison with the reference withheld from all HLSmith agents and compare the validity rate and geometric-mean speedup against the reported 14/14 and 6.91x/4.24x figures.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-A states that HLSmith receives 'a corresponding reference HLS implementation used as the gold reference' as part of its input, and Section V-B reports the 14-kernel ChatHLS comparison without stating whether this reference is withheld from the LLM agents. If the reference HLS code appears in any Translator or Optimizer prompt, few-shot context, or validation feedback visible to the model, then HLSmith is effectively handed the gold solution for each kernel, while ChatHLS is not. That would trivially explain the 14/14 validity rate and the large geometric-mean speedups, and would undercut the guarded-recipe/staged-flow attribution. Table IV and Figure 8 admit that the RTL traces are not published, so the comparison cannot be independently replayed; this makes the reference-input ambiguity even harder to resolve post hoc. The concern is load-bearing because, unlike the reproducibility gap alone, it would invalidate the central claim even if every reported RTL cycle count and co-simulation pass is accurate.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HLSmith, an agentic framework that translates plain C/C++ programs into optimized HLS designs. It combines a library of guarded transformation recipes distilled from expert HLS code, a staged and feedback-driven orchestration flow (flash vs. multi-step), and a tool-grounded model-adaptation pipeline using SFT and GRPO. The authors evaluate HLSmith on HLSFactory-28 PolyBench kernels, comparing against ChatHLS and reporting a 14/14 RTL-valid design rate versus ChatHLS's 8/14, with geometric-mean speedups of 6.91x over eight common passes (4.24x on a seven-kernel sensitivity set). Additional experiments measure the effect of expertise policies, a setup router, and fine-tuned open-weight models.","tokens_in":19730,"tokens_out":4913,"duration_ms":41339,"significance":"If the central comparison is valid, this is a potentially significant advance: it separates reusable, guarded HLS expertise from workflow control and shows that structured, prerequisite-aware optimization can outperform a prior state-of-the-art agentic framework. The paper has several genuine strengths: matched within-model comparisons, explicit disclosure of which evidence rows include golden-output or RTL checks, and exclusion of evaluation kernels from model-adaptation training data. However, the two largest claims—14/14 validity and 6.91x speedup over ChatHLS—currently rest on (a) an unspecified role for the gold reference HLS implementation in the agentic loop, and (b) self-supplied RTL cycle counts with no independent output replay. These issues are load-bearing for the headline result and must be resolved before the contribution can be fully assessed.","major_comments":[{"comment":"The manuscript lists 'a corresponding reference HLS implementation used as the gold reference' as an input to HLSmith (Section III-A) but never states that this reference is withheld from the LLM agents. Because the ChatHLS comparison in Section V-B and Figure 8 is the basis for the 14/14 validity and 6.91x geometric-mean speedup claims, the possibility that the reference HLS code appears in Translator/Optimizer prompts, few-shot context, or validation feedback would make the comparison trivially explainable and would invalidate the guarded-recipe/staged-flow attribution. Please state explicitly whether the reference HLS implementation is visible to any agent during translation or optimization; if it is visible, repeat the ChatHLS comparison with the reference withheld.","section":"Section III-A; Section V-B; Figure 8"},{"comment":"The RTL-cycle and output-validity evidence for the ChatHLS comparison is self-reported: Table IV labels the additional RTL rows 'Supplied RTL cycles; no independent output replay,' and the Figure 8 caption states that 'The CSVs omit generated RTL and output traces, preventing independent value replay.' The headline numbers (14/14 versus 8/14 validity, 6.91x speedup) therefore rest entirely on numbers supplied by the authors. This is a load-bearing reproducibility gap; please release the generated RTL, co-simulation logs, and output traces, or run an independent replay of the comparison.","section":"Table IV; Figure 8 caption"},{"comment":"The teacher-sweep rows in Table IV are labeled 'no golden-output or RTL check,' yet Table V and Figure 6(a-c) present large benchmark-baseline speedups (17.46x, 14.51x, 11.39x) from these same sweeps. These numbers are HLS synthesis-cycle estimates without independent output verification. The paper should clearly separate claims that are backed by CPU-golden C simulation and RTL co-simulation from claims that are only synthesis estimates, so that the abstract's 'functionally correct designs' statement is not implicitly attributed to unverified synthesis results.","section":"Table IV; Table V; Figure 6"}],"minor_comments":[{"comment":"The abstract reports 'a geometric mean speedup of 4.24x over ChatHLS,' but Section VI reports 6.91x over the eight common passes and 4.24x on the seven-kernel sensitivity set. Please reconcile or qualify the headline number so the abstract matches the detailed results.","section":"Abstract; Section VI"},{"comment":"There is a 'Table??' placeholder in the text describing the expertise tuple in Eq. (2). Please insert the intended table or reference.","section":"Section IV-B"},{"comment":"Several sentences are garbled, e.g., 'excluding it gives 4.24×on the other The 6 HLSmith-only passes improve recorded solve rate' and 'and and beating expertise off cases.' Please proofread and rewrite these passages.","section":"Section V-B"},{"comment":"The caption contains the typo 'HLS-Factgory'; it should be 'HLSFactory.'","section":"Figure 4 caption"},{"comment":"The manuscript itself notes that 'The reported 2.02× median ratio compares different C-simulation-passing subsets and is neither paired nor a geometric mean.' Given this disclosure, the later conclusion that small models 'achieve decent optimization' should not lean on this unpaired result; consider removing it or clearly marking it as preliminary.","section":"Section V-D"}],"recommendation":"major_revision","confidential_remarks":"The key gate for this paper is disclosure of whether the gold reference HLS implementation is visible to the HLSmith agents. The paper is otherwise careful about labeling evidence boundaries, but the current wording in Section III-A genuinely leaves open the possibility that the reference solution is provided to the model, which would invalidate the ChatHLS comparison. I would ask the authors to resolve this before any further editorial decision, and to provide the RTL traces or an independent replay for the central comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core contribution is more solid than the packaging. Encoding HLS optimization as guarded, ordered recipes with a bottleneck-conditioned router is a genuinely useful framing, and the paper earns credit for isolating the expertise effect with matched within-model comparisons. The fine-tuning results, especially the GRPO diagnostic with the 7B model, are carefully set up and honestly labeled. If the framework works as described, it could make LLM-based HLS agents substantially more credible.\n\nNow the soft spots, in proportion. The biggest one is the gold-reference ambiguity. Section III-A lists a \"reference HLS implementation used as the gold reference\" as part of the input, and nowhere does the paper state that this reference is withheld from the Translator or Optimizer agents. The gold reference is also the baseline for the speedup ratios, so it is definitely present in the pipeline. If it leaks into any prompt or few-shot context, the 14/14 validity and 6.9x speedup over ChatHLS become trivial to explain. The paper needs an explicit sentence, ideally a controlled experiment, showing the reference is not visible to the model.\n\nThe second issue is independent verification. Figure 8's caption admits the CSVs omit RTL and output traces, and Table IV says \"Supplied RTL cycles; no independent output replay.\" For the strongest claim in the abstract, that is a real gap. I don't doubt the authors measured what they say, but \"supplied\" is not the same as \"reproducible,\" and the artifact is not public.\n\nMinor points: the unresolved \"Table??\" in Section IV-B is sloppy but fixable; the ChatHLS comparison covers only 14 kernels, and syr2k is influential enough that the 4.24x vs 6.91x numbers depend on whether you include it.\n\nThe citation pattern looks fair, and the paper is properly cautious about what is a search ceiling versus a measured result. The authors also disclose limitations in the evaluation matrix, which I appreciate.\n\nWho is this for? Anyone working on LLM-driven hardware design or HLS automation. It deserves a serious referee: the mechanism is novel enough, the experiments are thoughtfully designed, and the flaws are fixable in revision. I'd want the authors to clarify the gold-reference handling, release the RTL traces, and note the 14-kernel limitation in the abstract. My recommendation: send it out, but flag the gold-reference question to the reviewers before they start.","headline":"The guarded-recipe idea is real and the within-model ablations are well done, but the headline ChatHLS numbers rest on evidence that can't be replayed, and the paper never says the gold HLS reference is kept out of the model's prompts.","tokens_in":20256,"tokens_out":1909,"would_cite":false,"duration_ms":18305,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HLSmith claims that guiding LLM agents with guarded, ordered HLS optimization recipes and staged compiler feedback converts plain C/C++ into valid, faster FPGA accelerators, completing all 14 evaluated kernels versus 57% for the leading…","keywords":["high-level synthesis","LLM agents","FPGA accelerators","C-to-HLS translation","guarded optimization recipes","synthesis feedback","model fine-tuning","PolyBench"],"falsifier":"Re-run the 14-kernel comparison with identical tool version, device, and CPU-golden outputs, replaying RTL co-simulation for every candidate and comparing executed cycle counts from the logs; if HLSmith's valid count drops below 14, or the geometric-mean RTL speedup over the baseline differs materially from 6.91x (4.24x on the sensitivity set), the paper's central claim is refuted.","tokens_in":19347,"feed_emoji":"⚙️","tokens_out":9057,"duration_ms":69917,"temperature":0.7,"pith_summary":"This paper argues that LLMs fail to translate plain C/C++ into high-performance FPGA hardware not because they cannot code, but because they lack the hardware architect's procedural know-how: which transformation to apply, in what order, and under what preconditions. To supply that know-how, it builds HLSmith, a framework whose core is a library of guarded recipes, each keyed to a diagnosed bottleneck, listing structural prerequisites, ordered code changes, unsafe cases to avoid, and expected outcomes. An orchestrator runs a staged loop: synthesize the baseline, read the tool's report, diagnose the bottleneck, route a matching recipe, validate the edit by simulation and synthesis, and keep the best valid design. On a fixed 28-kernel polyhedral benchmark suite, the paper reports that HLSmith beats the prior agent-orchestration baseline on every common kernel, with 14 of 14 RTL-valid designs against 8 of 14, and geometric-mean speedups of 6.91x over the eight common passes (4.24x on a seven-kernel sensitivity set). The reason to care: if the recipe-plus-feedback mechanism holds up, LLM-based hardware design shifts from hoping the model has hardware intuition to engineering that intuition as reusable, checkable procedure.","feed_headline":"LLM HLS agent: 14/14 valid RTL designs, 6.9x speedup over baseline","feed_subtitle":"Guarded, ordered optimization recipes plus synthesis feedback beat a leading prior agent framework on every common kernel.","key_machinery":"The load-bearing object is the guarded recipe tuple $s = \\langle b, p, a, q, g, e \\rangle$, along with the orchestration loop that only applies a recipe when its bottleneck matches the tool's diagnosis and its preconditions hold in the current code. The tuple is the interface between distilled HLS expertise and the LLM agents: it turns 'know how to optimize HLS' into a routable, checkable procedure, and the staged loop—baseline synthesis, diagnosis, recipe application, validation, keep-or-restore—enforces transformation order and prevents the model from wandering. A second mechanism is the early setup router, which ranks workflow-depth and expertise-policy combinations from the plain source and initial report, and cuts the search space from ten setups to five while staying within 5% of the exhaustive optimum on 18 of 19 kernels.","core_discovery":"The central discovery is that C/C++-to-HLS optimization can be decomposed into bottleneck-conditioned, guarded transformation recipes and driven by a staged, compiler-feedback loop, and that this decomposition is what lets LLM agents produce functionally correct, faster accelerators. Each recipe is a tuple $s = \\langle b, p, a, q, g, e \\rangle$: a bottleneck $b$ found in the synthesis report, structural prerequisites $p$, the architectural action $a$, an ordered checklist $q$, unsafe conditions $g$ to avoid, and expected results $e$. The Action Router admits a recipe only when $b$ matches the current diagnosis, $p$ holds in the current code, and no guard $g$ is triggered; the Optimizer applies the ordered steps; the HLS tool's reports are checked against $e$; and the orchestrator keeps the best valid design, reverting on failure. The paper's evidence is the comparison against the leading prior agent-orchestration framework on the polyhedral benchmark suite: HLSmith produces RTL-valid designs on all 14 kernels it attempts, versus 8 for the baseline, and its designs use fewer executed RTL cycles on every one of the eight common passes, with a geometric-mean ratio of 6.91x (4.24x excluding one influential kernel). The same workflow, when used to record trajectories from commercial frontier models, fine-tunes open-weight models down to 7B parameters, with reported peak per-kernel synthesis gains of 3.61x.","pith_inferences":["We infer that the guarded-recipe representation could be made even stronger by machine-checking the preconditions and guards with a static analyzer, so that LLM judgment is only needed for the final code edit, not for deciding applicability.","We infer that the recipe tuple is a form of searchable expert knowledge that could bootstrap itself: successful trajectories from one set of kernels could be mined into new library entries, growing coverage beyond the initial expert-derived recipes.","The paper's own inspection notes that the fine-tuned 7B model's improvements come from simpler transformations such as loop fission, pipelining, and unrolling, and not from tiling, double buffering, or dataflow; we infer that reward tuning currently under-exploits the architectural restructuring the library encodes, so combining GRPO with explicit recipe supervision may yield further gains.","Because the RTL traces and output values are not released, we infer that an independent reproducibility check is the first test the community should run before relying on the speedup ratios."],"forward_implications":["If the claims hold, LLM-based HLS optimization no longer depends on the model's implicit hardware intuition: explicit guarded recipes plus synthesis feedback can steer agents to valid, faster designs, which is a reusable recipe for other hardware-generation tasks.","The 14/14 RTL-valid rate versus 8/14 implies the staged recipe flow improves correctness and performance together, not one at the cost of the other, in this evaluation.","The 5-of-10 setup router implies roughly half the workflow configurations can be pruned up front on the basis of source code and the initial HLS report while staying within 5% of the exhaustive best on most kernels, lowering compute cost.","The fine-tuning results imply open-weight models as small as 7B can absorb validated optimization behavior from commercial teacher trajectories, lowering the cost and privacy barrier of using front-tier API models for HLS.","The up-to-252x speedups, if representative, imply that the gap between 'compiles and simulates' and 'actually fast' in HLS is bridgeable by structured architectural restructuring rather than pragma tuning alone."],"supporting_citations":[{"why":"Supplies the prior agent-orchestration baseline whose validity rate and RTL cycles define the comparison.","marker":"[13]"},{"why":"Provides the staged human-optimized HLS kernel examples from which the recipe library is distilled.","marker":"[6]"},{"why":"Supplies the fixed 28-kernel benchmark construction flow used for evaluation.","marker":"[2]"},{"why":"Defines the underlying polyhedral benchmark kernel suite.","marker":"[17]"},{"why":"Provides the group-relative policy optimization algorithm used for open-model reward tuning.","marker":"[21]"},{"why":"Provides the quantized efficient fine-tuning method used to train adapters.","marker":"[9]"},{"why":"Supplies the bootstrap method used for paired confidence intervals on speedups.","marker":"[11]"},{"why":"Is the open-weight model used in the sweep and fine-tuning studies.","marker":"[20]"}],"fun_headline_variants":["Guarded HLS recipes turn LLM agents into 14/14 valid accelerators","LLM HLS agent: guarded recipes + feedback beat prior framework 6.9x","Expert-guided HLS framework: every kernel valid, 6.9x speedup","HLSmith: bottleneck-conditioned recipes for correct, faster FPGA designs","From C/C++ to HLS: agentic framework hits 100% valid designs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers rest on RTL cycle counts and co-simulation pass/fail results that the authors supply without releasing generated RTL or output traces for independent value replay, so the core comparison assumes those reported measurements are accurate and comparable to the baseline's.","fun_headline_variants_meta":{"raw":{"variants":["Guarded HLS recipes turn LLM agents into 14/14 valid accelerators","LLM HLS agent: guarded recipes + feedback beat prior framework 6.9x","Expert-guided HLS framework: every kernel valid, 6.9x speedup","HLSmith: bottleneck-conditioned recipes for correct, faster FPGA designs","From C/C++ to HLS: agentic framework hits 100% valid designs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1726,"prompt_tokens":1174,"completion_tokens":552,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":790,"completion_tokens_details":{"reasoning_tokens":444}},"tokens_in":790,"tokens_out":552,"duration_ms":4584,"temperature":1.0,"reasoning_tokens":444,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:29:35.361901+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 14-kernel comparison with identical tool version, device, and CPU-golden outputs, replaying RTL co-simulation for every candidate and comparing executed cycle counts from the logs; if HLSmith's valid count drops below 14, or the geometric-mean RTL speedup over the baseline differs materially from 6.91x (4.24x on the sensitivity set), the paper's central claim is refuted.","supporting_citations":[{"cited_title":"ChatHLS: Towards systematic design automation and optimization for high-level synthesis,","cited_arxiv_id":null,"evidence_quote":"Supplies the prior agent-orchestration baseline whose validity rate and RTL cycles define the comparison."},{"cited_title":"Understanding performance differences of FPGAs and GPUs,","cited_arxiv_id":null,"evidence_quote":"Provides the staged human-optimized HLS kernel examples from which the recipe library is distilled."},{"cited_title":"PolyBench/C: The polyhedral benchmark suite,","cited_arxiv_id":null,"evidence_quote":"Defines the underlying polyhedral benchmark kernel suite."},{"cited_title":"Efron and R","cited_arxiv_id":null,"evidence_quote":"Supplies the bootstrap method used for paired confidence intervals on speedups."}],"review_version":2}