{"id":"b9c1b887-5acb-4b3f-93f4-e4225615bdfb","arxiv_id":"2501.00642","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A context-plus-compiler-feedback agent lifts LLM code-generation success rates for hardware description languages that were absent from LLM training data.","lead":"This paper presents HDLAgent, a prompt-and-feedback framework that helps off-the-shelf large language models write code in hardware description languages they were never trained on, such as Chisel, PyRTL, and DSLX. The authors report that the framework lifts success rates from near zero to around 40-80% on small benchmark tasks, which could make it easier for hardware designers to adopt new HDLs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing uncertainty is that the few-shot examples and HDL descriptions were selected on the same benchmark used for the headline rates; Section 3.1 admits picking the description that performs best on these LLMs, so the reported 0-to-35/59 gains may be benchmark-specific rather than a…","rationale":"The central claim is plausible and the paper contains real evidence: ablations isolate Description/Few-shot/Compile/Fixes, results are shown across six LLMs and four HDLs, compiler grounding is iterated to convergence, and QoR is analyzed. The reason I would not reject or accept outright is that the generalization claim rests on the main context being a neutral language primer plus compiler feedback. Section 3.1 openly states that the HDL description was chosen by performance with the evaluation LLMs, and Section 5.2 confirms a search over summarization prompts. Since the same HDLEval benchmark supplies the headline success rates, the possibility of test-set selection is concrete, not hypothetical. The few-shot categories also map onto the simple operations that dominate HDLBits-derived tasks, so the examples may function as task templates. This does not prove fraud or even that the reported numbers are wrong; it means the paper currently cannot support the broad \"enabling new HDLs\" interpretation. The numeric inconsistency in the conclusion (GPT-4 Verilog 34% to 72% vs Table 3's 72% base/76% agent) is real but secondary; it should be corrected. The verification step that would settle the issue is to release the exact prompts and evaluate on held-out, independently written tasks. Until that is done, the appropriate disposition is the reader's original CONDITIONAL verdict, not a stronger one.","tokens_in":16691,"tokens_out":8857,"duration_ms":79341,"concrete_test":"Release the exact main-context prompts (HDL descriptions, all few-shot snippets, prefix/suffix) and the full set of compiler-context sample fixes. Then run two checks: (1) hold out all HDLEval tasks whose required operation matches one of the five few-shot families (bit operations, reductions, loops, multiplexing, multiply-add) and recompute the success rates; if the reported gains concentrate in those families, the few-shot templates are doing task-specific work. (2) Build a fresh held-out set of 30 small tasks per HDL, written independently of HDLEval/HDLBits, and run HDLAgent with the released prompts unchanged; if the held-out success rate is materially lower (e.g., more than 10 points), the headline generalization claim needs to be reframed as benchmark-specific.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 states: \"we opt for simplicity by selecting the description that performs best with Mix-8x7B and GPT-3.5n.\" Since the headline numbers are computed on HDLEval, the same benchmark on which those descriptions were compared (see Figure 7), the main context is effectively fitted to the evaluation set. Section 5.2 shows the authors tried multiple summarization prompts and kept the best, and the paper does not report any held-out validation. The few-shot examples are not generic: they cover bit operations, reductions, loops, multiplexing, and a multiply-add block, which are the same simple operations that dominate HDLEval's HDLBits-derived tasks. The compiler-context \"sample fixes\" are also unspecified; if they were harvested from prior runs on the benchmark, that is a second leakage channel. For the central claim that HDLAgent enables LLMs to use new HDLs, what must be true is that the context transfers to unseen tasks. The manuscript does not currently demonstrate this. This is not an accusation; it is the minimal condition for external validity, and the paper's own disclosure of selection makes it the first thing to check.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes HDLAgent, an agentic prompting framework that wraps off-the-shelf LLMs to generate code in HDLs underrepresented in training data (Chisel, PyRTL, DSLX) as well as Verilog. The framework has a 'main context' consisting of an LLM-generated HDL summary, few-shot examples, and prompt prefix/suffix, and a 'compiler context' that iterates up to eight times on compiler errors, optionally with sample fixes. The evaluation uses HDLEval and VerilogEval across six LLMs and reports large improvements; for example, Table 3 shows GPT-3.5n on Chisel rising from 0/134 to 80/134 and Mix-8x7B on PyRTL rising from 0/134 to 48/134 on HDLEval-Comb. The paper also analyzes pass@k, iteration counts, token cost, execution time, QoR, and derives HDL-specific design recommendations.","tokens_in":16962,"tokens_out":8537,"duration_ms":80495,"significance":"If the reported gains hold outside the specific benchmark, HDLAgent would be a practical, fine-tuning-free method for making LLMs usable for niche HDLs, with implications for tool adoption and for HDL designers. The paper's strengths include breadth across LLMs and HDLs, an explicit component ablation, top@k and token-cost analysis, QoR measurements, and honest discussion of failure modes. The main caveat is that the main-context components were selected using the same HDLEval benchmark on which the headline numbers are reported; therefore external validity needs to be established before the central claim can be fully credited.","major_comments":[{"comment":"The paper states in Section 3.1 that 'we opt for simplicity by selecting the description that performs best with Mix-8x7B and GPT-3.5n,' and Section 5.2/Figure 7 ranks summarization prompts by accuracy on HDLEval. Because the headline gains in Table 3 and the abstract are computed on the same HDLEval data, the main-context prompt has effectively been selected on the evaluation set. The paper needs a genuine held-out evaluation (e.g., a development/test split of HDLEval made before prompt selection, or an independent set of tasks in each new HDL) before the central claim that HDLAgent transfers to unseen HDL programs is supported. Without this, the reported 0-to-60% improvements could be benchmark-specific rather than evidence of general enablement of new HDLs.","section":"Section 3.1, Section 5.2, Figure 7, Table 3"},{"comment":"The 'sample fix' mechanism is a load-bearing component: the Fixes rows in Figures 3-6 show substantial additional gains beyond the Compile step, and Section 3.2 says HDLAgent appends a sample fix when one exists. However, the paper never states where these sample fixes come from, how many are used, or whether they were harvested from the HDLEval problem set. This is a second potential leakage channel in addition to the main-context selection. Please specify the provenance of sample fixes, provide the actual fixes in an appendix, and confirm that they are generic and independent of the evaluation problems.","section":"Section 3.2, Figures 3-6"},{"comment":"The conclusion states that HDLAgent raised the Verilog success rate of GPT-4 from 34% to 72%. I could not locate a 34% baseline in the body: Table 3 reports GPT-4 with Verilog on HDLEval-Comb at 97/134 (72.4%) Base and 102/134 (76.1%) HDLAgent at k=1. If the 34% figure comes from a different benchmark or subset, it should be identified explicitly; otherwise the conclusion is inconsistent with the paper's own data.","section":"Conclusion, Section 5.1, Table 3"}],"minor_comments":[{"comment":"The subsection heading 'PyR TL' contains a typo; it should read 'PyRTL'.","section":"Section 5.1"},{"comment":"The code sample contains 'inpput' (misspelling of 'input'); if this is literal code, it may confuse readers trying to reproduce the example.","section":"Listing 2"},{"comment":"Table 3 entries appear to be numbers of tests passed out of 134 HDLEval-Comb tests, but the table does not state the denominator; please state this directly so readers can convert to percentages.","section":"Table 3"},{"comment":"The claim of over 90% success on concise examples is based on Figure 11, which uses GPT-4 only; please qualify the claim by LLM and by benchmark condition.","section":"Section 5.5, Figure 11"},{"comment":"The paper mentions a per-language filter/detector for removing English and finding code boundaries but gives only the Verilog example; please specify the filter rules for Chisel, PyRTL, and DSLX for reproducibility.","section":"Section 3.3"},{"comment":"Reference [14] (FinanceBench) does not appear to support the claim that placing the prompt after the context achieves better results; please provide an appropriate citation.","section":"References, Section 3.3"},{"comment":"The comparison between a full reference manual and a summary reports percentages (e.g., GPT-4 from 77% to 66%, Mix-8x7B from 59% to 33%) without tying them to a specific table or benchmark condition; please clarify the conditions for these numbers.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's main result is plausible, but the combination of selecting the main context on HDLEval and not yet providing full prompt/fix details makes it hard to certify transfer. The self-authored HDLEval benchmark is not inherently a problem; the issue is that it is used for both context selection and headline evaluation. I would be willing to move to accept after a held-out validation study and full disclosure of prompts, few-shot examples, and sample-fix provenance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The qualitative result here is solid: an agent wrapper built from standard pieces—LLM-generated language summary, few-shot examples, compiler feedback with sample fixes—takes off-the-shelf LLMs from near-zero to usable success rates on Chisel, PyRTL, and DSLX. The paper does a real service by ablating each component (Description, Few-shot, Compile, Fixes) across six LLMs and four HDLs, and the per-HDL failure analysis is thoughtful. The claim that this lowers the adoption barrier for new HDLs is plausible and worth taking seriously.\n\nThe biggest soft spot is the one the stress-test note flags: Section 3.1 admits the HDL description was chosen by picking the version that performed best on the evaluation LLMs, and Figure 7 shows prompts sorted by benchmark accuracy. Since the headline rates are computed on HDLEval, the same benchmark used for that selection, the reported gains are at least partly tuned to that test set. The paper does not report any held-out tasks or a separate validation split, so we do not actually know how the context transfers to unseen problems. This is not fatal—the summaries are generic language descriptions, not per-task memorization—but it is the main reason to be cautious about the exact numbers.\n\nOne correction to the reader's take: Table 3 reports test counts, not percentages. The abstract's zero-to-35% for PyRTL/Mix-8x7B and zero-to-59% for Chisel/GPT-3.5n match Table 3 (48/134 and 80/134 respectively). The conclusion's \"GPT-4 Verilog from 34% to 72%\" is still unexplained and needs a pointer to the right row, but it is not a contradiction of the body tables.\n\nThe other issues are minor in comparison: no code or prompts are released yet, the main benchmark is self-authored (though VerilogEval provides a partial external anchor, only for Verilog), and the few-shot examples cover the simple operations that dominate HDLEval, which again tilts the evaluation toward easy tasks. The paper should also disclose how the \"sample fixes\" for compiler errors were obtained; if they were harvested from prior runs on the benchmark, that is a second leakage channel.\n\nWho should read this: anyone working on LLM-assisted hardware design or on making LLMs usable for low-resource programming languages. It is an engineering contribution, not a deep theoretical one, but it is an honest and fairly thorough one for what it claims. With the selection procedure disclosed and a held-out task set added, it would be a nice data point. As-is, I would send it to a serious referee, but the referee should push hard on external validity.\n\nRecommendation: accept for review, with revision likely conditional on held-out validation and code release.","headline":"A credible low-resource-HDL coding agent with a genuinely informative ablation, but the language summaries were tuned on the same benchmark used for the headline numbers, so the gains are likely optimistic until held-out validation appears.","tokens_in":17484,"tokens_out":3040,"would_cite":true,"duration_ms":29282,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Wrapping an off-the-shelf LLM with a distilled HDL summary, few-shot examples, and iterative compiler feedback lifts its success on chip languages that were absent from training from near zero to 35-59%, and above 90% for small snippets.","keywords":["HDLAgent","large language models","hardware description languages","code generation","compiler feedback","few-shot learning","transfer learning","Chisel"],"falsifier":"Run HDLAgent on a brand-new HDL, or on held-out problems chosen after the context is fixed, using summaries and examples created without ever seeing the test suite; if success rates stay near the no-agent baseline for that language, the reported gains are benchmark-specific rather than evidence of transferable capability.","tokens_in":16486,"feed_emoji":"🤖","tokens_out":11021,"duration_ms":85354,"temperature":0.7,"pith_summary":"This paper argues that off-the-shelf large language models can be made productive for hardware description languages (HDLs) that were missing from their training data, without fine-tuning or retraining. The authors introduce HDLAgent, a prompt-and-iteration wrapper that supplies the model with a compact summary of the target HDL, a few short worked examples, and a loop that feeds compiler error messages back until the code compiles. On the HDLEval and VerilogEval benchmarks, the agent raised PyRTL success from 0% to 35% with Mixtral 8x7B and Chisel from 0% to 59% with GPT-3.5-turbo-0125, and it pushed success above 90% for small snippets across all tested HDLs. The authors read this as evidence that new HDLs can ride on existing LLM knowledge, lowering a barrier to adoption that would otherwise make LLMs a brake on hardware-language innovation.","feed_headline":"A compiler-feedback loop lifts LLM chip code from 0% to 59%","feed_subtitle":"The recipe works across six off-the-shelf models and four chip languages without any fine-tuning.","key_machinery":"The load-bearing object is the two-part context that HDLAgent feeds to the LLM at each step. The main context contains an LLM-generated summary of the target HDL, chosen to be concise and to emphasize syntax that differs from Verilog, a fixed set of few-shot examples for common constructs, and prefix and suffix statements that pin down input-output conventions and forbid English prose. The compiler context is the iterative loop: after a failed compile, the agent sends the whole previous code snippet with the compiler error message, plus an optional generic sample fix, and asks for a corrected program; a per-language filter strips non-code text and locates code boundaries. The paper attributes the gains to transfer learning in the LLM, with the summary acting as a bridge from Verilog knowledge to the new HDL.","core_discovery":"The central claim is that the gap between LLMs and emerging HDLs is not a training-data problem that only fine-tuning can fix; a generic agent recipe can close much of it at inference time. HDLAgent composes two memory blocks: a main context that distills the HDL's reference manual into a short description plus few-shot examples of tricky syntax such as bit operations, reductions, loops, multiplexing, and multiply-add blocks, and a compiler context that, on a compile failure, reintroduces the entire latest code snippet with the raw compiler error message and, when available, a sample fix. With up to eight iterations, this grounded loop moves success rates from near zero for all six tested LLMs on Chisel, PyRTL, and DSLX to a range where even the weakest model reaches 28% on PyRTL and 53% on Verilog, and where small examples exceed 90% on every HDL. The paper also reports that feeding the full reference manual is worse than a summary, and that for DSL-based HDLs like Chisel and PyRTL the combination of description plus compiler feedback matters more than either alone.","pith_inferences":["If the context-construction step is fully automated by having an LLM summarize any language manual with the same generic prompt, the recipe may transfer to non-hardware DSLs and other niche programming languages.","The reported gains are likely optimistic for a truly novel HDL, because the paper selects the summary that scores best on the evaluation LLMs; a blind test on a fresh HDL would be needed to confirm how much transfer is real.","The observation that summaries beat full manuals suggests that difference-focused documentation, not exhaustive references, is what limited-context models need, which could inform how new languages write their own onboarding guides.","Combining HDLAgent with a small fine-tuned repair model, in the spirit of existing debugger agents, is a natural next step toward closing the large-design gap the paper documents."],"forward_implications":["New HDLs can become usable by off-the-shelf LLMs immediately, without waiting for fine-tuning datasets or closed-model retraining cycles.","For snippets under roughly 25 lines of equivalent Verilog, HDLAgent reaches over 90% success on Chisel, PyRTL, DSLX, and Verilog across the tested LLMs, which the paper argues is enough for learners asking for small code examples.","Because success still collapses on modules beyond about 75 lines, the paper identifies large-design generation as the open problem that agent wrappers alone do not solve.","The authors propose concrete HDL and compiler design changes, such as clearer error messages that distinguish DSL syntax from host-language syntax and multi-output port conventions in DSLX, that would reduce the number of iterations HDLAgent needs."],"supporting_citations":[{"why":"Supplies the HDLEval benchmark used to measure success rates across Chisel, PyRTL, DSLX, and Verilog.","marker":"[44]"},{"why":"Provides the VerilogEval-Human and Machine test sets used as reference benchmarks for Verilog.","marker":"[20]"},{"why":"Prior work on using compiler errors to fix Verilog that the paper's sample-fix mechanism extends to multiple HDLs.","marker":"[33]"},{"why":"Related agent that grounds generation with testbench feedback; HDLAgent differs by omitting simulation and focusing on compile feedback.","marker":"[32]"},{"why":"The self-reflection technique that motivates the chained LLM calls in HDLAgent's iterative loop.","marker":"[36]"},{"why":"Shows that few-shot prompting can substitute for instruction alignment, a precedent for HDLAgent's few-shot examples avoiding fine-tuning.","marker":"[18]"},{"why":"Represents the fine-tuning alternative that HDLAgent deliberately avoids, grounding the paper's claim that its approach works without retraining.","marker":"[21]"}],"fun_headline_variants":["Agent loop lifts LLM HDL code from 0% to 59%","Compiler feedback boosts LLM chip coding without fine-tuning","Inference-time agent teaches LLMs new hardware languages","Six models, one agent: HDL success jumps from zero","Grounding LLMs with compiler errors improves chip design"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's transferable gains rest on the HDL summaries and example snippets being generic descriptions of each language, but the authors pick the summary that scores highest on the evaluation models, so if those summaries encode knowledge of the test problems, the reported increases will not generalize to an unseen HDL.","fun_headline_variants_meta":{"raw":{"variants":["Agent loop lifts LLM HDL code from 0% to 59%","Compiler feedback boosts LLM chip coding without fine-tuning","Inference-time agent teaches LLMs new hardware languages","Six models, one agent: HDL success jumps from zero","Grounding LLMs with compiler errors improves chip design"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1373,"prompt_tokens":860,"completion_tokens":513,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":431}},"tokens_in":476,"tokens_out":513,"duration_ms":5503,"temperature":1.0,"reasoning_tokens":431,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:45:19.537024+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HDLAgent on a brand-new HDL, or on held-out problems chosen after the context is fixed, using summaries and examples created without ever seeing the test suite; if success rates stay near the no-agent baseline for that language, the reported gains are benchmark-specific rather than evidence of transferable capability.","supporting_citations":[{"cited_title":"HDLEval Benchmarking LLMs for Multiple HDLs","cited_arxiv_id":null,"evidence_quote":"Supplies the HDLEval benchmark used to measure success rates across Chisel, PyRTL, DSLX, and Verilog."},{"cited_title":"Rtlfixer: Automatically fixing rtl syntax errors with large language models, 2024","cited_arxiv_id":null,"evidence_quote":"Prior work on using compiler errors to fix Verilog that the paper's sample-fix mechanism extends to multiple HDLs."},{"cited_title":"Autochip: Automating hdl generation using llm feedback, 2023","cited_arxiv_id":null,"evidence_quote":"Related agent that grounds generation with testbench feedback; HDLAgent differs by omitting simulation and focusing on compile feedback."},{"cited_title":"Rtlcoder: Outperforming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution, 2024","cited_arxiv_id":null,"evidence_quote":"Represents the fine-tuning alternative that HDLAgent deliberately avoids, grounding the paper's claim that its approach works without retraining."}],"review_version":1}