{"id":"8b563e11-43aa-4e45-9f05-637ac9893326","arxiv_id":"2505.19734","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ReChisel, an LLM agent with reflection and an escape mechanism for non-progress loops, significantly improves Chisel code generation success rates across five LLMs and three benchmarks.","lead":"ReChisel is a new LLM agent that writes Chisel hardware code, then uses compiler and simulation feedback to fix errors in a loop. In tests across five LLMs on 216 module-level problems, it lifts success rates from 26-45 percent baseline to 73-85 percent Pass@1, approaching Verilog-generation agents.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CommonErrorKnowledge provenance is undisclosed; if it was harvested from the 216 evaluation cases, ReChisel's reported gains partly reflect benchmark-specific leakage rather than general Chisel code-generation ability.","rationale":"The reader's weakest_assumption is the same as mine: the CommonErrorKnowledge injected into the Reviewer prompt is never sourced. I agree that this is the most load-bearing concern. I considered alternatives: benchmark filtering could inflate absolute numbers, the escape mechanism lacks an ablation, and no confidence intervals are reported. Those are real but secondary. Benchmark filtering affects the Verilog comparison in roughly the same way, and the zero-shot baselines are computed on the same filtered set, so relative gains remain meaningful. The escape mechanism is one component of the loop, but the paper does not quantify its contribution, and a missing ablation is less threatening than a potential test-set dependency. Lack of confidence intervals is a reporting issue, not a threat to the direction of the result. The CommonErrorKnowledge issue is different: it changes what the experiment measures. If the prompt was tuned on the test cases, the evaluation partially measures how well the prompt encodes seen errors rather than how well the reflection mechanism generalizes. Section IV-B's language—'We found that the types of syntax errors tend to be densely distributed' and Table II's 'examples of common errors made by the LLM'—implies the taxonomy came from observed LLM errors, plausibly from the same benchmark runs. Because ReChisel is open-sourced, provenance can be checked, but the paper should disclose it regardless. A held-out evaluation would settle whether the method generalizes. Therefore I keep the reader's CONDITIONAL verdict; no change to the recommendation is needed beyond making this condition explicit.","tokens_in":12817,"tokens_out":7294,"duration_ms":84395,"concrete_test":"Freeze the final Reviewer prompt (with CommonErrorKnowledge) and run ReChisel on a newly constructed held-out set of 40-60 Chisel module specifications that were not used in any development or evaluation step, using the same models (Claude 3.5 Sonnet and GPT-4o) and the same Pass@1 protocol (10 trajectories per case, max 10 iterations). Compare held-out Pass@1 to the reported 84.98% and 77.46%; a drop of more than about 10 percentage points would indicate that the in-context error knowledge is benchmark-specific. As a secondary check, inspect the open-source repository for scripts that build CommonErrorKnowledge from the benchmark datasets to confirm provenance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-B states that ReChisel injects 'CommonErrorKnowledge'—pre-organized causes and fix guidance for common Chisel errors—into the Reviewer prompt, with examples in Table II. The paper never states where this taxonomy came from. The natural reading is that it was assembled from the authors' own error analyses, plausibly from the same baseline runs on the same three benchmarks (VerilogEval, HDLBits, RTLLM) used for evaluation. If the Reviewer prompt contains error patterns and fixes distilled from these 216 test cases, the reported success rates (e.g., Claude 3.5 Sonnet Pass@1 from 33.33% to 84.98% at n=10) measure the benefit of prompt-level information about the test distribution, not just the reflection and escape loop. The headline claim promises effective Chisel code generation generally and compares with Verilog agents; a benchmark-informed reviewer prompt transfers poorly to unseen specifications. No ablation removes CommonErrorKnowledge, and no held-out evaluation is reported, so the magnitude of this effect is unknown. This is the most load-bearing gap in an otherwise plausible system: the reflection loop itself is standard and the gains are consistent, but the prompt's provenance determines whether the headline numbers generalize.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ReChisel is an LLM-based agentic system for automatic Chisel code generation. It couples a Generator, Inspector, Reviewer, and external Compiler/Simulator in an iterative reflection loop: compiler and simulation feedback is summarized into a trace, the Reviewer produces a revision plan, and an escape mechanism discards detected non-progress loops. The system is evaluated on 216 module-level cases filtered from VerilogEval's Spec-to-RTL, AutoChip's HDLBits, and RTLLM, using five LLMs. The reported results show large gains over zero-shot baselines, e.g., Claude 3.5 Sonnet Pass@1 rising from 33.33% to 84.98% at ten iterations, and performance comparable to AutoChip on Verilog generation. The paper also presents a baseline study comparing Chisel and Verilog generation capabilities.","tokens_in":12972,"tokens_out":2534,"duration_ms":30838,"significance":"If the headline result holds, ReChisel is a meaningful contribution: it is one of the first agentic systems targeting Chisel, it is open-sourced, and its reflection loop is externally grounded in compiler and simulator feedback rather than being a purely self-referential prompting scheme. The reported improvements are consistent across five models and three benchmarks, which strengthens the plausibility of the central claim. However, the significance depends on whether the CommonErrorKnowledge injected into the Reviewer prompt was constructed independently of the evaluation cases and whether the ablation evidence supports the claimed contributions of the escape mechanism. The paper also provides a useful baseline comparison of Chisel versus Verilog LLM code generation, even though this part is not the main contribution.","major_comments":[{"comment":"The provenance of CommonErrorKnowledge is undisclosed and this is load-bearing for the reported gains. Section IV-B states that the authors 'pre-organized the causes and corresponding fix guidance for each error and included this information within the prompts,' but the paper never states whether this knowledge was derived from the same 216 evaluation cases used in Section V. If the error taxonomy and fixes were harvested from VerilogEval, HDLBits, and RTLLM, the Reviewer prompt contains information about the test distribution, and the reported Pass@1 gains (e.g., Claude 3.5 Sonnet from 33.33% to 84.98%) partly measure prompt-level leakage rather than the reflection loop alone. The paper needs either an explicit statement that CommonErrorKnowledge was constructed before and independently of the benchmark suite, or an ablation that removes it and a held-out evaluation on cases not used to build the taxonomy.","section":"IV-B, Table II"},{"comment":"The escape mechanism is never ablated. The paper motivates the mechanism with the non-progress loop example in Fig. 4 and claims it enables the system to break out of cycles, but no experiment compares ReChisel with and without the escape mechanism. Since the mechanism is presented as a central component alongside reflection, its contribution to the final success rates should be quantified. Without such an ablation, it is unclear whether the gains come from the reflective loop alone or from the escape component, and the paper's attribution of the improvement to both mechanisms is not supported by the reported results.","section":"IV-C and V-B"},{"comment":"The reported success rates lack confidence intervals and significance testing. Each case is tested ten times, but the paper reports only point estimates, and several comparisons are close (e.g., Claude 3.5 Sonnet versus Haiku at Pass@1 and n=10: 84.98% versus 84.51%). Without per-case variance or bootstrap confidence intervals, the reader cannot assess whether differences between models or between ReChisel and AutoChip are meaningful. This is particularly relevant to the claim that ReChisel achieves performance 'comparable' to Verilog agentic systems, since the comparison in Table IV shows differences of a few percentage points.","section":"V-A, V-B, Table III"},{"comment":"The filtering of the benchmark set is not characterized enough to rule out selection bias. The paper excludes cases that are incompatible with Chisel, have missing reference code, or are designed for debugging/completion, but it does not report how many cases were removed per benchmark or how the filtered subset compares with the original benchmarks on difficulty. If the filtering disproportionately removes hard cases, the absolute success rates on the 216-case set may overstate ReChisel's ability on full benchmark suites. Reporting per-benchmark filtered counts and baseline performance on both filtered and unfiltered sets would address this concern.","section":"V-A"}],"minor_comments":[{"comment":"The sentence 'Table I demonstrates the performance improvements achieved by ReChisel' appears to refer to Table III, since Table I reports baseline comparisons, not ReChisel performance. This should be corrected.","section":"V-B"},{"comment":"The stacked bar chart would benefit from a numeric breakdown per model and an explicit statement of whether the percentages are computed over all generated attempts or over a fixed number of samples; this would make the error-type analysis easier to interpret.","section":"Fig. 1"},{"comment":"The description of the escape mechanism's cycle detection says the Inspector compares 'the current feedback' with trace entries and checks whether errors occur at the same location with identical causes, but the exact prompt or algorithm used to determine 'identical causes' is not specified; a short pseudocode or formal condition would improve reproducibility.","section":"IV-C"},{"comment":"The evaluation reports temperature and top-p as default configurations, but the specific default values are not listed; since Pass@k is computed from ten samples per case, stating the sampling configuration would make the results more reproducible.","section":"V-A"}],"recommendation":"major_revision","confidential_remarks":"The main concern is the undisclosed origin of CommonErrorKnowledge. The reader's conditional verdict is appropriate: the reflection loop is externally grounded and the gains are consistent, but the headline numbers cannot be interpreted as measuring general Chisel code-generation ability until the provenance of the in-context error knowledge is clarified and an ablation is run. I would also encourage the editors to require confidence intervals, since several comparative claims rest on small point-estimate differences."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ReChisel wraps an LLM in a reflection loop with compiler/simulator feedback, adds an escape mechanism for non-progress loops, and targets Chisel code generation. The gains are large and consistent: Pass@1 rises from roughly 26–45% to 73–85% across five models on a filtered 216-case module-level benchmark. The comparison with AutoChip on the same benchmarks is a fair way to test whether Chisel can become a viable LLM target, and the answer is a credible yes. The escape mechanism is the one genuinely new piece; I don't see it in AutoChip or Reflexion.\n\nThe central claim holds up about as well as the evidence allows. The baseline gap between Chisel and Verilog is real, the reflection loop is externally grounded (compiler and simulator feedback), and the improvement is monotone with iterations. The code and data are open-sourced, which helps.\n\nThree soft spots, in proportion. First, the CommonErrorKnowledge in-context prompt. Section IV-B says the authors pre-organized causes and fix guidance and included it in the prompts, but they never say where that taxonomy came from. If it was mined from the same 216 test cases, part of the reported gain is prompt-level overfitting to the test distribution, not general Chisel ability. The examples in Table II look generic — misspellings, missing IO wrappers, type mismatches — so the risk is moderate, not fatal, but it's not zero. An ablation with and without that prompt would settle it, and none is reported. Second, the escape mechanism isn't ablated, so its contribution is unknown. Third, the statistical reporting is thin: no confidence intervals, no significance tests, and the filtering from the original benchmarks to 216 cases isn't quantified.\n\nThis paper deserves a serious referee. I'd send it to review with a request for three additions: an ablation of CommonErrorKnowledge, an ablation of the escape mechanism, and explicit disclosure of how the prompt content was constructed. If those come back clean, it's a solid addition to the LLM-for-HDL line. If the CommonErrorKnowledge was harvested from the test set, the reported gains will shrink on unseen specs, and the paper should be reframed as a benchmark-specific prompt-tuning result.","headline":"ReChisel's reflection loop is a plausible and useful contribution; the undisclosed CommonErrorKnowledge prompt provenance is the main thing I'd want clarified before trusting the headline numbers.","tokens_in":13563,"tokens_out":3884,"would_cite":true,"duration_ms":42652,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReChisel claims that an LLM agent reflecting on compiler and simulator feedback can raise Chisel code generation from roughly 26–45 percent to 73–85 percent Pass@1 across five models, matching state-of-the-art Verilog-generating agents.","keywords":["Chisel","LLM code generation","reflection","hardware description language","agentic system","Verilog","RTL generation"],"falsifier":"A concrete check is to run ReChisel on the same 216 cases with the CommonErrorKnowledge section removed from the Reviewer prompt, and separately with a taxonomy built only from a held-out set of Chisel errors; if Pass@1 drops by more than a few points when that knowledge is removed, the reported gains depend on benchmark-derived hints rather than on the reflection loop alone.","tokens_in":1676,"feed_emoji":"🔁","tokens_out":2436,"duration_ms":51150,"temperature":0.7,"pith_summary":"The paper aims to show that Chisel, a Scala-based hardware description language with far less public code than Verilog, can still be a viable target for LLM-based hardware generation if the LLM is wrapped in an agentic loop. The proposed system, ReChisel, iteratively generates Chisel code, compiles it to Verilog, runs a simulation, and feeds concrete compiler and simulation errors back to the LLM for revision. On a filtered set of 216 module-level cases drawn from three established Verilog benchmarks, this lifts Pass@1 from roughly 26–45 percent in zero-shot settings to 73–85 percent across five mainstream LLMs. The central point is that reflection on executable feedback can largely close the performance gap between Chisel and Verilog generation, even when the underlying language is scarce in training data.","feed_headline":"Reflection loop lifts LLM Chisel code success to ~85 percent","feed_subtitle":"Compiler and simulator errors drive iterative fixes, matching Verilog-level agents despite scarce Chisel training data.","key_machinery":"The load-bearing mechanism is the reflection loop composed of four roles: a Generator that writes Chisel code, a Compiler and Simulator that act as external tools converting Chisel to Verilog and executing the testbench, an Inspector that maintains a trace of all feedback and detects repeated failures, and a Reviewer that converts the trace into a structured revision plan. Two feedback types drive the loop: compiler feedback, which gives the error location, description, and suggested fix for syntax and static-analysis errors, and simulation feedback, which lists failed test points with input stimuli, expected output, and actual output. An escape mechanism detects non-progress loops by comparing the current error against previous trace entries; when the same location and same cause recur, the intervening iterations are discarded and a fresh revision plan is generated from the step before the loop. The prompts also embed CommonErrorKnowledge, a pre-organized set of common syntax errors and fix guidelines used as in-context learning to make reviewer feedback faster and more accurate.","core_discovery":"The paper's central claim is that an LLM-based agentic system with a reflection loop can make Chisel code generation competitive with state-of-the-art Verilog code generation systems. Concretely, ReChisel takes a natural-language specification and a testbench, asks an LLM to write Chisel code, compiles that code to Verilog through the standard Chisel/FIRRTL toolchain, and simulates the resulting Verilog against the testbench. When compilation or simulation fails, an Inspector maintains a trace of all previous attempts, a Reviewer analyzes the trace to produce a revision plan with error location, root cause, and fix, and the Generator produces a corrected version. The paper reports that on 216 filtered cases from VerilogEval, AutoChip's HDLBits, and RTLLM, ReChisel improves LLM success rates substantially, and for GPT-4o it even outperforms AutoChip, a comparable agentic system that generates Verilog directly, on Pass@5 and Pass@10. The authors frame this as evidence that Chisel is not inherently a poor target for LLM-driven hardware design once feedback-based refinement is available.","pith_inferences":["The method is fundamentally dependent on the availability of a testbench for every specification; without executable tests, neither compilation-only feedback nor simulation feedback can drive the loop, so the reported gains may not transfer to open-ended spec-to-RTL tasks where testbenches are absent.","The paper does not state where the CommonErrorKnowledge used in the Reviewer prompts came from; if that error taxonomy and its fixes were derived from the same VerilogEval, HDLBits, or RTLLM cases used for evaluation, the reported gains would include test-set leakage, and a held-out evaluation could shrink them.","The escape mechanism's cycle detection is a heuristic judged by the LLM itself (same error location plus same cause), so its precision is not measured; counting how often the mechanism fires and whether discarded iterations ever contained useful fixes would clarify its real contribution.","The comparison with AutoChip is limited to three models and three benchmarks; an obvious extension is to test whether the same ReChisel loop, applied to Verilog generation instead of Chisel, would improve Verilog agents as well, which the paper does not do."],"forward_implications":["If ReChisel's reported numbers hold, Chisel becomes a practical language for LLM-assisted RTL generation despite having roughly 1/20th the public code of Verilog.","Reflection on compiler and simulator feedback appears to be a transferable recipe for low-resource HDLs beyond Chisel, since the loop depends on executable feedback rather than language-specific training data.","The escape mechanism addresses a known failure mode of self-correcting LLM agents, specifically the tendency to cycle through the same ineffective fix; detecting and discarding such loops is a general technique applicable to other code-generation agents.","The near-parity with AutoChip on Verilog suggests that the bottleneck in LLM hardware design is not the language itself but the availability of feedback signals during generation.","Because ReChisel is open-sourced, the specific prompts, trace structures, and escape logic can be reproduced and adapted by others working on HDL generation."],"supporting_citations":[{"why":"Defines Chisel as a Scala-embedded HDL, establishing the language that ReChisel targets and whose compilation path determines the feedback signals.","marker":"[2]"},{"why":"Supplies the Pass@k metric used to measure ReChisel's success rates across multiple generations.","marker":"[5]"},{"why":"Provides the HDLBits problem set that contributes part of the 216 filtered evaluation cases.","marker":"[9]"},{"why":"Describes FIRRTL, the intermediate representation that Chisel compiles to before Verilog, which underlies the compiler feedback loop.","marker":"[12]"},{"why":"Provides the RTLLM benchmark that contributes evaluation cases and a baseline for LLM RTL generation.","marker":"[17]"},{"why":"Provides the VerilogEval Spec-to-RTL benchmark that contributes evaluation cases and a reference for specification-to-RTL tasks.","marker":"[21]"},{"why":"Introduces the reflection concept for language agents, which ReChisel adapts to HDL code generation with compilation and simulation feedback.","marker":"[23]"},{"why":"Defines AutoChip, the Verilog-focused agentic system that ReChisel compares against to demonstrate comparable or better performance.","marker":"[26]"}],"fun_headline_variants":["LLM reflection lifts Chisel code success to Verilog-level","Reflection makes LLM Chisel code match Verilog agents","Chisel code generation gets a boost from LLM reflection","Compiler feedback drives LLM to craft better Chisel code"],"cache_read_input_tokens":15744,"weakest_assumption_plain":"The paper never states where the CommonErrorKnowledge injected into the Reviewer prompt came from; if it was harvested from the same evaluation benchmarks used for testing, the reported success rates include test-set leakage and the gains over zero-shot baselines would shrink on unseen specifications.","fun_headline_variants_meta":{"raw":{"variants":["LLM reflection lifts Chisel code success to Verilog-level","Reflection makes LLM Chisel code match Verilog agents","Chisel code generation gets a boost from LLM reflection","Compiler feedback drives LLM to craft better Chisel code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00046,"raw_usage":{"total_tokens":2315,"prompt_tokens":965,"completion_tokens":1350,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":1281}},"tokens_in":581,"tokens_out":1350,"duration_ms":5397,"temperature":1.0,"reasoning_tokens":1281,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:07:05.478817+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check is to run ReChisel on the same 216 cases with the CommonErrorKnowledge section removed from the Reviewer prompt, and separately with a taxonomy built only from a held-out set of Chisel errors; if Pass@1 drops by more than a few points when that knowledge is removed, the reported gains depend on benchmark-derived hints rather than on the reflection loop alone.","supporting_citations":[{"cited_title":"Chisel: constructing hardware in a scala embedded language,","cited_arxiv_id":null,"evidence_quote":"Defines Chisel as a Scala-embedded HDL, establishing the language that ReChisel targets and whose compilation path determines the feedback signals."},{"cited_title":"Problem sets - hdlbits,","cited_arxiv_id":null,"evidence_quote":"Provides the HDLBits problem set that contributes part of the 216 filtered evaluation cases."},{"cited_title":"Reusability is firrtl ground: Hardware construction languages, compiler frameworks, and transformations,","cited_arxiv_id":null,"evidence_quote":"Describes FIRRTL, the intermediate representation that Chisel compiles to before Verilog, which underlies the compiler feedback loop."},{"cited_title":"Au- tochip: Automating hdl generation using llm feedback,","cited_arxiv_id":null,"evidence_quote":"Defines AutoChip, the Verilog-focused agentic system that ReChisel compares against to demonstrate comparable or better performance."}],"review_version":1}