{"id":"16c6528b-23eb-4cbb-bf04-4fd2b5a34520","arxiv_id":"2504.19144","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning reasoning LLMs on prompt-guided Chisel and Verilog-derived training data substantially improves Chisel code generation correctness and design variability.","lead":"This paper builds and releases Chisel-focused AI models that turn hardware designs into working Chisel code more reliably than general-purpose coding models. It may make agile chip design faster by giving hardware engineers a cheaper, open-source helper for a niche programming language.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No decontamination check: training on The Stack v2 and public Verilog sources may overlap with VerilogEval-Human and RTLLM v2.0, so reported Pass@k and syntax gains could reflect memorization rather than domain adaptation.","rationale":"The paper is a plausible engineering contribution, and the public release of datasets and models is real support. My concern is not that the authors misreported numbers but that the evaluation may be contaminated by construction. Section IV-A explicitly builds training data from The Stack v2 and from public Verilog collections (Fig. 2a labels MG-Verilog), while Section V-B evaluates on VerilogEval-Human and RTLLM v2.0, which are the same kind of public GitHub-derived RTL material. No decontamination is described. Because the Decompile-to-Chisel task is trained on (Verilog -> Chisel) pairs, the model could memorize source-to-target mappings for benchmark files that happen to be in the training corpus. This would inflate ChiseLLM Pass@1/Pass@5 and syntax correctness relative to baselines that saw the same code only in pretraining, and would make the central claim that 'prompt-guided reasoning trace distillation' drives the gains unsupported. The reader's concern about running original testbenches on emitted SystemVerilog is legitimate but secondary; it would mostly affect semantic Pass@k through interface mismatches, and it applies symmetrically to all models. A contamination audit is cheap and decisive. If overlap is negligible, the paper's empirical claims are much stronger; if not, the headline numbers need to be recomputed on a disjoint subset. I would keep the verdict CONDITIONAL, adding decontamination as a necessary condition, and also ask the authors to report the high-variance exclusion threshold in the variability judge, which is a separate but smaller issue.","tokens_in":15297,"tokens_out":9429,"duration_ms":97898,"concrete_test":"Run a decontamination audit: normalize all Verilog source files in VerilogEval-Human and RTLLM v2.0 (strip comments and whitespace, canonicalize identifiers), then compute exact-file hashes and maximum 50-token n-gram overlap against the raw ChiseLLM-Decompile-Base and ChiseLLM-Completion-Base corpora. Report the fraction of benchmark problems with an exact match or >80% snippet overlap. If the contaminated fraction exceeds roughly 5%, rerun Table II on only the disjoint subset; if the ChiseLLM advantage over Qwen2.5-Coder baselines largely disappears, the headline gains are explained by leakage rather than by the proposed distillation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim—that prompt-guided reasoning-trace distillation, not corpus exposure, produces the gains—rests entirely on Pass@k and syntax results over VerilogEval-Human and RTLLM v2.0 (Section V-B, Table II). Yet Section IV-A builds the training data from 'the-stack-v2-dedup' plus 'publicly accessible ... Verilog datasets' (the latter labeled MG-Verilog in Fig. 2a), and the benchmarks are themselves public, GitHub-derived Verilog collections. No decontamination step or overlap analysis is reported anywhere. This is especially damaging for Decompile-to-Chisel, where the training set consists of (Verilog source, Chisel target) pairs: if a benchmark Verilog file or a near variant appears in the training corpus, the model can memorize the exact source-to-answer mapping and inflate Pass@1/Pass@5 relative to baselines that never saw those files. The same overlap would also inflate syntax correctness on benchmark-derived prompts. Because Table II and the abstract's 18.85%/26.32% improvements are the only quantitative support for the method, the absence of a contamination audit makes the central claim currently unverified. A secondary concern is the reader's interface-mismatch risk when running original testbenches on emitted SystemVerilog, but that applies symmetrically to all models, whereas contamination differentially favors ChiseLLM.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ChiseLLM, a dataset construction and fine-tuning pipeline intended to adapt reasoning large language models to Chisel hardware description generation. The authors collect Chisel and Verilog sources from public repositories, construct completion and decompile instruction datasets, use DeepSeek-R1 with prompt-guided templates to synthesize reasoning traces, and fine-tune Qwen2.5-Coder-7B and 32B. They evaluate against VerilogEval-Human and RTLLM v2.0 using Pass@k and syntax correctness, and use LLM-as-a-judge to score variability. The reported results claim large improvements over base models and performance comparable to commercial systems.","tokens_in":15545,"tokens_out":4865,"duration_ms":42091,"significance":"If the results hold, this is a practically useful contribution to a low-resource domain: it provides open datasets and models for Chisel, demonstrates a prompt-guided reasoning distillation recipe, and shows that smaller open-weight models can be adapted to a niche hardware language with substantial gains. The strengths are the concrete pipeline, public release of datasets and models, and use of a standard Pass@k evaluation with compilation-based syntax checking. However, the current reporting of the headline numbers is internally inconsistent, the evaluation is not decontamination-checked, and the variability metric rests on a partially self-referential LLM judge. These issues currently prevent the quantitative claims from being fully trusted.","major_comments":[{"comment":"The headline improvements do not match the data in Table II. The abstract and Section I state that ChiseLLM-7B and ChiseLLM-32B improve 'syntax correctness' by 18.85% and 26.32%, respectively. Computed from Table II, the 26.32% figure is the average absolute improvement in Pass@5 for 32B across the four benchmark columns, not a syntax-correction gain; the actual syntax-correctness improvements for 32B are 2.98, 10.78, 16.05, and 33.90 percentage points (mean 15.93). The 18.85% equals the 7B syntax improvement averaged only over the two VerilogEval-Human columns, not an overall average. The paper must correct the abstract, introduction, and any other summary statements so that every reported percentage is clearly tied to a defined metric and the set of tasks.","section":"Abstract, Section I, Table II"},{"comment":"No decontamination or overlap analysis is reported between the training data and the evaluation benchmarks. The training data come from the-stack-v2-dedup and public Verilog datasets (Section IV-A, Fig. 2a), while VerilogEval-Human and RTLLM v2.0 are public, GitHub-derived Verilog collections used for testing. Because the Decompile-to-Chisel training set consists of (Verilog source, target Chisel) pairs, any benchmark Verilog file or near variant appearing in training could inflate Pass@k and syntax results relative to baselines that did not see those files. The authors should report a contamination check (e.g., exact or near-duplicate detection of benchmark prompts and source files in the training corpora) or explicitly decontaminate the training data and re-run the evaluation.","section":"Section IV-A, Section V-B, Table II"},{"comment":"The variability evaluation is partially self-referential and the exclusion rule is underspecified. The ChiseLLM models were fine-tuned on reasoning traces that explicitly demonstrate the exact variant patterns (configurable, functional, structural) that the judge is asked to score, and the judge is an LLM rather than human evaluation. In addition, the paper states that 'samples with high evaluation variance are not included' but does not give the variance threshold or the number of excluded samples. This can bias the reported 47.58% improvement. Please report the exclusion statistics, provide a judge-human agreement or inter-judge agreement measure, and add an analysis showing that the variability score is not simply a reward for reproducing the training-distribution reasoning patterns.","section":"Section IV-B2, Section V-B, Variability Evaluation Method, Figure 4"},{"comment":"The validity of Pass@k as a measure of Chisel quality depends on an assumption that is not tested. The paper compiles generated Chisel to SystemVerilog with ChiselStage.emitSystemVerilog and then runs the original Verilog testbenches, which assumes that the emitted SystemVerilog matches the benchmark module interface and semantics closely enough for testbench failures to reflect real Chisel errors. The paper should state how many generated samples failed at the emission or simulation stage and, ideally, validate the pipeline on a small set of human-written Chisel designs with known-correct SystemVerilog equivalents. This issue applies symmetrically to all models, but it affects the absolute Pass@k values and the interpretation of cross-model comparisons.","section":"Section V-B, Correctness Evaluation Method"}],"minor_comments":[{"comment":"Figure 4 reports mean variability scores without error bars or confidence intervals, even though the text discusses variance and uncertainty; please add dispersion measures or a table with standard deviations.","section":"Figure 4"},{"comment":"The statement that 'ChiseLLM-32B consistently ranks as the top or second-best performer across all metrics' is not supported by Table II: on VerilogEval-Human Spec-to-Chisel P@5 and syntax, DeepSeek-R1-671B is higher, and on several columns ChiseLLM-32B is not second-best. Please qualify this claim.","section":"Table II, Section V-C"},{"comment":"The reference for The Stack v2 is given as 'A. L. et al.'; please provide the full author list or a proper citation format so that readers can verify the dataset.","section":"Section IV-A, Reference [38]"},{"comment":"The paper promises a 'standardized evaluation system' but does not release the judge prompt or the list of generated variants; making these public would significantly improve reproducibility and trust in the variability results.","section":"Section V-B, Variability Evaluation Method"}],"recommendation":"major_revision","confidential_remarks":"The mismatch between the abstract numbers and Table II is a serious internal inconsistency that any reviewer or reader will notice immediately; it should be treated as a blocking issue for the current version. The contamination concern is the main scientific risk: given that the training data are public RTL sources and the benchmarks are public Verilog collections, the absence of any overlap analysis leaves the central domain-adaptation claim unverified. I would encourage the editor to request a concrete decontamination analysis before considering acceptance, and to verify that the corrected numbers in the revision match the tables."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ChiseLLM is a useful, artifact-first contribution to low-resource hardware code generation. The authors fine-tune Qwen2.5-Coder models on prompt-guided reasoning traces for two Chisel tasks, release the datasets and models, and evaluate by compiling generated Chisel and simulating against Verilog testbenches. Worth engaging with, but the headline numbers are not reliably reported and there is an open contamination question that needs closing before the gains are believable.\n\nWhat is actually new: the pipeline. Mixed Chisel/Verilog corpus, line-level documentation annotation, Deepseek-R1 distillation with task-specific thinking patterns—syntax reflection for Spec-to-Chisel, three explicit variant classes for Decompile-to-Chisel. That is a genuine extension over ReChisel and ChatChisel, and the ablation shows both datasets help individually and combine well. The external correctness evaluation (Scala compile + ChiselStage emit + Verilog testbench simulation) is the right way to measure functional correctness, and the baselines include strong open and commercial models. Public models and data are real and make the work reproducible.\n\nSoft spots, in rough order of importance.\n\nFirst, contamination. Training data is built from the-stack-v2-dedup and MG-Verilog; the benchmarks, VerilogEval-Human and RTLLM v2.0, are public GitHub-derived Verilog collections. No decontamination or overlap analysis is reported. For Decompile-to-Chisel, the training pairs are Verilog-to-Chisel, so any benchmark Verilog file that appears in training can directly inflate Pass@k through memorization rather than domain adaptation. This is not an accusation of cheating; it is a required audit before the central empirical claim can be trusted.\n\nSecond, reporting. The abstract says syntax correctness improved by 18.85% and 26.32%. Table II does not support those as syntax gains: 26.32% matches the 32B Pass@5 average, and 18.85% does not match any average I can reproduce from the table. The abstract is misleading and should be corrected.\n\nThird, variability. The judge is an LLM, samples with high variance are removed without reporting the threshold, and the models were trained to exhibit the very variant patterns being judged. That metric needs a precise protocol and at least a small human-validated subset.\n\nFourth, missing baselines. ReChisel and ChatChisel are cited but never compared. For a Chisel-specific paper, a direct comparison with those systems is necessary.\n\nNone of these is a fatal flaw in the method itself. The core idea is plausible and the artifacts make it testable. As it stands, the paper deserves a serious referee, but not acceptance yet. For anyone working on LLM-assisted hardware design, the datasets and models are immediately useful. I would bring it to reading group and cite it, with a caveat about the open questions.","headline":"Useful open Chisel code-generation artifacts, but the abstract's numbers are misreported and the training/benchmark overlap needs an audit before the gains can be trusted.","tokens_in":16130,"tokens_out":3998,"would_cite":true,"duration_ms":38201,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Domain-adapted reasoning traces make small open LLMs competitive at Chisel hardware code generation.","keywords":["Chisel","hardware construction language","LLM code generation","reasoning models","domain adaptation","reasoning trace distillation","agile hardware development","RTL decompilation"],"falsifier":"Run the same design specifications through native Chisel testbenches, compiling each generated module with the Scala compiler and exercising it with Chisel's own testing constructs. If the models' Pass@k drops sharply under native testing, the SystemVerilog bridge was inflating the reported results.","tokens_in":15059,"feed_emoji":"⚙️","tokens_out":5180,"duration_ms":47941,"temperature":0.7,"pith_summary":"This paper argues that general reasoning models fail to improve Chisel code generation unless they are adapted to the hardware domain. It presents ChiseLLM, a dataset-and-model pipeline that distills prompt-guided reasoning traces into small open code models. If correct, the work shows that a 32B model can reach the Chisel generation quality of much larger commercial systems, making LLM-assisted agile hardware development practical at low cost. The paper also introduces two task formulations, Spec-to-Chisel and Decompile-to-Chisel, and reports that domain adaptation raises variability design ability substantially.","feed_headline":"Small tuned model matches giant rivals on Chisel hardware code","feed_subtitle":"Prompt-guided reasoning traces push open LLMs past commercial models on Chisel syntax and Pass@k.","key_machinery":"The central mechanism is prompt-guided reasoning trace distillation: a strong reasoning model is given prompts that inject Chisel documentation fragments, benchmark reference answers, variant-pattern specifications, and Chisel feature descriptions, and its resulting reasoning traces are used to fine-tune smaller base models. This transfers task-specific thinking patterns rather than merely teaching new facts. The data pipeline also includes filtering Chisel and Verilog sources, converting them into completion and decompilation instruction datasets, and mixing those datasets in a 3:7 ratio for training.","core_discovery":"The central claim is that domain adaptation through prompt-guided reasoning trace distillation converts generic code models into capable Chisel generators, while unadapted reasoning models do not bring consistent benefits. On Verilog-derived benchmarks, ChiseLLM-7B and ChiseLLM-32B improve syntax correctness by 18.85% and 26.32% respectively over their base models, and the 32B model matches or beats much larger commercial systems on several functional-correctness measures. The same training lifts variability design ability by 47.58% relative to a baseline reasoning model. The authors attribute the gain to the distilled thinking patterns: the models learn to reason about the module's function, recall Chisel syntax from documentation, consider configurable, functional, and structural variants, and only then emit code.","pith_inferences":["If the reported gains hold, the same prompt-guided distillation recipe should transfer to other hardware construction languages and to other low-resource domain-specific languages, since the bottleneck appears to be task-specific thinking patterns rather than model scale.","The correctness numbers rest on compiling generated Chisel to SystemVerilog and running Verilog testbenches; a native Chisel testbench suite would be a stronger check and might change the absolute Pass@k values.","A natural extension would be to compile generated parameterized modules at several configurations and verify that each configuration is synthesizable and functionally correct, testing whether the variability improvements survive real parameter sweeps."],"forward_implications":["Smaller open models, around 7B and 32B parameters, can be made to generate Chisel that is syntactically and functionally competitive with much larger commercial systems, cutting the compute cost of LLM-assisted hardware design.","Unadapted reasoning models cannot be assumed to help on low-resource hardware languages; domain adaptation is a precondition for practical gains.","Combining a Chisel code-completion dataset with a Verilog-to-Chisel decompilation dataset produces larger gains than either dataset alone, pointing to a synergistic effect.","Injecting variant-pattern guidance into reasoning traces teaches models to produce parameterized, configurable designs, supporting design-space exploration and agile hardware methodology."],"supporting_citations":[{"why":"Supplies the distillation methodology and the strong reasoning model used as teacher and as a baseline.","marker":"[16]"},{"why":"Provides the the-stack-v2-dedup source from which the Chisel completion dataset is extracted.","marker":"[38]"},{"why":"Supplies the RTLLM v2.0 benchmark used for correctness evaluation.","marker":"[44]"},{"why":"Provides the VerilogEval-Human benchmark and related LLM hardware design work.","marker":"[10]"},{"why":"Defines the base code models that are fine-tuned to produce ChiseLLM.","marker":"[21]"},{"why":"Defines the Spec-to-Chisel task and motivates it as a practical generation scenario.","marker":"[14]"},{"why":"Defines the Decompile-to-Chisel task and prior programmatic Verilog-to-Chisel conversion.","marker":"[36]"},{"why":"Supports the assumption that fine-tuning mainly teaches models to use existing knowledge rather than acquire new knowledge.","marker":"[40]"},{"why":"Supplies the LLM-as-a-Judge method used for variability design evaluation.","marker":"[45]"}],"fun_headline_variants":["Domain-adapted LLMs beat generic reasoning on Chisel","Distilled reasoning traces boost Chisel code generation","Small tuned models match big rivals on Chisel hardware","Reasoning-tuned LLMs improve Chisel syntax by 26%","ChiseLLM: domain adaptation wins on Chisel code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported Pass@k numbers treat Verilog testbenches as a proxy for Chisel correctness: generated Chisel is compiled to SystemVerilog and run against the original Verilog testbench, so the whole comparison assumes that translation preserves the benchmark's interfaces and semantics closely enough for failures to reflect real Chisel errors.","fun_headline_variants_meta":{"raw":{"variants":["Domain-adapted LLMs beat generic reasoning on Chisel","Distilled reasoning traces boost Chisel code generation","Small tuned models match big rivals on Chisel hardware","Reasoning-tuned LLMs improve Chisel syntax by 26%","ChiseLLM: domain adaptation wins on Chisel code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000289,"raw_usage":{"total_tokens":1699,"prompt_tokens":957,"completion_tokens":742,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":660}},"tokens_in":573,"tokens_out":742,"duration_ms":7173,"temperature":1.0,"reasoning_tokens":660,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T06:00:13.796331+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same design specifications through native Chisel testbenches, compiling each generated module with the Scala compiler and exercising it with Chisel's own testing constructs. If the models' Pass@k drops sharply under native testing, the SystemVerilog bridge was inflating the reported results.","supporting_citations":[{"cited_title":"DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the distillation methodology and the strong reasoning model used as teacher and as a baseline."},{"cited_title":"Starcoder 2 and the stack v2: The next generation,","cited_arxiv_id":null,"evidence_quote":"Provides the the-stack-v2-dedup source from which the Chisel completion dataset is extracted."},{"cited_title":"OpenLLM-RTL: Open Dataset and Benchmark for LLM-Aided Design RTL Generation","cited_arxiv_id":null,"evidence_quote":"Supplies the RTLLM v2.0 benchmark used for correctness evaluation."},{"cited_title":"Revisiting VerilogEval: A year of improvements in large-language models for hardware code generation,","cited_arxiv_id":null,"evidence_quote":"Provides the VerilogEval-Human benchmark and related LLM hardware design work."},{"cited_title":"ChatChisel: Enabling Agile Hardware Design with Large Language Models,","cited_arxiv_id":null,"evidence_quote":"Defines the Spec-to-Chisel task and motivates it as a practical generation scenario."},{"cited_title":"(system)verilog to chisel translation for faster hardware design,","cited_arxiv_id":null,"evidence_quote":"Defines the Decompile-to-Chisel task and prior programmatic Verilog-to-Chisel conversion."},{"cited_title":"LIMA: Less is more for alignment,","cited_arxiv_id":null,"evidence_quote":"Supports the assumption that fine-tuning mainly teaches models to use existing knowledge rather than acquire new knowledge."}],"review_version":1}