{"id":"ce3ee74c-168d-449a-8fd5-7333a15e632e","arxiv_id":"2504.14560","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A 7B Verilog generation model with a verified 5K dataset, two-stage training, and an adaptive reasoning router reaches 57.8% pass@1 on VerilogEval-human, 10.4 points over the prior best open-source Verilog-specific model.","lead":"ReasoningV is a 7-billion-parameter AI model that writes Verilog hardware descriptions from natural language, using a small verified dataset, a two-stage training recipe, and an adaptive difficulty router. It reports 57.8% pass@1 on VerilogEval-human, close to Gemini-2.0-flash, with up to 78% lower token use.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA result lacks a train/eval overlap check: ReasoningV-5K is filtered from PyraNet (GitHub+LLM) while VerilogEval/RTLLM are HDLBits-derived, so the reported gains could be memorization.","rationale":"The reader and I converge on the same weakest point: the empirical SOTA claim is the paper's central contribution, and every headline number in Table V is produced by a model trained on a dataset that could contain the benchmark problems. This is a missing guard in an otherwise standard pipeline, not an integrity accusation. I considered whether the adaptive-configuration discrepancy is more load-bearing: RV-Adaptive scores 53.0% on VerilogEval-human versus 57.8% for RV-Complete, so 'preserving performance' in the adaptive claim is benchmark-specific and should be stated more carefully. But that is a reporting clarity issue and does not affect the complete model's SOTA claim. The contamination question is more fundamental because if overlap is substantial it invalidates the headline result entirely. The concrete overlap check and deduplicated retraining run would settle the matter. If overlap is negligible, the paper's empirical claims are materially supported and the conditional can be lifted; if not, the central claim is unsupported. Since this condition is currently unmet, the reader's CONDITIONAL verdict stands unchanged.","tokens_in":11977,"tokens_out":6300,"duration_ms":55344,"concrete_test":"Compute exact and near-duplicate overlap between ReasoningV-5K and the 329 VerilogEval/RTLLM benchmark instances: normalize whitespace/comments, then match by exact code equivalence plus code-token and description n-gram similarity at the same 0.8 cosine threshold the paper uses for redundancy elimination. Report the fraction of benchmark problems with a near-duplicate training sample. If that fraction is non-negligible, retrain the Stage-2 model after removing all overlapping samples and re-measure VerilogEval-human pass@1; a substantial drop would confirm contamination, while a change under about 2pp would show the gains are robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition is that the reported 57.8% pass@1 and +10.4pp margin measure generalization, not memorization of evaluation problems. ReasoningV-5K is constructed in Sec. III-A by filtering PyraNet, a ~690K GitHub/LLM-derived Verilog dataset, and the model is then evaluated in Sec. IV-A on VerilogEval-human/machine and RTLLM. VerilogEval-human is drawn from HDLBits problems, and HDLBits-style solutions are common in public Verilog repositories, so overlap with PyraNet is plausible. The paper reports multi-dimensional filtering, deduplication within PyraNet, and per-sample functional verification, but it never reports any de-duplication or similarity check between the final 5,334 training instances and the 329 evaluation problems. The functional check in Sec. III-A uses each sample's own testbench, so it cannot detect benchmark overlap: a canonical HDLBits solution trivially passes its own testbench. Because the filtering criteria favor clean, compilable, well-formed code, they may if anything preferentially retain benchmark-like canonical solutions. Without an overlap analysis, the +23.5pp gain over the base Qwen2.5-Coder-7B (Table VI) and the claimed SOTA status are not distinguishable from train/eval leakage. This is the central premise on which the paper's headline empirical claims rest, and it is currently unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReasoningV, a 7B-parameter Verilog code generation model built on Qwen2.5-Coder-7B, with three main contributions: (1) ReasoningV-5K, a 5,334-instance dataset filtered from PyraNet and augmented with functionally verified solutions, testbenches, and distilled reasoning paths; (2) a two-stage training procedure that performs LoRA fine-tuning on the large OriGen dataset in Stage 1 and full-parameter fine-tuning on ReasoningV-5K in Stage 2; and (3) an adaptive reasoning mechanism that uses a lightweight Judge Adapter to classify problem difficulty and select direct, standard, or extended reasoning modes to reduce token consumption. The paper reports 57.8% pass@1 on VerilogEval-human and 73.6% on VerilogEval-machine, claiming state-of-the-art results among open-source models and up to 78% token savings with the adaptive mode.","tokens_in":12323,"tokens_out":6647,"duration_ms":54749,"significance":"If the results hold, this is a practically valuable contribution to hardware code generation: a 7B open model approaching the VerilogEval-human performance of Gemini-2.0-flash, with an efficient adaptive-inference mechanism that could lower deployment costs. The release of the model, dataset, and code supports reproducibility and follow-up work, and the two-stage training recipe plus the verified dataset are potentially reusable assets for the community. The paper also includes standard benchmarks, pass@k metrics, and functional verification with Icarus Verilog, which strengthens the empirical foundation. However, the significance is conditional on ruling out training/evaluation contamination and on obtaining cleaner ablation evidence, because the headline pass@1 numbers and the claimed SOTA margin currently rest on the untested assumption that the filtered PyraNet-derived training data is disjoint from the VerilogEval and RTLLM evaluation problems.","major_comments":[{"comment":"The paper provides no overlap or contamination analysis between the ReasoningV-5K training set, which is derived from the GitHub/LLM-sourced PyraNet dataset, and the VerilogEval and RTLLM evaluation benchmarks, which are HDLBits-derived and often present in public Verilog repositories. The functional verification described in Section III-A uses each sample's own testbench, so it cannot detect whether a benchmark problem appears in the training data. This is a load-bearing issue for the reported pass@1 gains and SOTA claims; please add a decontamination analysis (e.g., exact-match, n-gram overlap, or AST similarity between the 5,334 training instances and the 329 evaluation problems) and report the number of matched samples and the performance after excluding them.","section":"III-A and IV-A"},{"comment":"The ablation of the two-stage training method confounds dataset size with training approach: RV-RET uses full-parameter training on only 5K instances, while RV-CT uses LoRA on 220K instances. The conclusion that Stage 2 alone cannot compensate for insufficient foundational knowledge is therefore not supported, because the comparison does not control for the amount of training data. Please add controlled ablations, such as full-parameter training on OriGen-220K, LoRA on ReasoningV-5K, or data-size-matched variants, to isolate the contribution of data scale versus parameter-update strategy.","section":"III-B / Table VI"},{"comment":"The claim that adaptive reasoning preserves performance while reducing tokens is not uniformly supported by the data: on VerilogEval-human, RV-Adaptive (53.0% pass@1) is 4.8 percentage points lower than Forced Hard (57.8%), although it matches or slightly exceeds Forced Hard on VerilogEval-machine and RTLLM. Please report statistical uncertainty (e.g., confidence intervals or multiple seeds) for pass@1 and explicitly characterize the benchmark-dependent accuracy/efficiency tradeoff. Also, the abstract states token savings of \"up to 75%\" while the introduction and Section III-C state \"up to 78%\"; this should be made consistent.","section":"IV-C / Table VII"},{"comment":"The Judge Adapter is a core component of the adaptive mechanism, but the paper does not report its classification accuracy on a held-out set, nor the distribution of selected modes on the evaluation benchmarks. Without these details, it is difficult to determine whether the observed token savings come from genuinely effective difficulty routing or from a skewed difficulty distribution in the benchmarks. Please include Judge Adapter accuracy and the mode selection frequencies for VerilogEval-human, VerilogEval-machine, and RTLLM.","section":"III-C"}],"minor_comments":[{"comment":"The abstract says token savings of \"up to 75%\" while the introduction and experiments say \"up to 78%\"; please unify these numbers.","section":"Abstract"},{"comment":"The claim of exceeding the \"previous best open-source model by 10.4 percentage points\" is imprecise because Qwen2.5-Coder-32B (47.6% pass@1 on VerilogEval-human) outperforms OriGen-Gen-LoRA (47.4%); the comparison should be explicitly stated as against the previous best Verilog-specific open-source model, not all open-source models.","section":"Abstract and IV-B"},{"comment":"The text after Eq. (1) says that \"pi represents the probability that a randomly sampled solution is correct\" and then \"estimate pi as the fraction of correct solutions among these samples\"; the notation is inconsistent with the formula, which uses c and n. Please revise this sentence to correctly define c and the estimator.","section":"IV-A, Eq. (1)"},{"comment":"The operational description states that the Judge Adapter is disabled during code generation; please clarify how the selected reasoning mode is applied after the adapter's classification, since the reader cannot see the intended control flow from the text.","section":"III-C"}],"recommendation":"major_revision","confidential_remarks":"The potential train/eval overlap is the most serious concern: VerilogEval and RTLLM derive from HDLBits and are likely present in public GitHub repositories, which are a source of PyraNet. The paper's filtering pipeline does not exclude benchmark problems, and its per-sample functional verification cannot detect such overlap. I would recommend asking the authors for a complete decontamination analysis and a clear statement of any matched samples before considering publication. The ablation confound and the imprecise SOTA claim are also fixable but require additional experiments and careful rewording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, clearly written empirical systems paper for Verilog code generation, and the headline result is plausible but not yet trustworthy. The authors construct a 5,334-instance verified dataset from PyraNet, train Qwen2.5-Coder-7B in two stages—LoRA on OriGen, then full-parameter tuning on ReasoningV-5K—and add a lightweight difficulty router that picks a direct, standard, or extended reasoning mode. They report 57.8% pass@1 on VerilogEval-human and 10.4 points over the prior open-source best, with token savings up to 78%. If the numbers are real, a 7B open model is competitive with Gemini-2.0-flash on this task and much cheaper.\n\nWhat is actually new: the combination is new in the Verilog domain. Filtered verified data with distilled reasoning paths, LoRA-then-full fine-tuning, and adaptive reasoning depth have all appeared separately elsewhere, but putting them together for RTL generation is a real system contribution. The evaluation is broad, the citations are on point, and the compression-ratio analysis is a useful sanity check on the data filtering.\n\nThe soft spots. The main one is exactly the stress-test concern: there is no train/eval overlap analysis. PyraNet is assembled from GitHub and LLM-generated Verilog; VerilogEval and RTLLM are built from or around HDLBits-style problems, and those problems are all over public Verilog repositories. The paper's per-sample functional verification uses each sample's own testbench, which cannot detect benchmark overlap—a canonical HDLBits solution passes its own testbench. The filtering criteria favor clean, canonical code, which may if anything make retention of benchmark-like samples more likely. Until the authors run a similarity or containment check between ReasoningV-5K and the evaluation problems, the +23.5-point gain over the base model and the SOTA claim remain a mixture of generalization and possible memorization. This is fixable, but it is load-bearing.\n\nThe ablation study has a secondary confound: Stage 1 differs from Stage 2 in both dataset and training method, so the individual contribution of each choice is not cleanly identified. The token-saving claim is also slightly oversold: on VerilogEval-human, adaptive mode is 4.8 pass@1 below forced Hard mode, so 'preserving performance' is only true on the other two benchmarks. Minor: the abstract says up to 75% while the introduction and Table VII say 78%.\n\nFor people working on LLM-based RTL generation, this is a useful data point and a cautionary one. The paper is honest in its limitations section and not incoherent. It belongs in peer review, but a serious referee should make the contamination analysis a condition of acceptance. Good reading-group material, especially as a case study in benchmark hygiene.","headline":"A well-executed Verilog code-generation systems paper with a plausible but unsecured SOTA claim; the missing train/test overlap check is the load-bearing gap, and the paper deserves a serious referee with that condition.","tokens_in":12847,"tokens_out":5468,"would_cite":true,"duration_ms":52522,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a 7B-parameter open Verilog model, trained on a small verified dataset with two-stage tuning and adaptive reasoning, reaches 57.8% pass@1 on VerilogEval-human and cuts token use by up to 78%.","keywords":["Verilog code generation","large language models","RTL generation","reasoning paths","adaptive inference","functional verification","parameter-efficient fine-tuning","hardware design automation"],"falsifier":"Compare every ReasoningV-5K problem statement and solution against VerilogEval and RTLLM using token n-gram overlap and AST similarity; if a substantial fraction of benchmark problems appears in the training set, the reported gains could be memorization, and a held-out set of newly written hardware problems would be needed to confirm generalization.","tokens_in":11808,"feed_emoji":"⚙️","tokens_out":5689,"duration_ms":46091,"temperature":0.7,"pith_summary":"The paper tries to show that a compact, open Verilog code generator can reach the level of much larger and closed commercial models by fixing three things: training data quality, reasoning depth, and inference cost. It builds ReasoningV-5K, a dataset of 5,334 functionally verified problem–solution–testbench triples with distilled reasoning paths, trains a 7B model in two stages, and adds a lightweight difficulty classifier that routes easy tasks to direct generation and hard tasks to extended reasoning. If correct, the result is that targeted data and training design matter more than model scale for RTL code generation, and that reasoning can be made affordable. The strongest reported outcome is 57.8% pass@1 on VerilogEval-human, 10.4 points ahead of the prior best open-source model and close to a leading commercial model, while the adaptive mechanism reduces token consumption by up to 78%.","feed_headline":"7B model rivals Gemini on Verilog code generation","feed_subtitle":"Verified data, two-stage training, and adaptive reasoning lift pass@1 to 57.8% while cutting tokens up to 78%.","key_machinery":"The load-bearing mechanism is the three-stage pipeline built around a small verified reasoning dataset. ReasoningV-5K is constructed by filtering roughly 690K PyraNet samples down to 5,334 instances that compile, pass simulation against their testbenches, and carry distilled reasoning paths; this dataset supplies both the supervised reasoning signal and the difficulty labels. Two-stage training first applies LoRA on a large general Verilog corpus for foundational syntax, then full-parameter fine-tuning on ReasoningV-5K for reasoning depth. At inference, a Judge Adapter—a LoRA classifier on the same backbone—labels each problem Easy, Medium, or Hard and selects direct, standard, or extended reasoning modes with matching token budgets. These components together are what the paper credits for both accuracy and efficiency.","core_discovery":"The central claim is that a hybrid reasoning strategy—intrinsic reasoning trained into the weights plus an inference-time router that adapts reasoning depth to problem difficulty—yields state-of-the-art Verilog generation among open models. ReasoningV-Complete, built on a 7B base, reaches 57.8% pass@1 on VerilogEval-human, 73.6% on VerilogEval-machine, and 44.6% on RTLLM, surpassing the previous best open-source model by 10.4 percentage points and coming within 1.7 points of Gemini-2.0-flash while using a fraction of the parameters. The adaptive variant preserves most of this accuracy while using up to 78% fewer tokens than always running full reasoning, and on the machine benchmark it matches the full-reasoning pass@1 exactly.","pith_inferences":["If the benchmark scores are not inflated by training/evaluation overlap, the 10.4-point gain over the previous open model suggests that data verification and reasoning paths can substitute for much larger scale; the paper does not report a contamination analysis, so this remains an open check.","The judge adapter's routing policy could likely be distilled into a smaller network or merged into the main model, further cutting the small overhead of classification; the paper does not explore this.","Applying the same adaptive-reasoning scheme to other code-generation domains with cheap verifiers, such as Python unit tests or SQL, would be a natural test of whether the efficiency gains generalize beyond Verilog; the paper limits experiments to hardware benchmarks."],"forward_implications":["A 7B open model trained this way can outperform larger open models, so scaling parameters is not the only route to better RTL generation.","Adaptive reasoning can reduce inference cost by up to 78% without sacrificing accuracy on structured benchmarks, making reasoning-based Verilog generation practical at scale.","Functionally verified training samples with reasoning paths are sufficient to improve pass@1 substantially over synthetic or syntax-only datasets.","The same recipe—verified data, two-stage training, and difficulty routing—could transfer to other hardware description languages or formal-verification tasks."],"supporting_citations":[{"why":"PyraNet is the source dataset whose Tier2 samples are filtered into ReasoningV-5K.","marker":"[16]"},{"why":"DeepSeek-R1 is used to distill reasoning paths and testbenches and serves as the strongest commercial baseline.","marker":"[32]"},{"why":"OriGen is the previous best open-source model that ReasoningV-Complete surpasses by 10.4 points.","marker":"[15]"},{"why":"VerilogEval provides the human and machine benchmark tasks and testbenches used for evaluation.","marker":"[11]"},{"why":"RTLLM supplies the third evaluation benchmark of real-world RTL problems.","marker":"[12]"},{"why":"Qwen2.5-Coder-7B is the base model that all training stages start from.","marker":"[35]"},{"why":"Icarus Verilog performs the functional simulation that filters and verifies ReasoningV-5K samples.","marker":"[34]"},{"why":"LoRA is the parameter-efficient method used in Stage 1 and for the Judge Adapter.","marker":"[36]"}],"fun_headline_variants":["Adaptive reasoning cuts Verilog tokens 78% without losing accuracy","7B Verilog model beats open rivals, nearly matches Gemini","Hybrid reasoning: Verilog code gen with 78% token savings","Adaptive reasoning: open Verilog model rivals Gemini at 57.8%","Token-trimming Verilog model rivals Gemini's pass@1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claims stand on the assumption that the PyraNet-derived training samples in ReasoningV-5K do not overlap with the VerilogEval and RTLLM evaluation problems, so the reported pass@1 gains reflect generalization rather than memorization.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive reasoning cuts Verilog tokens 78% without losing accuracy","7B Verilog model beats open rivals, nearly matches Gemini","Hybrid reasoning: Verilog code gen with 78% token savings","Adaptive reasoning: open Verilog model rivals Gemini at 57.8%","Token-trimming Verilog model rivals Gemini's pass@1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00058,"raw_usage":{"total_tokens":2735,"prompt_tokens":950,"completion_tokens":1785,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":1691}},"tokens_in":566,"tokens_out":1785,"duration_ms":12594,"temperature":1.0,"reasoning_tokens":1691,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:45:50.395733+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare every ReasoningV-5K problem statement and solution against VerilogEval and RTLLM using token n-gram overlap and AST similarity; if a substantial fraction of benchmark problems appears in the training set, the reported gains could be memorization, and a held-out set of newly written hardware problems would be needed to confirm generalization.","supporting_citations":[{"cited_title":"Icarus verilog: open-source verilog more than a year later,","cited_arxiv_id":null,"evidence_quote":"Icarus Verilog performs the functional simulation that filters and verifies ReasoningV-5K samples."},{"cited_title":"Lora: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"LoRA is the parameter-efficient method used in Stage 1 and for the Judge Adapter."}],"review_version":1}