{"id":"f7879dd4-a8d0-4f4b-bf3b-4076ff457a12","arxiv_id":"2501.12702","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A paradigm-based workflow with information-list reuse and a two-phase loop improves LLM-generated Verilog pass rates on VerilogEval, with the full-dataset result built from a hybrid of baseline and proposed-method outputs.","lead":"This paper introduces a prompting workflow that breaks Verilog code generation into structured steps, such as building truth tables and state-transition tables, and uses a two-phase loop to improve LLM output. It reports higher testbench pass rates on the VerilogEval benchmark, but the main comparison was run only on tasks that a baseline model got wrong.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported full-dataset gains rest on a hybrid evaluation—proposed method run only on baseline-failed hard tasks—so an untested regression on easy tasks could erase the claimed improvements.","rationale":"The weakest load-bearing point is exactly the one the reader identified: the experimental protocol does not establish the proposed method's full-dataset performance. The paper explicitly says the proposed framework was applied only to tasks the baseline failed, so the reported FULL row is either a hybrid estimate or a subset score compared against a full-dataset baseline. In both cases, the central quantified claim that the method significantly improves functional correctness on VerilogEval-human is not supported by the data as presented. I did not find an internal inconsistency in the paradigm-block decomposition itself; the method is plausible and the worked examples are informative, but plausibility does not substitute for a clean full-dataset measurement. The additional issues—no error bars, test-set hyperparameter selection, and calling percentage-point differences 'percent improvements'—are secondary but compound the main concern. A revised evaluation on all tasks, or at least a regression check on easy tasks, would settle the question. Therefore I keep the reader's conditional verdict unchanged.","tokens_in":13032,"tokens_out":5151,"duration_ms":51347,"concrete_test":"Re-run the full two-phase workflow on all VerilogEval-human tasks (same GPT-4-0613, temperature 0.5, Ntotal=10, (N1,N2,N3)=(7,2,1), W=0.95), computing Pass@k from the resulting code samples only. Compare with baseline 42.0/58.6/62.2. If the new full-dataset values fall below or near baseline, the reported Table I gains are an artifact of selective evaluation; if they reproduce within sampling noise, the central claim is confirmed. As a cheaper check, run the workflow on a random sample of tasks the baseline passed and measure the regression rate; if it exceeds about 5%, the hybrid construction inflates the result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline Pass@k table (Table I) is not a direct full-dataset measurement of the proposed method. Section IV states: \"To reduce token costs, we first conducted the baseline experiments and then selected the hard tasks, for which the baseline method could not generate valid code to pass the testbench after 10 retries. The hard tasks are then processed by the proposed framework.\" Thus the proposed workflow was evaluated only on the tasks the baseline already failed. If Table I's FULL row is a hybrid (baseline results for easy tasks plus proposed results for hard tasks), it assumes without evidence that the method never regresses on easy tasks; the workflow adds classification, multi-step prompts, and re-ranking, all of which can fail on tasks the baseline handled directly. If FULL reports only the hard-task subset, then comparing it to a full-dataset baseline is not a valid comparison. The paper provides no separate run on the easy tasks, so the claimed 4.7, 11.0, and 14.7 percentage-point gains are not supported. The absence of error bars and the fact that search parameters (N1, N2, N3, W) were selected on the same test set amplify the uncertainty.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-free, human-expert-inspired framework for automatic Verilog generation with LLMs. The method classifies each specification as sequential, combinational, or general, then routes it to one of three paradigm blocks (SEQU, COMB, BEHAV). Each block decomposes generation into information extraction, intermediate formatted artifacts (truth tables, state-transition tables, component lists), and code synthesis, with PyEDA used for combinational simplification. A two-phase multi-round workflow with fail-safe and short-cut mechanisms is introduced to improve testbench pass rates under a fixed number of verification runs. Experiments on VerilogEval report Pass@k improvements over a single-conversation baseline for GPT-4 and GPT-4o-mini, plus a comparison with AutoChip.","tokens_in":13254,"tokens_out":2829,"duration_ms":31059,"significance":"If the reported gains are validated, the paper would make a useful contribution: it offers a practical, training-free alternative to fine-tuning and retrieval-augmented generation for HDL code generation, with a clear and well-structured workflow. The paper is also commendable for its detailed qualitative analysis of failure modes, including examples of information-list quality affecting downstream code correctness. However, the headline quantitative claims are currently not supported as stated because the evaluation on the main benchmark is conducted only on baseline-failed hard tasks, and the full-dataset Pass@k numbers appear to be a hybrid that assumes no regression on easy tasks. The lack of statistical uncertainty, the use of the test set for parameter selection, and the small AutoChip comparison further weaken the evidence. The central idea remains defensible, but the evaluation needs substantial revision before the stated conclusions can be accepted.","major_comments":[{"comment":"The main experimental claim is not a direct full-dataset measurement. The paper states: 'To reduce token costs, we first conducted the baseline experiments and then selected the hard tasks, for which the baseline method could not generate valid code to pass the testbench after 10 retries. The hard tasks are then processed by the proposed framework.' Yet Table I reports Pass@k for the FULL row against a baseline computed over the full dataset. If the FULL row is a hybrid (baseline results on easy tasks plus proposed results on hard tasks), it assumes without any evidence that the proposed workflow does not regress on tasks the baseline already solved. The workflow adds classification, multi-step prompting, and re-ranking, all of which can fail on easy tasks. If FULL instead reports results only on the hard-task subset, then comparing it with a full-dataset baseline is invalid. The paper must clarify the exact composition of the FULL row and provide results from running the proposed method on the full dataset, or at minimum on the easy-task subset, with the combined results reported transparently.","section":"Section IV, Table I"},{"comment":"The Pass@k estimator in Eq. (1) assumes that the n code samples for a task are exchangeable draws from a single sampling distribution. In the proposed two-phase workflow, the n=10 samples are not i.i.d.: phase I samples come from SEQU/COMB, phase II reuses selected information lists and switches to BEHAV, and fail-safe and short-cut modes allocate the remaining testbench executions differently depending on earlier outcomes. Applying the standard Pass@k formula to samples generated by this adaptive procedure is not methodologically justified, and the reported Pass@1/5/10 numbers may therefore be biased in an unknown direction. The authors should either use a protocol that draws n independent completions per task under exactly the workflow being evaluated, or explicitly derive and justify an estimator appropriate for this adaptive sampling scheme.","section":"Section IV, Eq. (1)"},{"comment":"The search parameters (N1, N2, N3), the short-cut threshold W, and the maximum iteration Smax appear to be tuned on the same VerilogEval test set. In particular, Fig. 9 compares configurations (5,3,2) and (7,2,1) on the same tasks and then selects (7,2,1) for the final reported results. This is a form of test-set tuning that can inflate the reported gains, especially combined with the small per-task sample size (n=10). The authors should either evaluate on a held-out split of VerilogEval (or a separate benchmark), or report the full selection procedure and the variance of results across multiple runs or random seeds. Without such information, the headline improvements of 4.7%, 11.0%, and 14.7% cannot be taken at face value.","section":"Section IV, Fig. 9 and Table I"},{"comment":"The comparison with AutoChip is based on only ten randomly selected tasks for each circuit type, with no random seed, no confidence intervals, and no breakdown of whether the selected tasks are representative. For a claim that the proposed method 'outperforms AutoChip, particularly in combinational logic,' this evidence is too thin. The comparison also needs to ensure that both methods are given the same computational budget (e.g., same total number of testbench executions and same number of LLM calls). Table II should report the full task list or a larger benchmark subset, and the authors should state the exact protocol used for AutoChip, including how its iteration count n=10 is aligned with the proposed method's Ntotal.","section":"Section IV, Table II"}],"minor_comments":[{"comment":"The label 'FULL' is ambiguous: it could mean the full dataset, the full set of all generated samples, or the hard-task subset under the full workflow. Please define explicitly in the table caption or the text.","section":"Section IV, Table I"},{"comment":"The x-axis label 'Percentage of Error Rate Intervals' is confusing; the x-axis appears to represent intervals of testbench error rates, while the y-axis represents configurations. Consider renaming the axis and adding a note that the proportions are computed over hard tasks only.","section":"Section IV, Fig. 10"},{"comment":"The statement that 'the COMB procedure did not show any improvement in the VerilogEval-machine dataset' is consistent with Table I, but it would help to state explicitly that this means the reported FULL improvement on VerilogEval-machine comes entirely from SEQU and BEHAV, which is a nontrivial observation about where the gains originate.","section":"Section IV, paragraph 3"},{"comment":"The paradigm block names are rendered as 'SEQU', 'COMB', and 'BEHA V' in the figure and text. The inconsistent spacing in 'BEHA V' should be fixed to 'BEHAV' throughout for readability.","section":"Section III, Fig. 2"},{"comment":"Reference [9] duplicates the content of reference [2]; please merge or renumber. Also, the access date format for reference [31] is inconsistent with the other entries.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a reasonable prompting-and-workflow contribution that could be useful to the hardware code-generation community. The main barrier to acceptance is not the method itself but the evaluation methodology: the hard-task-only protocol, the hybrid full-dataset numbers, the absence of error bars, and the test-set tuning all prevent the reader from trusting the stated quantitative gains. I would encourage the editor to request a revision that either runs the proposed method on the full dataset or clearly reports separate easy-task and hard-task results with proper statistical treatment, and to strengthen the AutoChip comparison substantially."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis one is worth reading for the workflow design, but don't trust Table I as a direct comparison. The proposed method is only run on the tasks where the baseline already failed after 10 retries; the reported full-dataset Pass@k gains therefore appear to be a hybrid that carries baseline results for the easy tasks forward and assumes no regression. That assumption is untested. Since the workflow adds classification, multi-step prompts, and re-ranking, regression on easy tasks is possible, and the easy tasks are the majority.\n\nWhat's actually new: the paradigm blocks (information extraction, state-transition table or JSON truth table, three-always-block generation for sequential logic, PyEDA for combinational simplification) and the two-phase reuse of high-quality information lists. The components are not individually novel, but the integration for Verilog is new. The paper also shows a sensible classification improvement by looking at generated code rather than just the spec, and the error analysis in Fig. 11 is a nice, honest examination of why information lists fail. Credit is due.\n\nThe soft spots are concentrated in the evaluation. No error bars, n=10 per task, and the search parameters (N1,N2,N3,W) were selected by comparing two configurations on the same test set. The AutoChip comparison uses only ten arbitrary tasks and is not convincing. Those are real issues, but they are fixable: run the proposed method on all tasks, report separate easy/hard results, and add basic variance information.\n\nThe core idea is plausible and the central argument—that structured intermediate representations help LLMs generate Verilog—holds up conceptually. The current evidence, however, does not support the specific improvements in Table I. This is a solid paper for a workshop or a revised journal version, but it needs a full-data evaluation before the headline claims are publishable.\n\nMy recommendation: send it to peer review, but the reviewers should demand a complete evaluation. The method deserves further work, not desk rejection.","headline":"A plausible LLM-based Verilog generation workflow, but the headline Pass@k gains are not directly measured—the method was run only on tasks the baseline failed, so the claimed improvements rest on an untested hybrid evaluation.","tokens_in":13798,"tokens_out":1690,"would_cite":true,"duration_ms":17446,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that decomposing Verilog generation into expert-style paradigm blocks and a two-phase reuse workflow raises testbench pass rates by up to 14.7 percentage points on VerilogEval-human.","keywords":["Verilog generation","hardware description language","large language models","hallucination mitigation","paradigm blocks","chain-of-thought","two-phase workflow","Pass@k"],"falsifier":"Run the proposed workflow on every task in VerilogEval (or a random sample that includes the easy tasks), count testbench passes directly, and compare Pass@k with the baseline on the same full task set; if the gains vanish or easy-task regressions offset hard-task gains, the central claim fails.","tokens_in":12820,"feed_emoji":"🧩","tokens_out":5013,"duration_ms":43752,"temperature":0.7,"pith_summary":"This paper claims that the main obstacle to LLM-generated Verilog is not lack of hardware knowledge but the models' weakness at multi-step tasks, and that a human-expert-inspired workflow can fix this without fine-tuning, extra databases, human oversight, or testbench feedback. The method decomposes generation into paradigm blocks that extract an information list, convert it into a structured form, and then write code, with a two-phase loop that reuses the most promising intermediate lists. On the VerilogEval-human benchmark, it reports gains of 4.7, 11.0, and 14.7 percentage points in Pass@1, Pass@5, and Pass@10 over single-conversation generation with GPT-4, and larger relative gains with GPT-4o-mini. A sympathetic reader would care because training-free improvements in hardware code generation bear directly on speeding up custom chip design.","feed_headline":"Structured prompts lift Verilog pass rates 14.7%","feed_subtitle":"Human-inspired steps get GPT-4 to generate Verilog that passes testbenches more often, with no fine-tuning or database.","key_machinery":"The machinery is the paradigm block, a small prompt-and-script pipeline that mirrors human design practice: information extraction into an explicit list, conversion into a machine-friendly intermediate format, and code generation from that format. SEQU (sequential) routes through a state-transition table and sequential always blocks, COMB (combinational) routes through a JSON truth table simplified by PyEDA into a sum-of-products expression, and BEHAV is a general fallback that reuses the best information lists. The two-phase workflow carries the argument by making intermediate artifacts reusable and ranked by testbench pass rate, so that a limited number of simulation rounds is spent on the most promising reasoning traces.","core_discovery":"The central claim is that LLM hallucination in HDL generation is largely a multi-step reasoning failure, and that structuring the task into the same division-of-labor a human designer would use makes generated Verilog functionally correct more often. Concretely, the paper's paradigm blocks force the model first to spell out every input-output relation explicitly, then to cast that information into a state-transition table for sequential logic or a JSON truth table for combinational logic, then to produce code from those artifacts; an external tool, PyEDA, performs the logic simplification LLMs do poorly. A two-phase workflow then treats the testbench pass rate as a quality signal for the intermediate information lists, keeps the top candidates, and reuses them in a general BEHAV block when the specialized blocks fail. The paper reports that this pipeline outperforms direct generation and a testbench-feedback baseline, and that classification of circuit type improves when based on naively generated code rather than on the specification text alone.","pith_inferences":["A direct full-dataset run, rather than the hard-task-only evaluation with baseline carryover, could plausibly change the headline numbers; the paper does not report one.","Because the pipeline is orthogonal to fine-tuning and retrieval, combining it with either could push pass rates further; the paper does not test this.","The testbench pass rate as a quality score for intermediate artifacts is a transferable idea: any generation task with a cheap verifier could rank and reuse its intermediate states the same way.","The gains attributed to the paradigms may partly come from simply spending more tokens and more simulation rounds on hard tasks, since the baseline gets one conversation per task."],"forward_implications":["On VerilogEval-human, the method beats direct one-shot generation at every Pass@k tested, for both GPT-4 and GPT-4o-mini.","On VerilogEval-machine, the method lifts GPT-4 Pass@5 and Pass@10 by more than five percentage points, though the combinational-logic procedure alone shows no gain there.","In the direct comparison against AutoChip on a hard-task subset, the method reaches 50 percent versus 40 percent on sequential logic and 70 percent versus 0 percent on combinational logic at Pass@10.","Circuit-type classification is more accurate when the LLM judges the type from its own first-draft Verilog code (98.3 percent) than from the specification text alone (94.9 percent) on the human-written set.","The hardest tasks, with testbench error rates above 0.8, barely shrink, so the method improves moderate errors more than extreme ones."],"supporting_citations":[{"why":"Supplies the VerilogEval tasks, specifications, testbenches, and ground truth used for all Pass@k measurements.","marker":"[25]"},{"why":"Chain-of-Thought prompting is the reasoning strategy the paradigm blocks extend into structured, multi-round form.","marker":"[28]"},{"why":"AutoChip is the testbench-feedback baseline that the hard-task subset comparison must beat.","marker":"[24]"},{"why":"Provides the three-always-block coding technique used for sequential-logic code generation.","marker":"[30]"},{"why":"Defines the Pass@k evaluation metric used for functional correctness.","marker":"[34]"},{"why":"Specifies the GPT-4 model used for the main experiments and analysis.","marker":"[18]"},{"why":"Icarus Verilog is the simulator that executes testbenches to compute pass rates.","marker":"[33]"}],"fun_headline_variants":["Human-expert-inspired steps improve Verilog generation accuracy","Paradigm blocks cut LLM hallucinations in HDL code generation","Two-phase workflow improves Verilog testbench pass rates","Tool-assisted LLM flow yields more correct Verilog"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline gains assume that running the new workflow only on tasks the baseline already failed, and carrying over baseline scores for the rest, estimates full-dataset performance; the paper never checks whether the structured workflow could regress on tasks the baseline solved easily.","fun_headline_variants_meta":{"raw":{"variants":["Human-expert-inspired steps improve Verilog generation accuracy","Paradigm blocks cut LLM hallucinations in HDL code generation","Two-phase workflow improves Verilog testbench pass rates","Tool-assisted LLM flow yields more correct Verilog"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000702,"raw_usage":{"total_tokens":3156,"prompt_tokens":923,"completion_tokens":2233,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":2168}},"tokens_in":539,"tokens_out":2233,"duration_ms":17299,"temperature":1.0,"reasoning_tokens":2168,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:53:07.495643+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the proposed workflow on every task in VerilogEval (or a random sample that includes the easy tasks), count testbench passes directly, and compare Pass@k with the baseline on the same full task set; if the gains vanish or easy-task regressions offset hard-task gains, the central claim fails.","supporting_citations":[{"cited_title":"VerilogEval: Evaluating large language models for Verilog code generation,","cited_arxiv_id":null,"evidence_quote":"Supplies the VerilogEval tasks, specifications, testbenches, and ground truth used for all Pass@k measurements."},{"cited_title":"Chain-of-Thought prompting elicits reasoning in large language models,","cited_arxiv_id":null,"evidence_quote":"Chain-of-Thought prompting is the reasoning strategy the paradigm blocks extend into structured, multi-round form."},{"cited_title":"Coding techniques in verilog for finite state machine designs in FPGA,","cited_arxiv_id":null,"evidence_quote":"Provides the three-always-block coding technique used for sequential-logic code generation."},{"cited_title":"Icarus Verilog: open-source Verilog more than a year later,","cited_arxiv_id":null,"evidence_quote":"Icarus Verilog is the simulator that executes testbenches to compute pass rates."}],"review_version":1}