{"id":"5d9a9a5a-a9b3-4bba-97d3-d290c6cbbd8b","arxiv_id":"2412.04485","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AIvril2, a multi-agent LLM framework with EDA tool feedback, raises RTL functional pass rates to 77% for Verilog and 66% for VHDL on VerilogEval-Human.","lead":"AIvril2 wraps large language models in two feedback loops that use EDA compiler and simulator logs to fix RTL syntax and functional errors, and reports functional pass rates of 77% for Verilog and 66% for VHDL on VerilogEval-Human. A generalist should read this because it tests whether agentic self-correction can make LLM-generated hardware code reliable enough for real design flows.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 66% VHDL pass@1F lacks an independent oracle: VerilogEval-Human has no VHDL testbenches, but Section 4.1 says evaluation used 'testbenches provided in the benchmark suite'; if those testbenches were LLM-generated, the VHDL headline measures self-consistency, not correctness.","rationale":"The reader's CONDITIONAL verdict is appropriate. The Verilog result is grounded by VerilogEval-Human official testbenches, but the VHDL claim is not, and the paper's own methodology in Section 3.1 makes the Code Agent the generator of the testbench that guides functional convergence. The missing provenance of the VHDL testbenches is therefore not a style issue: it determines whether 66% is an external measurement or a self-consistency score. I see no reason to reject the paper outright; the engineering framework and relative improvements over zero-shot baselines are plausible and internally consistent. However, a conditional acceptance should require release and independent validation of the VHDL evaluation artifacts, and ideally multi-seed runs for all headline numbers. The reader identified the same weakest assumption, so I mark agreement and recommend no change to the verdict.","tokens_in":9129,"tokens_out":6312,"duration_ms":85493,"concrete_test":"Ask the authors to release the exact VHDL translations and testbenches used in Table 1 for all 156 VerilogEval-Human problems, together with the final generated VHDL RTL for each problem. Then take a random subset of 20 problems, have two independent VHDL design engineers who have not seen the authors' testbenches write VHDL testbenches from the original problem specifications, and simulate the authors' final VHDL RTL (Claude 3.5 Sonnet configuration) against these independent testbenches. If pass@1F on this subset is substantially below 66% (e.g., below 55%) or the independent testbenches reveal that the authors' testbenches are wrong on any problem, the VHDL headline is an artifact of self-generated testbench overfitting; if the rate holds near 66%, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline VHDL number (66% pass@1F for Claude 3.5 Sonnet) is the least securely grounded claim. VerilogEval-Human is a Verilog-only benchmark, so there are no 'testbenches provided in the benchmark suite' for VHDL, contrary to what Section 4.1 implies. The paper never describes how the 156 problems were translated to VHDL, who wrote the VHDL testbenches, or whether those testbenches were independently validated. This is load-bearing because the framework's own functional loop is driven by a testbench that the Code Agent generates in Section 3.1 before writing RTL. If the final VHDL 'functional' evaluation used the same self-generated testbenches, then pass@1F only confirms that the model's RTL satisfies the model's own interpretation of the spec. A self-generated testbench may have weak assertions, missing corner cases, or assertions that encode the model's incorrect initial RTL behavior; the 'All tests passed successfully!' message is emitted by the testbench itself. The Verilog result does not share this flaw because official VerilogEval testbenches provide an independent oracle. But the abstract's '66% for VHDL' and the claimed language-agnostic generality rest entirely on the unspecified VHDL evaluation artifacts. A secondary contributor is the '3.4x' headline: Table 2 compares 77% against ChipNemo-13B's 22.4%, a cross-paper comparison that is not apples-to-apples and ignores closer prior results such as AIVRIL at 67.3%. The VHDL provenance issue is the more fundamental one, since it affects whether a headline number measures anything at all.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents AIVRIL2, a multi-agent LLM framework for RTL code generation that combines a syntax optimization loop and a functional optimization loop. The loops use a Code Agent, a Review Agent, and a Verification Agent, with corrective feedback derived from EDA tool logs (Vivado). The framework is evaluated on all 156 VerilogEval-Human problems for both Verilog and VHDL, using Claude 3.5 Sonnet, GPT-4o, and Llama3-70B. Reported results include 100% syntax pass rates for five of six configurations and functional pass rates of 77% (Verilog) and 66% (VHDL) for Claude 3.5 Sonnet, stated as a 3.4x improvement over prior methods.","tokens_in":9456,"tokens_out":5420,"duration_ms":47338,"significance":"If the Verilog results are reproducible, this is a useful practical contribution: it demonstrates that iterative EDA-log feedback can raise zero-shot LLM syntax pass rates to near 100% and functional pass rates substantially, and it is one of the first attempts to evaluate LLM-generated VHDL on the VerilogEval-Human problem set. The framework is LLM-agnostic in principle, uses standard EDA tools, and the Verilog final metric appears to rely on the official benchmark testbenches. However, the strength of the empirical claims is limited by the missing provenance of the VHDL evaluation artifacts, the selective comparison used for the 3.4x headline, and the absence of any variance or reproducibility analysis.","major_comments":[{"comment":"Section 4.1 states that pass@1F was “determined by executing the testbenches provided in the benchmark suite,” but VerilogEval-Human is a Verilog-only benchmark, so there are no VHDL testbenches “provided in the benchmark suite.” The manuscript does not describe how the 156 problems were translated to VHDL, who wrote the VHDL testbenches, or how those testbenches were validated. This matters because Section 3.1 makes the Code Agent generate the testbench before RTL generation; if the VHDL testbenches were produced in that same way, the 66% pass@1F for Claude 3.5 Sonnet in Table 1 would measure self-consistency rather than functional correctness. The authors need to provide the VHDL benchmark and testbench artifacts and specify their provenance before the VHDL claims can be accepted.","section":"§4.1, Table 1"},{"comment":"The abstract’s “3.4x enhancement over prior methods” is based on comparing the 77% Verilog pass@1F with ChipNemo-13B’s 22.4% in Table 2, which is the lowest value in that table. The closest prior system, AIVRIL, reaches 67.3%, so the improvement over the strongest listed prior approach is about 1.14x, not 3.4x. As written, the headline overstates the advance and should be reworded to say “3.4x over the weakest listed baseline” or should compare with the best prior result.","section":"Abstract, §4.3, Table 2"},{"comment":"No repeated runs, confidence intervals, or variance information are reported. The unbiased pass@k estimator with k=1 is simply the empirical pass rate; given stochastic LLM sampling at temperature 0.2 and top_p 0.1, the reported differences (e.g., 72.44 vs. 77 for GPT-4o and Claude 3.5 Sonnet) may not be stable. The paper should report multiple seeds or at least bootstrap intervals for the headline pass@1F numbers.","section":"§4.1, Table 1"},{"comment":"The functional optimization loop uses a self-generated testbench as its internal oracle, and the stopping condition is that testbench’s “All tests passed successfully!” message. Even if the final Verilog metric uses the official benchmark testbenches, the paper provides no evidence that a self-generated testbench pass implies a pass on the official testbench (for example, an analysis of how often the internal loop stopped on a self-generated pass but failed the official testbench). Without such an analysis, the framework’s internal self-verification claim is incomplete.","section":"§3.1–§3.3"}],"minor_comments":[{"comment":"The framework name is written inconsistently as “AIVRIL 2” and “AI VRIL 2”; please standardize.","section":"Throughout"},{"comment":"The reference to the unbiased pass@k estimator [14] is confusing for k=1, where it reduces to an empirical proportion; consider simply reporting the fraction of passing designs.","section":"§4.1"},{"comment":"The “N/A” in the VHDL ΔF cell for the AIVRIL2 (Llama3-70B) row is unexplained; since the baseline is 0%, the caption should define how an improvement is computed in that case.","section":"Table 1"},{"comment":"The caption says the latency “accounts for the execution times of EDA tools,” but the bars appear to include LLM response time; clarify whether LLM API call time is included and report error bars or per-iteration distributions.","section":"Figure 3"},{"comment":"The reported average improvements of 38.28% for Verilog and “at least 69.44%” for VHDL are not derived in the text; showing the averaging formula would help reproducibility.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The VHDL testbench provenance issue is the key blocker: the paper claims a VHDL evaluation on a benchmark suite that does not contain VHDL testbenches. If the authors can supply independently validated VHDL testbenches or clarify their origin, the paper could become acceptable after revision. I also recommend the editor ask the authors to reframe the 3.4x claim, since the comparison against ChipNemo-13B is the weakest baseline in the table. No code or evaluation scripts are mentioned, which further limits reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one for the Verilog result, not the VHDL headline. The two-loop agentic framework—syntax loop under a Review Agent, functional loop under a Verification Agent, testbench-first—is a real, incremental contribution over the authors' prior AIVRIL, and the Verilog numbers look solid: 100% pass@1S and 77% pass@1F for Claude 3.5 Sonnet on VerilogEval-Human, using the benchmark's official testbenches as an external oracle. The authors also report consistent gains across three LLMs and give a latency breakdown that makes the cost of the loops concrete. That part deserves referee time.\n\nThe soft spots are where the stress-test note lands. Section 4.1 says pass@1F was \"determined by executing the testbenches provided in the benchmark suite,\" but VerilogEval-Human is a Verilog-only benchmark. The paper never says who wrote the VHDL testbenches, how the 156 problems were translated to VHDL, or whether those testbenches were independently checked. Given the framework's own functional loop is driven by a testbench the Code Agent generates before writing RTL, the 66% VHDL number could just be the model agreeing with itself—the \"All tests passed successfully!\" message is emitted by the model's own testbench. That is load-bearing, not a footnote. The Verilog result does not share this flaw because the official VerilogEval testbenches provide an independent oracle.\n\nThe \"3.4x\" headline is a smaller issue. It comes from comparing 77% against ChipNemo-13B's 22.4% in Table 2, while the authors' own prior AIVRIL sits at 67.3%, which makes the real gain over the closest prior system about 1.14x. The abstract's best-case framing overstates that.\n\nI also want to note the absence of code, prompts, and multi-seed runs. No error bars, temperature/top_p fixed at 0.2/0.1. These are standard requests for an evaluation paper of this kind, and they are fixable in revision. The framework description itself is coherent, and the Verilog core claim does not depend on the VHDL provenance problem.\n\nWho gets value from this paper: people building LLM-driven RTL tools, and benchmark builders who care about evaluation hygiene. Recommendation: send it to peer review with a request for the VHDL eval artifacts and a rewritten headline. The Verilog contribution is worth serious referee time; the VHDL claim needs evidence before it can be published as stated.","headline":"The Verilog core result is worth peer review; the VHDL headline is not yet supported because the testbench provenance is missing.","tokens_in":10046,"tokens_out":1796,"would_cite":true,"duration_ms":16441,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A self-verifying agentic framework lifts LLM-generated RTL functional pass rates to 77% for Verilog and 66% for VHDL.","keywords":["RTL code generation","Large language models","Verilog","VHDL","Multi-agent systems","Self-verification","EDA feedback","Functional verification"],"falsifier":"Rerun the Claude 3.5 Sonnet configuration on the same 156 problems with the same hyperparameters but using an independently created, human-checked VHDL testbench suite in place of the paper's unspecified VHDL testbenches; if the functional pass rate falls well below 66%, the VHDL claim depends on the missing testbench provenance rather than on the framework.","tokens_in":8899,"feed_emoji":"🔧","tokens_out":10228,"duration_ms":81141,"temperature":0.7,"pith_summary":"This paper tries to establish that the main bottleneck in LLM-generated hardware description code is not the model's raw ability but the absence of automated correction loops, and that a language-agnostic agentic framework can close most of the gap. Its proposed system, AIVRIL2, separates correction into a syntax loop and a functional loop: a Code Agent writes both the testbench and the RTL, a Review Agent converts Vivado compiler logs into corrective prompts, and a Verification Agent converts simulator failure logs into behavioral corrective prompts while the testbench is held fixed. On all 156 VerilogEval-Human problems, the best configuration (Claude 3.5 Sonnet) reaches 100% syntax $\\mathrm{pass@1}$ and 77% functional $\\mathrm{pass@1}$ for Verilog and 66% functional $\\mathrm{pass@1}$ for VHDL, which the paper reports as a 3.4x improvement over the best prior method and a large lift over each model's zero-shot baseline. A sympathetic reader would care because it suggests generic LLMs, without fine-tuning or retrieval augmentation, can be steered toward hardware-grade correctness by closing the loop with EDA tool feedback.","feed_headline":"Self-correcting LLM RTL flow hits 77% functional pass","feed_subtitle":"EDA tool logs drive syntax and functional fixes, lifting Verilog pass@1 to 77% and VHDL to 66%.","key_machinery":"The key machinery is a two-stage, testbench-first feedback loop. A testbench is first generated from the user prompt and frozen as the oracle; a Review Agent reads Vivado compiler logs and emits corrective syntax prompts, while a Verification Agent reads simulator logs and emits corrective functional prompts; both feed a single Code Agent that regenerates the RTL. What does the work is the invariant that the testbench never changes across iterations, so each functional pass/fail measurement is a comparison of the new RTL against the same unchanging criterion, and the error logs are distilled into targeted prompts rather than raw text.","core_discovery":"On the paper's own terms, the discovery is that a fixed testbench generated from the user specification, combined with two alternating correction loops driven by real EDA tool logs, turns weak zero-shot RTL generations into designs that pass functional checks. The Syntax Optimization loop feeds compiler diagnostics through a Review Agent that returns line-level corrections, and the Functional Optimization loop feeds simulator traces through a Verification Agent that returns behavior-level corrections; only the RTL changes between iterations. With Claude 3.5 Sonnet this lifts Verilog $\\mathrm{pass@1S}$ from 91.03% to 100% and $\\mathrm{pass@1F}$ from 60.23% to 77%, and VHDL $\\mathrm{pass@1S}$ from 88.46% to 100% and $\\mathrm{pass@1F}$ from 53.85% to 66%. The same machinery works for weaker models: Llama3-70B goes from 1.28% syntax $\\mathrm{pass@1}$ for VHDL to 58.87%, and from 0% functional $\\mathrm{pass@1}$ to 32.69%.","pith_inferences":["Editorial inference: because the oracle testbench is written by the same Code Agent that writes the RTL, the reported functional pass rates measure self-consistency against an LLM-authored specification of 'correct' at least as much as genuine conformance to the original natural-language intent.","Editorial inference: the 66% VHDL figure rests on testbenches whose provenance is not described in the paper; a plausible stress test is to rerun the VHDL loop with independently written, human-verified testbenches and compare functional $\\mathrm{pass@1}$.","Editorial inference: the fixed-testbench invariant could be pushed further by perturbing the testbench between iterations (mutation or coverage-guided expansion), which may catch overfitting to the initial oracle and improve generalization to unseen test cases."],"forward_implications":["Functional $\\mathrm{pass@1}$ of 77% for Verilog and 66% for VHDL is reachable from a generic commercial LLM with no fine-tuning and no retrieval augmentation.","Syntax errors can be driven to zero in all reported configurations except Llama3-70B on VHDL, which still improves from a 1.28% baseline to 58.87%.","Even an LLM with essentially no VHDL training data (Llama3-70B, 0% functional baseline) can be recovered to 32.69% functional $\\mathrm{pass@1}$ by the same loops.","The average functional improvement over corresponding baselines is at least 38.28 percentage points for Verilog and 69.44 percentage points for VHDL.","The framework's language-agnostic design implies the same two-loop correction structure should transfer to other HDLs, provided matching EDA tool logs are available."],"supporting_citations":[{"why":"It supplies the 156-problem VerilogEval-Human benchmark and the ground-truth testbenches used to compute the reported Verilog pass rates.","marker":"[5]"},{"why":"It provides the ChipNeMo-13B baseline (22.4% functional pass rate) against which the paper's 3.4x improvement is measured.","marker":"[1]"},{"why":"It establishes the prior ReAct/RAG-based syntax-correction approach that AIVRIL2 extends, and it is listed as a comparison baseline.","marker":"[3]"},{"why":"It gives the principal self-correction baseline and motivates AIVRIL2's testbench-first design by showing degraded performance with self-generated testbenches.","marker":"[4]"},{"why":"It is the authors' prior AIVRIL framework, whose 67.3% Verilog functional pass rate is the strongest immediate baseline that AIVRIL2 improves on.","marker":"[7]"},{"why":"It defines the unbiased pass@k estimator used for all reported syntax and functional pass measurements.","marker":"[14]"},{"why":"It provides the ReAct reasoning-and-action loop that the agents use to convert EDA log feedback into corrective prompts.","marker":"[10]"}],"fun_headline_variants":["AIvril2: EDA logs steer LLMs to 77% Verilog pass","Self-verifying agent loop lifts RTL pass rates 3.4x","RTL generation gets agent fix: 77% functional pass","EDA feedback loop sharpens LLM RTL to 100% syntax"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 66% VHDL functional pass rate assumes that valid, independent VHDL testbenches exist and are used for all 156 VerilogEval-Human design problems, but the paper never says where those VHDL testbenches come from or how they were checked.","fun_headline_variants_meta":{"raw":{"variants":["AIvril2: EDA logs steer LLMs to 77% Verilog pass","Self-verifying agent loop lifts RTL pass rates 3.4x","RTL generation gets agent fix: 77% functional pass","EDA feedback loop sharpens LLM RTL to 100% syntax"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000792,"raw_usage":{"total_tokens":3510,"prompt_tokens":984,"completion_tokens":2526,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":2443}},"tokens_in":600,"tokens_out":2526,"duration_ms":18231,"temperature":1.0,"reasoning_tokens":2443,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:53:53.017819+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the Claude 3.5 Sonnet configuration on the same 156 problems with the same hyperparameters but using an independently created, human-checked VHDL testbench suite in place of the paper's unspecified VHDL testbenches; if the functional pass rate falls well below 66%, the VHDL claim depends on the missing testbench provenance rather than on the framework.","supporting_citations":[{"cited_title":"VerilogEval: evaluating large language models for verilog code generation,","cited_arxiv_id":null,"evidence_quote":"It supplies the 156-problem VerilogEval-Human benchmark and the ground-truth testbenches used to compute the reported Verilog pass rates."}],"review_version":1}