{"id":"059ccbb6-e85e-4660-8407-b9fa48cb73dd","arxiv_id":"2507.16808","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"LLM-based RTL optimizers degrade on timing-heavy mutants, but the study's own data and methods do not fully support the headline claim.","lead":"This paper tests whether large language models can optimize RTL hardware code that contains complex timing logic, using a new benchmark and mutation-based evaluation method. It reports that LLMs handle simple logic well but struggle with timing control flow and clock domains, though the evidence is weakened by methodological issues.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The metamorphic mutants in Sections III-B3 and III-B4 are never shown to be cycle-exact; without sequential equivalence, RQ3/RQ4 comparisons conflate added latency with added complexity.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the metamorphic mutants are asserted to be semantically equivalent, but the described transformations in Sections III-B3 and III-B4 inherently alter cycle counts by inserting pass-through states and synchronizer registers. Because the evaluation metrics include delay, area, and power, and because the paper's central claim is that LLM-based methods fail specifically on complex timing logic, the validity of the mutant set is foundational. Section III-C's claim of formal verification is not backed by a documented sequential equivalence proof or reported results; ABC is typically a combinational equivalence and synthesis tool, and the simulation protocol is unspecified. Without cycle-exact equivalence, the observed performance drop on mutants cannot be attributed to timing-logic complexity rather than to changed behavior. This is a correctness risk that breaks the evaluation, not merely a disagreement with current consensus. I see no separate concern strong enough to override this one; the reader's rejection is appropriate, and the proposed check would settle whether the concern actually lands by restricting the analysis to provably equivalent mutants.","tokens_in":18657,"tokens_out":5207,"duration_ms":59661,"concrete_test":"Run a full sequential equivalence check on all 72 mutant pairs from the timing control flow and clock domain subsets (Tables I and II). For each pair, use SymbiYosys (or an industry sequential equivalence checker) with bounded model checking plus k-induction, initializing from the reset state, and prove that the original and mutant exhibit identical outputs for every input sequence up to at least 100 clock cycles; also simulate 1000 randomized input traces and compare waveforms cycle-by-cycle. Then recompute Tables I and II using only the mutant pairs that pass cycle-exact equivalence. If the passing subset changes the qualitative ranking between LLM-based methods and Yosys, the reported findings are artifacts of non-equivalent mutants.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim depends on the premise that each 'mutant' RTL design is semantically equivalent to the original, including cycle-by-cycle timing behavior (Section III-C). The metamorphosis strategies described in Sections III-B3 and III-B4 violate this premise in the reported designs: pass-through states in FSM transformation (S1 to S1A to S1B to S1C to S2) add register stages that delay state transitions, and the clock-domain strategy in Listing 3 inserts synchronizer registers (syncreg1, syncreg2) and an extra combinational stage between regA and regB, increasing the number of cycles from input to output. The paper claims formal verification with 'ABC' and Icarus simulation in Section III-C, but no sequential equivalence procedure or results are documented; ABC is most commonly used for combinational equivalence, and the simulation setup is not described. If the mutants have different latency, then the observed drop in optimization effectiveness for timing control flow and clock domains may simply reflect that the mutants are different circuits with extra storage elements, not that the LLMs fail to understand timing logic. The 'key insight' (optimization effectiveness should remain consistent for semantically equivalent code) is therefore untestable in the current evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a metamorphic evaluation methodology and a new benchmark for assessing LLM-based RTL code optimization across four areas: logic operations, data paths, timing control flow, and clock domains. The authors generate 'mutant' RTL designs that are claimed to be semantically equivalent to original designs but more complex, and they compare optimization effectiveness (wires, cells, area, delay, power) of LLM-based methods (RTLRewriter, GPT-4, Claude-3.7-sonnet) against Yosys on original vs. mutant code. The reported findings are that LLM-based methods outperform Yosys on logic and data-path optimization but do not outperform Yosys on timing control flow and clock-domain optimization.","tokens_in":19009,"tokens_out":8524,"duration_ms":93472,"significance":"The question is timely and practically important: if the finding holds, it would caution against using current LLM-based methods for timing-critical RTL optimization and motivate timing-aware model designs. The paper also contributes a benchmark and an evaluation idea that could be reused by the community. A particular strength is the inclusion of Yosys as an external baseline, which avoids the circularity of evaluating LLMs only against themselves. However, the empirical claims are currently undermined by load-bearing problems in the normalization scheme and, more fundamentally, in the semantic equivalence of the generated mutants.","major_comments":[{"comment":"The normalization formula Ratio(d)_A = M_A(d)/M_Yosys(d) normalizes by Yosys on the same instance d, but Tables I and II are labeled 'normalized to Yosys org' and report mutant rows whose denominators cannot be Yosys(mutant): if they were, the Yosys mutant row would be identically 1.00, yet it is reported as 1.01/2.93/1.00 and 1.07/1.04/1.00. The mutant-vs-original comparisons (e.g., GPT org area 1.78 vs GPT mut 7.93 in Table I) therefore compare ratios with different denominators, and the conclusion that optimization effectiveness degrades on mutants is not supported. The analysis must be redone with one consistent normalization, preferably per-instance ratios relative to Yosys on the same instance, with the mutant/original comparison made on those ratios.","section":"Section IV-E (Eq. 1) and Tables I-II"},{"comment":"The FSM metamorphosis replaces a direct transition S1→S2 with S1→S1A→S1B→S1C→S2. In a synchronous FSM each inserted state occupies at least one clock cycle, so the mutant cannot be cycle-equivalent to the original: either the intermediate states do not assert the same outputs as S1, changing the visible behavior, or they do, extending the time spent in the green/active state. The paper's assertion that 'the overall functionality and behavior are fully consistent with the original code' is therefore questionable. Because Section III-C reports no sequential equivalence checking procedure or results, RQ3's comparisons conflate added latency with added timing complexity.","section":"Section III-B3 and Fig. 5"},{"comment":"The clock-domain metamorphosis splits a single always block into two clock domains and inserts synchronizer registers syncreg1 and syncreg2 plus an extra combinational stage between regA and regB. Even with identical-frequency clocks, data at regB is delayed by at least two additional clock cycles compared to the original, and the relationship is worse if clk2 is a multiple of clk1. Listing 3 is therefore not a semantics-preserving transformation under the paper's cycle-level notion of RTL semantics, and the RQ4 results may simply reflect the added synchronizer stages rather than any LLM-specific difficulty with clock-domain logic.","section":"Section III-B4 and Listing 3"},{"comment":"The paper claims that formal verification and simulation with ABC and Icarus guarantee semantic equivalence of originals and mutants, but no equivalence-checking command, netlist comparison, or quantitative results are described. ABC's commonly used `cec` checks combinational equivalence and cannot, without a documented sequential equivalence flow, certify equivalence of FSMs with extra states or synchronizer chains. Without such evidence, the key insight that optimization effectiveness should be invariant for semantically equivalent code is untested, and the metamorphic evaluation cannot support the paper's conclusions.","section":"Section III-C"},{"comment":"The experimental claims rest on single-run point estimates from stochastic LLMs with a single fixed prompt. No standard deviations, confidence intervals, or statistical tests are reported for any RQ, and the benchmark is limited to the authors' own 54/27/40/32 cases with no per-case distributions. The differences that drive the conclusions (e.g., Claude mut area 0.55 vs GPT mut area 7.93 in Table I) may not be reproducible or statistically significant. At minimum, repeated runs with variance reporting and, ideally, tests across the per-instance ratios are needed before the findings can be accepted.","section":"Section IV-B, IV-D, IV-F through IV-I"}],"minor_comments":[{"comment":"The captions of Listing 1 and Listing 3 both say 'Example of Metamorphosis strategies in Logic Operation Optimization', but Listing 1 shows a counter and Listing 3 illustrates clock-domain metamorphosis; both captions should be corrected.","section":"Listings 1 and 3"},{"comment":"There are several typos, including 'series issues' in the abstract (likely 'serious issues') and 'oftware programming languages' in Section II-B (missing 's' in 'software').","section":"Abstract and Section II-B"},{"comment":"The description 'we invited a number of domain experts to classify' lacks detail: the number of experts, their qualification criteria, and inter-rater agreement are not reported, which weakens the benchmark-construction threat discussion.","section":"Section III-A"},{"comment":"The benchmark is described only by counts (54, 27, 40, 32) and two source references; a table or appendix listing the module names, sources, licenses, and the classification result for each case would be needed for reproducibility.","section":"Section IV-B"},{"comment":"The t-SNE discussion says blue/green/red points show that MR-3 is farther from the original than some semantic-breaking variants, but no quantitative embedding distances are given; adding actual distance values would make the motivating observation more concrete.","section":"Section II-B and Fig. 2"},{"comment":"The definitions of wires, cells, area, delay, and power are prose-level; please specify the synthesis technology library and the exact Yosys pass script (e.g., synth, abc, opt flags), since these settings determine all reported metrics.","section":"Section IV-C"}],"recommendation":"reject","confidential_remarks":"The paper addresses a timely question and the idea of metamorphic evaluation with Yosys as an external anchor is attractive. However, the mutant equivalence problem is not a presentation issue: the described transformations for timing control flow and clock domains cannot be cycle-equivalent, so the central evaluation for RQ3/RQ4 is invalid. Even if the normalization inconsistency were fixed, the conclusions would still rest on non-equivalent mutants. I therefore recommend rejection, although a substantially revised version with truly cycle-equivalent mutants and a corrected normalization could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper is the first to apply metamorphic testing to LLM-based RTL optimizers, and the four-category benchmark (logic, datapath, timing control flow, clock domains) is a useful contribution if it ever ships publicly. Second, the two headline findings about timing logic — RQ3 and RQ4 — are not supported by the paper's own data, because the mutant designs are probably not semantically equivalent at the cycle level, and the normalization used in the tables contradicts the formula in Section IV-E.\n\nWhat's new: the benchmark construction is careful about category separation, and the logic/data-path results include concrete failure examples (GPT-4 wire bloat, Yosys trouble with nesting). Those parts are informative and mostly plausible.\n\nWhere it falls down. (1) The FSM strategy inserts pass-through states (S1→S1A→S1B→S1C→S2). Each intermediate state consumes a clock cycle, so a transition that used to take one cycle now takes four. The clock-domain strategy does the same thing: Listing 3 adds syncreg1 and syncreg2 before regB, adding two cycles of latency. The paper says ABC and Icarus simulation prove equivalence, but no sequential equivalence check or simulation setup is described, and ABC on its own is not a sequential equivalence proof. The mutants are different circuits with extra storage elements, not just more complex versions of the same timing behavior. (2) The normalization: Section IV-E says ratios are \"relative to the Yosys\" on the same input, but Table I lists Yosys mut at 1.01× delay and 2.93× area. If the denominator were Yosys on the mutant, that row would be all 1.00. The tables clearly use Yosys on the original code as the denominator. That is a different quantity, and it makes the mutant-vs-original comparison ambiguous.\n\nCredit where due: the paper honestly lists threats in Section V, which is more than many empirical papers do. The abstract, however, overstates the timing-logic result as established fact.\n\nWho should read it: people building LLM-based RTL optimization systems will want to know this benchmark exists. But they should not cite it as evidence that LLMs are worse than Yosys on timing logic, because that conclusion is not yet established.\n\nRecommendation: I would send it to peer review, with a clear request for major revision. The core idea deserves a chance. The authors need to (a) prove or fix the cycle-exactness of the mutants, (b) align the normalization with the tables or vice versa, and (c) release the benchmark. If they do that, the paper could be a solid reference. As it stands, the load-bearing claims are unsupported.","headline":"A useful metamorphic benchmark for LLM-based RTL optimization, but the timing-logic conclusions rest on unproven cycle equivalence and an inconsistent normalization.","tokens_in":19390,"tokens_out":3448,"would_cite":false,"duration_ms":36308,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LLM-based RTL optimization methods outperform compilers only on combinational logic and data paths, and lose to compilers on timing control flow and clock domain optimization because LLMs fail to grasp clock-cycle…","keywords":["empirical study","large language models","RTL code optimization","metamorphic testing","timing logic","clock domain crossing","finite state machine","logic synthesis"],"falsifier":"A direct test would be to verify each mutant with a sequential equivalence checker that compares cycle-by-cycle behavior over all reachable states, then re-run the benchmark only on mutants proven equivalent; if LLM optimizers still degrade on those mutants, the paper's attribution stands, and if they do not, the reported gap is an artifact of changed timing.","tokens_in":18442,"feed_emoji":"⏱️","tokens_out":7152,"duration_ms":73146,"temperature":0.7,"pith_summary":"The paper asks whether large language models can genuinely optimize register-transfer-level (RTL) hardware code when the hard part is timing, not syntax. It introduces a metamorphosis evaluation: each RTL snippet is rewritten into a semantically equivalent but more complex mutant, and an optimizer is judged on whether its gains survive the rewrite. On logic operations and data paths, LLM-based optimizers, including RTLRewriter, GPT-4, and Claude-3.7-sonnet, match or beat the open-source synthesizer Yosys. On timing control flow and clock domain logic, they do not beat Yosys, and their results consistently degrade on the mutants. The paper attributes this to LLMs' weak grasp of clock-cycle semantics and offers the benchmark and metamorphosis protocol as a reusable way to test future optimizers.","feed_headline":"LLM RTL optimizers lose to compilers on timing-critical logic","feed_subtitle":"A metamorphosis benchmark shows LLMs handle logic and data paths but miss clock-cycle semantics.","key_machinery":"The central mechanism is the metamorphosis transformation set. For each optimization area, a rewrite rule produces a mutant intended to be functionally identical but harder to optimize: Boolean algebra rewrites and redundant terms for logic operations; always-true branches and cascaded multiplexers for data paths; inserted pass-through states and subdivided FSM states for timing control flow; and split synchronous clock domains with extra synchronizer registers for clock domains. The evaluation then normalizes every optimizer's synthesis result against Yosys's own synthesis of the same design and compares original versus mutant ratios. The load-bearing idea is symmetry: an optimizer that truly understands RTL should not lose effectiveness merely because the same function is expressed with more complex timing structure.","core_discovery":"The paper's central claim is a split verdict on LLM-based RTL optimization. For combinational-style code, including logic operations, data paths, and multiplexer structures, LLM-based methods optimize as well as or better than the compiler-based baseline, and their performance on metamorphosed mutants is close to their performance on the original code. For timing-intensive code, including finite state machine control flow and multi-clock-domain logic, the same methods fail to beat Yosys and show consistently worse normalized synthesis metrics, such as wires, cells, area, delay, and power, on mutants than on originals. The paper interprets this as evidence that LLMs do not yet reason about clock cycles and temporal constraints, and it supports that reading with an embedding experiment in which one timing-preserving mutation is placed farther from the original than some semantics-breaking mutations.","pith_inferences":["The state-machine and clock-domain metamorphoses insert pass-through states and synchronizer registers, which add clock cycles or latency; if those mutants are not cycle-by-cycle equivalent to the originals, the benchmark measures optimizer response to behaviorally different code rather than pure syntactic complexity.","A decisive follow-up would re-run the benchmark after sequential equivalence checking of each mutant, because the formal verification and simulation step the authors report does not by itself rule out cycle-count changes.","Because all ratios are normalized to Yosys's synthesis of the same design, the comparison conflates the optimizer's competence with Yosys's treatment of complex FSM and clock-domain constructs; a different synthesis ground truth could shift the relative rankings.","The embedding result suggests a predictive test: a temporal-reasoning probe on clock-edge expressions might identify which mutant constructs an LLM will misoptimize, giving a cheap way to extend the benchmark beyond the four current categories."],"forward_implications":["If the claim holds, current LLM-guided RTL rewriting should not be trusted for timing control flow or clock domain optimization without independent verification.","The metamorphosis protocol gives future evaluations a standard test: run any new LLM optimizer on original and mutant RTL and require that optimization quality does not drop on mutants.","In practice, LLM-based rewriting is appropriate for logic-operation and data-path portions, while timing-critical sections should stay with compiler passes or receive explicit timing-aware guidance.","Prompt content matters: the paper shows that simply instructing the LLM to remove non-executing or redundant code eliminates much of the data-path mutant gap, so zero-shot prompting understates achievable results.","Closing the timing gap will require giving LLMs clock-cycle semantics, for example explicit timing information in the prompt or a temporally aware front end, rather than relying on their text-based training."],"supporting_citations":[{"why":"Supplies the LLM-based RTL optimizer used as the main baseline and the zero-shot prompt template reused across all experiments.","marker":"[5]"},{"why":"Provides the Yosys compiler baseline, the synthesis ground truth for metric normalization, and the ABC solver used for formal equivalence checks.","marker":"[7]"},{"why":"Supplies Verilog benchmark cases that the new RTL optimization benchmark draws from, including logic, data-path, and FSM patterns.","marker":"[9]"},{"why":"Supplies additional RTL benchmark cases that seed the four benchmark subsets.","marker":"[10]"},{"why":"Inspires the metamorphosis methodology by evaluating semantic robustness through original versus semantically equivalent transformed programs.","marker":"[28]"},{"why":"Provides the metamorphosis-style evaluation template adapted here, comparing model performance on equivalent versus mutated inputs.","marker":"[29]"},{"why":"Provides the Icarus Verilog simulator used to check that each mutant behaves like the original RTL.","marker":"[30]"},{"why":"Motivates the embedding-distance analysis by assuming semantic similarity correlates with embedding distance in program space.","marker":"[25]"},{"why":"Supplies the CodeLlama model whose embeddings are used to show that LLMs misplace one timing-preserving mutation in semantic space.","marker":"[27]"}],"fun_headline_variants":["LLMs beat compilers on logic, lose on timing RTL","Timing logic defeats LLM RTL optimization","Metamorphosis benchmark shows LLM RTL timing blind spot","LLM optimizers improve RTL logic but miss clock semantics","LLMs fail to reason about clock cycles in RTL code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation stands on the claim that every mutant is semantically equivalent to its original, including cycle-by-cycle timing, because the inserted pass-through states and synchronizer registers may add latency and thereby change the behavior being optimized.","fun_headline_variants_meta":{"raw":{"variants":["LLMs beat compilers on logic, lose on timing RTL","Timing logic defeats LLM RTL optimization","Metamorphosis benchmark shows LLM RTL timing blind spot","LLM optimizers improve RTL logic but miss clock semantics","LLMs fail to reason about clock cycles in RTL code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1358,"prompt_tokens":1014,"completion_tokens":344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":260}},"tokens_in":630,"tokens_out":344,"duration_ms":3662,"temperature":1.0,"reasoning_tokens":260,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:00:41.167523+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to verify each mutant with a sequential equivalence checker that compares cycle-by-cycle behavior over all reachable states, then re-run the benchmark only on mutants proven equivalent; if LLM optimizers still degrade on those mutants, the paper's attribution stands, and if they do not, the reported gap is an artifact of changed timing.","supporting_citations":[{"cited_title":"Rtlrewriter: Methodologies for large models aided rtl code optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies the LLM-based RTL optimizer used as the main baseline and the zero-shot prompt template reused across all experiments."},{"cited_title":"Yosys-a free verilog synthesis suite,","cited_arxiv_id":null,"evidence_quote":"Provides the Yosys compiler baseline, the synthesis ground truth for metric normalization, and the ABC solver used for formal equivalence checks."},{"cited_title":"Semantic robustness of models of source code,","cited_arxiv_id":null,"evidence_quote":"Inspires the metamorphosis methodology by evaluating semantic robustness through original versus semantically equivalent transformed programs."},{"cited_title":"Icarus verilog,","cited_arxiv_id":null,"evidence_quote":"Provides the Icarus Verilog simulator used to check that each mutant behaves like the original RTL."},{"cited_title":"Exploiting code symmetries for learning program semantics,","cited_arxiv_id":null,"evidence_quote":"Motivates the embedding-distance analysis by assuming semantic similarity correlates with embedding distance in program space."}],"review_version":1}