{"id":"be11c682-b956-4700-bd85-e7115d320604","arxiv_id":"2608.12635","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A mutation-testing audit of RTL benchmark testbenches finds RTLLM v2.0 mostly below a 95% fault-kill floor and shows that an output-token cap can reorder a leaderboard.","lead":"This paper describes GateTruth, a mutation-testing tool that injects deliberate bugs into chip-design reference code and checks whether benchmark testbenches catch them. Auditing its own suite and the RTLLM benchmark, it reports that most RTLLM testbenches miss injected faults and that output-token caps can change model rankings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The below-floor count and 74.0% median assume surviving mutants are genuine faults; without formal equivalent-mutant filtering, the threshold comparison to the 95% floor is only a lower bound.","rationale":"The reader's weakest assumption bundles the Icarus oracle, module-alias normalization, and the absence of formal equivalent-mutant filtering. I agree with the reader's CONDITIONAL verdict, but the single most load-bearing unresolved issue is equivalent-mutant filtering, because the paper's central comparison is threshold-based: the reported 72% below-floor figure is a lower bound on true testbench sensitivity, and lower bounds do not directly support threshold claims. The paper's own Section 4 supplies concrete evidence that hand-reasoned equivalence is unreliable, and Section 10 identifies MCY as the principled alternative, so the concern is internally grounded rather than imported from outside consensus. The proposed check is feasible with existing open-source tooling and would settle whether the headline count survives formal filtering. Since the reader already conditioned acceptance on related oracle concerns, my read does not move the verdict; it sharpens which condition must be tested first.","tokens_in":34073,"tokens_out":3406,"duration_ms":39135,"concrete_test":"Run formal equivalence checking on all 320 surviving RTLLM v2.0 mutants from the -g2012 sweep, using MCY (YosysHQ mutation cover) or Yosys equiv_make/equiv_induct against each design's golden reference; exclude mutants proven equivalent from the denominator, recompute per-design kill rates, the median, and the below-95% count. If the 33-of-46 count and 74.0% median survive this exclusion, the concern is resolved; if several designs cross the 95% floor, the headline claim must be re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The external audit's central finding—33 of 46 audited RTLLM v2.0 designs below the 95% floor, with a 74.0% median kill rate—depends on treating every surviving mutant as a genuine fault that a stronger testbench should catch. GateTruth applies no formal equivalent-mutant filtering (Section 5, 'Operator set'; Section 9, 'Mutant generation, equivalence, and the role of the seed'). A behaviorally equivalent mutant cannot be killed by any testbench, so including such mutants in the denominator artificially depresses kill rates. The paper calls its rates 'conservative lower bounds,' but the headline is a threshold claim: a lower bound below 95% does not establish that the true kill rate is below 95%. The three 0% designs are robust because the paper manually inspected their surviving mutants and found inverted outputs for square_wave and adder_8bit, but for the remaining below-floor designs survivor equivalence is unknown. This is not a hypothetical risk: Section 4 documents that GateTruth's own hand-authored equivalence-exclusion table was falsified for 36 of 72 excluded mutants when actually tested, and Section 10 cites MCY's formal equivalence filtering as 'a more principled solution' to exactly this problem. If a substantial fraction of the 320 survivors in the -g2012 sweep are equivalent, the median and the 33-of-46 count could shift, with some designs potentially clearing the 95% floor.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GateTruth, a mutation-testing engine and methodology for auditing the rigor of RTL benchmark testbenches. It validates the methodology on a self-authored 68-task, dual-track reference suite, certifying 46 of 60 Track A testbenches at or above a 95% mutation-kill floor after two disclosed corrections, then applies the same engine to the external RTLLM v2.0 benchmark, reporting a median kill rate of 74.0% across 46 audited designs, with 33 of 46 below the 95% floor and three designs at 0%, including two with manually confirmed inverted outputs. The paper also reports that the CVDP public release cannot be audited because golden references are withheld, and that changing an output-token cap from 4096 to 16,384 tokens coincided with a leaderboard reordering. The manuscript is notable for its unusually extensive self-disclosure, including a fail-open bug in its own harness, a falsified equivalence-exclusion table, and unreproducible Track B submission files.","tokens_in":34311,"tokens_out":6436,"duration_ms":64841,"significance":"If the external audit result holds, this is a significant contribution to RTL benchmark methodology: it demonstrates a concrete, reproducible way to measure testbench fault-detection quality and provides the first public evidence that a widely used RTL-generation benchmark's testbenches are uneven, with a substantial fraction below a 95% kill-rate floor and three designs at 0%. The paper's self-audit practices—reporting two corrections to its own certification (60/60 to 49/60 to 46/60), disclosing a fail-open bug, publishing per-design JSON results, and running deterministic same-seed re-checks—are exemplary and materially increase confidence in the instrumentation. The external audit is also independent of fitted parameters, using a pinned vendor snapshot and a public testbench, which is a further strength. The significance would be higher still if the central threshold claim were not qualified by the absence of formal equivalent-mutant filtering.","major_comments":[{"comment":"The headline result—median kill rate 74.0%, 33 of 46 RTLLM v2.0 designs below the 95% floor, three at 0%—is computed without formal equivalent-mutant filtering. As Section 9 states, 'We do not detect or exclude equivalent mutants,' and the paper itself calls the rates 'a conservative lower bound on true sensitivity.' A lower bound below 95% does not establish that the true kill rate is below 95%; if some of the 320 surviving mutants in the -g2012 sweep are behaviorally equivalent to the reference, no testbench can kill them, and their presence in the denominator depresses the reported rates. The three 0% designs are supported by manual inspection of the surviving mutants, but the other 30 below-floor designs are not individually triaged. This is not a hypothetical risk: Section 4 documents that GateTruth's own hand-authored equivalence-exclusion table was falsified for 36 of 72 excluded mutants when tested, and Section 10 identifies MCY-style formal equivalence filtering as 'a more principled solution' and defers it to future work. I recommend either filtering equivalent mutants (or a sample of survivors) before computing the threshold comparison, or re-framing the central claim as a measured lower bound under the generic operator set rather than a certification that RTLLM testbenches fail the 95% floor.","section":"Section 6 (Table 1) and Section 9 ('Mutant generation, equivalence, and the role of the seed')"},{"comment":"The external audit's verdict oracle is a single simulator (Icarus Verilog 12.0 under -g2012) combined with an exact, whole-line pass-banner match, and Section 9 explicitly states that kills and survivors are not manually audited for simulator artifacts such as uninitialized-signal propagation. The paper does not cross-check the aggregate result under a second simulator; the only cross-simulator scoping is for radix2_div's baseline failure. If Icarus's event-ordering or banner matching misclassifies even a modest fraction of mutants, the 74.0% median and the 33-of-46 count would shift. The authors should either run a sample of designs under a second simulator (or a second oracle mode) and report the agreement rate, or present the headline as 'measured under Icarus Verilog 12.0 with exact banner matching' rather than as an unqualified property of the testbenches.","section":"Section 5 ('Verdict oracle') and Section 6"}],"minor_comments":[{"comment":"The phrase 'opt-inofficial flag' appears to be a typographical error for 'opt-in official flag' or 'opt-in, official flag.'","section":"Section 2"},{"comment":"In the discussion of multi_8bit, 'which-g2001 reports as a baresyntax error' should read 'which -g2001 reports as a bare syntax error.'","section":"Section 6"},{"comment":"The row label 'GateT ruth' is a typo and should read 'GateTruth.'","section":"Table 5"},{"comment":"The sentence 'the other 14 do not, a finding in its own right, not a caveat we omit here' is grammatically awkward; consider rephrasing for clarity.","section":"Section 6"},{"comment":"The term 'the same scepticism' uses British spelling; if the journal style is American English, this should be 'skepticism.'","section":"Section 9"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is unusually transparent and well-scoped, and the disclosure culture is a genuine strength. My main concern is that the headline external-audit claim is stated more strongly than the methodology currently supports: the absence of equivalent-mutant filtering is acknowledged in Section 9, but the threshold language in Section 6 does not carry that qualification into the central 33-of-46 and 74.0% figures. The Track B reproducibility gap (no retained submission files for any of the 56 runs) is also worth the editor's attention, though it is secondary to the external audit. I would be comfortable with acceptance after the central claim is either re-framed as a lower-bound measurement under a stated oracle condition or backed by a sample-based equivalence check on the surviving mutants."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is the first paper I've seen that points a mutation-testing engine at existing public LLM RTL-generation benchmarks and reports measured kill rates. That's a real, reusable contribution. Second, the paper is unusually honest: it reports two corrections that dropped its own certification from a false 60/60 to 46/60, discloses a fail-open bug, and commits per-design JSON results.\n\nThe external audit of RTLLM v2.0 is independent and pinned: vendor snapshot, fixed seed, sequential execution, byte-identical sample reruns. The three 0% designs were manually inspected, and the surviving faults include inverted primary outputs. That's strong evidence. The token-cap sensitivity rerun is suggestive, and the three confounds are disclosed.\n\nThe main soft spot is the framing of the headline count. The engine applies no formal equivalent-mutant filtering. The paper itself notes that surviving mutants may be behaviorally equivalent and that kill rates are conservative lower bounds. But the abstract and Section 6 present \"33 of 46 below 95%\" as a threshold fact. A lower bound below 95% does not establish that the true kill rate is below 95%. If a substantial fraction of the 320 survivors are equivalent, some of those 33 designs could clear the floor. The paper's own history shows hand-reasoned equivalence is unreliable—36 of 72 excluded mutants were later killed. So the count is real as measured, but the policy conclusion should be consistently stated as \"at least 33 of 46 below the floor under the unfiltered metric.\" Because the same unfiltered metric is applied to their own suite, the comparison is internally consistent; still, the absolute numbers would shift with equivalence filtering.\n\nOther soft spots are disclosed and mostly minor: single-simulator oracle, exact pass-banner match, module-name aliases for 26 of 46 designs, and Track B permanently unreproducible because submission sources were deleted. None of these change my view that the central finding is credible.\n\nWho this is for: anyone building or evaluating RTL-generation benchmarks, and anyone concerned with benchmark rigor generally. The paper deserves a serious referee. I'd recommend sending it to review with a request that the framing be tightened—either consistently re-state the headline as a lower bound, or better, run a sensitivity analysis on the survivor set with formal equivalence filtering.","headline":"A genuinely new mutation-testing audit of external RTL benchmarks, with unusually honest self-disclosure, but the headline below-floor count is a lower bound that needs clearer framing.","tokens_in":34892,"tokens_out":4221,"would_cite":true,"duration_ms":43178,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GateTruth injects seeded faults into RTL reference designs and shows that most RTLLM v2.0 testbenches fail to catch basic errors, with three catching nothing at all.","keywords":["mutation testing","RTL design benchmarks","testbench rigor","LLM hardware generation","GateTruth","RTLLM v2.0","kill rate","output token budget"],"falsifier":"Re-run the 320 surviving RTLLM mutants under a second simulator or an assertion-based oracle and check the inverted-output survivors in adder_8bit and square_wave: if those mutants fail under the alternative oracle, the 0% kill rates are artifacts of banner matching; if they still pass, the finding stands. Substituting a different generic operator set and observing whether the median and below-floor counts move materially would test representativeness.","tokens_in":33805,"feed_emoji":"🧪","tokens_out":8062,"duration_ms":70926,"temperature":0.7,"pith_summary":"GateTruth is a mutation-testing audit: inject deterministic semantic faults into a reference RTL design and measure what fraction the benchmark's testbench catches. The paper uses it to show that a passing testbench is not evidence of a correct design, and that widely used RTL-generation benchmarks may overstate model quality. It certifies 46 of its own 60 testbenches at or above a 95% kill floor, then points the same engine at RTLLM v2.0: median kill rate 74%, 33 of 46 auditable designs below the floor, three at 0%, including undetected inverted outputs. It also finds that an output-token cap can reorder a leaderboard, and concludes that mutation-kill certification should be standard reporting for RTL benchmarks.","feed_headline":"Mutation audit: most RTLLM testbenches miss basic injected faults","feed_subtitle":"Across 46 designs the median kill rate is 74%; three testbenches catch 0% of injected faults.","key_machinery":"The carrying mechanism is the GateTruth mutation engine: a deterministic, seeded generator that injects a fixed set of semantic operators into a golden RTL reference, including comparator-boundary flips, operator/logic/bitwise inversions, shift-direction inversions, reset and enable polarity flips, assignment deletion and hold, and output inversion, then scores how many the testbench kills under sequential Icarus Verilog simulation. The verdict oracle requires baseline validation before mutation, counts stillborn (non-compiling) mutants out of the denominator, separates formal-only kills from simulation kills, treats timeouts as indeterminate and charges them against the rate, and demands byte-identical re-runs for certification. For RTLLM, PASS is an exact whole-line pass-banner match; for GateTruth's own suite, assertion-based cocotb checks carry the verdict. This engine is what makes testbench rigor auditable rather than assumed.","core_discovery":"The central claim is that the rigor of an RTL benchmark's testbenches is measurable and, when measured, often poor. Applying the GateTruth mutation engine to RTLLM v2.0's 46 auditable designs yields a median kill rate of 74.0%, 33 of 46 designs below the 95% floor the paper holds its own suite to, and three designs at 0%; in two of those, an inverted primary output survives. The same engine certifies 46 of the paper's own 60 Track A testbenches at or above the floor, with 14 disclosed below it after two corrections. The paper also argues that the CVDP benchmark cannot be mutation-audited from its public release because golden reference solutions are withheld, and that an output-token cap is a first-order experimental variable capable of reordering a leaderboard.","pith_inferences":["Because the 0% designs include faults as basic as an inverted primary output, some highly cited RTL benchmark tasks may be near-vacuous for specific output signals, and per-design kill rates should not be averaged away.","If mutation-based gating spreads, the Goodhart pattern documented here suggests future benchmarks should report pre-revision scores and use formal equivalence filtering to prevent exclusion-driven inflation of kill rates.","The gap between the 74.0% per-design median and the 56.8% pooled kill rate implies that design-level risk is concentrated in larger designs, which are often the very designs used to separate models.","A natural extension would audit other public RTL benchmarks with the same generic operator set and a second simulator, to test whether these kill rates are simulator-specific."],"forward_implications":["If the audit is right, RTLLM v2.0's pass@k numbers overstate functional correctness on designs whose testbenches cannot catch even inverted outputs.","A 95% mutation-kill floor is achievable for most, but not all, authored testbenches; the 14 sub-floor tasks show the floor is not trivial.","Output-token budgets should be reported and controlled in any RTL benchmark, because an unreported cap mixes verbosity with capability in a leaderboard.","Public benchmarks that withhold golden references become unauditable by mutation testing, so auditability should be a design criterion for new benchmarks.","Mutation-kill certification, if adopted as a standard, would give users a per-design quality number instead of a single aggregate score."],"supporting_citations":[{"why":"Supplies the 50 public RTLLM v2.0 designs whose shipped testbenches and references are the external audit target.","marker":"[9, 8]"},{"why":"Defines the CVDP public release whose withheld reference solutions make mutation auditing structurally impossible, reported as a gap.","marker":"[10]"},{"why":"Introduces mutation testing, the method GateTruth re-purposes to audit benchmark testbenches.","marker":"[4]"},{"why":"Prior application of mutation analysis to qualify hardware testbench quality, supporting the method's validity in RTL contexts.","marker":"[5]"},{"why":"Derives RTL mutation operators from real reported bugs, giving a precedent for the operator set used in the audit.","marker":"[18]"},{"why":"Demonstrates formal equivalence checking to filter equivalent mutants, the alternative to the simulation-only kill rate GateTruth reports.","marker":"[19]"},{"why":"Uses mutation detection as a reinforcement-learning reward, independent evidence that mutation-based fault detection is a useful signal.","marker":"[6]"},{"why":"Establishes the pass@k-plus-testbench evaluation recipe for RTL generation that the paper argues lacks mutation rigor certification.","marker":"[7, 11]"}],"fun_headline_variants":["GateTruth: Mutation audit finds RTLLM testbenches mostly weak","Mutation testing exposes RTLLM benchmarks: median kill rate 74%","Three RTLLM testbenches catch zero injected faults","GateTruth audit: 72% of RTLLM designs fail 95% kill floor"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire audit stands on the assumption that the Icarus-based verdict oracle, which requires the baseline to pass and then matches an exact whole-line pass banner, faithfully represents what a testbench really catches, and that the fixed generic operator set is representative of the faults that matter.","fun_headline_variants_meta":{"raw":{"variants":["GateTruth: Mutation audit finds RTLLM testbenches mostly weak","Mutation testing exposes RTLLM benchmarks: median kill rate 74%","Three RTLLM testbenches catch zero injected faults","GateTruth audit: 72% of RTLLM designs fail 95% kill floor"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000561,"raw_usage":{"total_tokens":2733,"prompt_tokens":1079,"completion_tokens":1654,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":1574}},"tokens_in":695,"tokens_out":1654,"duration_ms":10864,"temperature":1.0,"reasoning_tokens":1574,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:03:43.251141+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 320 surviving RTLLM mutants under a second simulator or an assertion-based oracle and check the inverted-output survivors in adder_8bit and square_wave: if those mutants fail under the alternative oracle, the 0% kill rates are artifacts of banner matching; if they still pass, the finding stands. Substituting a different generic operator set and observing whether the median and below-floor counts move materially would test representativeness.","supporting_citations":[{"cited_title":"Functional testbench qualification by mutation analysis.VLSI Design, 2015:256474, 2015","cited_arxiv_id":null,"evidence_quote":"Prior application of mutation analysis to qualify hardware testbench quality, supporting the method's validity in RTL contexts."},{"cited_title":"Introduction to Mutation Coverage with Yosys (MCY)","cited_arxiv_id":null,"evidence_quote":"Demonstrates formal equivalence checking to filter equivalent mutants, the alternative to the simulation-only kill rate GateTruth reports."},{"cited_title":"GRPO with state mutations: Improving LLM-based hardware test plan generation.arXiv preprint arXiv:2601.07593, 2026","cited_arxiv_id":null,"evidence_quote":"Uses mutation detection as a reinforcement-learning reward, independent evidence that mutation-based fault detection is a useful signal."}],"review_version":1}