{"id":"8ea0654e-9495-40e2-85f8-7cab396ecc7c","arxiv_id":"2607.20638","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLMs answer simple waveform queries well but fail on multi-signal temporal reasoning, and event-time JSON waveforms beat standard VCD by 37–53% in accuracy, according to a new 360-question benchmark.","lead":"WaveformQA is a new 360-question benchmark that measures whether large language models can answer precise timing questions about digital circuit simulation waveforms. Its main finding: models reason 37–53% better from structured JSON waveforms than from the industry-standard VCD format, and still largely fail on multi-signal timing and event-ordering questions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"WaveformQA's 360 labels rely on undocumented conventions for ordinal edges, simultaneous transitions, and glitch filtering; a human-validation sample is needed before the 37–53% JSON-over-VCD gain can be taken at face value.","rationale":"The reader's weakest assumption is exactly the load-bearing point: the benchmark's validity rests on the generator's ground-truth semantics matching the natural-language templates. If the ordinal-edge counting, simultaneous-edge handling, or four-state glitch conventions differ from what a verification engineer means by the question, every accuracy number, including the headline 37–53% format gain, is computed against potentially wrong labels. The paper gives credit for real open-source traces, deterministic seeding, arithmetic self-consistency, and a plausible taxonomy, but none of these independently validate the labels. The other concerns the reader raised — missing error bars, the signal-count confound, and the unstated error-decomposition method — affect how precisely the numbers can be interpreted, but they do not threaten the core finding as directly as label semantics. A human-validation sample with edge-case probes is a bounded, concrete check that would settle whether the concern actually lands. Until then, the verdict should remain conditional, not because the paper is suspected of error, but because the semantics are unspecified and unvalidated.","tokens_in":13462,"tokens_out":4585,"duration_ms":40902,"concrete_test":"Run a stratified human-validation sample: select 30 questions from the 360 (covering all 24 subcategories, plus extra edge-case questions), and have two verification engineers, blind to the generator labels, answer from the original VCD using only the natural-language prompt. Deliberately include windows whose boundaries coincide with signal transitions, clocks whose first edge is at t=0, simultaneous edges on different signals at the same timestamp, and traces containing X/Z glitches. Compare their answers with the programmatic ground truth. If any edge-case convention disagrees, revise the generator/templates and re-run the Section IV-A format comparison on corrected labels; if the 37–53% gain persists, the central claim is supported. Also commit the exact edge-case decision rules to the repository.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-C and Appendix VI-C describe a programmatic generator that computes ground truth directly from the trace for templates like 'the 5th rising edge of clk', 'last value before', 'before/after/at the same time', and 'ignoring glitches'. The paper verifies labels only internally, never against human annotation or an independent reference. The intended semantics of these phrases are not uniquely defined by the templates. Three concrete ambiguities: (1) ordinal edge counting near window boundaries and at t=0 — does a transition exactly at t_start or t_end count? is the initial value at t=0 considered an edge? (2) simultaneous transitions on different signals — the JSON representation groups changes by signal with only a timestamp, so 'before/after/at the same time' has no defined order for same-timestamp events; the generator's internal order may differ from a verification engineer's reading. (3) glitch filtering — the 'Ignore Glitch' template has no stated criterion for what counts as a transient in four-state (0/1/X/Z) logic, and FSM state-sequence templates depend on unspecified state-encoding conventions. Any mismatch between generator conventions and the semantics a human associates with the template shifts the ground-truth labels. Since every accuracy figure, format comparison, and difficulty ranking in the paper is computed against these labels, the central claim inherits this uncertainty. The reported internal programmatic verification checks self-consistency, not semantic validity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WaveformQA, a question-answering benchmark for LLM temporal reasoning over digital waveforms. 360 questions are generated from VCD traces of five open-source RISC-V cores, with ground truth computed programmatically from the traces in a deterministic, seeded manner. Questions span eight categories and 24 subcategories, and are stratified across a 3x5 complexity grid. The paper compares two event-based representations (raw VCD and a structured JSON format) on Gemini 2.5 Pro, and evaluates four LLMs on the JSON format. The main empirical claims are that JSON yields a 37-53% in-context accuracy gain over VCD at a 15-30% token overhead; aggregate accuracy is dominated by context-window size; in-context accuracy declines 8-12% as transition count grows from 5k to 30k but is not consistently affected by signal count; and temporal ordering/multi-step/correlation questions are the hardest for all evaluated models.","tokens_in":13637,"tokens_out":8857,"duration_ms":68512,"significance":"If the ground-truth labels are semantically valid, WaveformQA fills a genuine gap: existing hardware benchmarks focus on HDL generation/debugging rather than trace-level temporal reasoning. The benchmark design is attractive in several respects: labels are computed programmatically with deterministic seeds and independent of LLM behavior, so there is no circularity or parameter fitting; traces come from real open-source RISC-V implementations; and the code/data release supports extension. The reported representation effect is large and practically relevant for EDA tooling. However, the validity of every quantitative claim rests on the correctness of the generator's interpretation of natural-language templates, which is not yet sufficiently documented or externally validated.","major_comments":[{"comment":"The ground-truth generator computes answers directly from traces, but several semantic conventions that determine the labels are not specified and are not uniquely determined by the templates. (i) For ordinal-edge questions, it is not stated whether a transition exactly at t_start/t_end counts, whether the initial value at t=0 counts as an edge, or whether 'rising edge' in four-state logic includes X->1 or Z->1. (ii) For Before/After, Ordering, and Which Changes First/Last, the JSON representation stores per-signal changes only by timestamp; when two signals change at the same time no tie-break rule is given, leaving 'which changes last' and 'order by next change' ambiguous. (iii) 'Ignore Glitch' does not define what counts as a transient, and FSM state-sequence/next-state questions do not document how multi-bit state encodings map to displayed state names. Since every accuracy, format-c","section":"§III-C / Appendix VI-C (Table IV)"},{"comment":"The headline '37–53% accuracy gain' is measured on Gemini 2.5 Pro only and rests on 15 questions per subcategory. With n=15, a difference of three or four questions moves a subcategory percentage by 20–27 percentage points, and the 95% confidence interval for a proportion near 0.5 is roughly ±25 points. The Limitations paragraph itself concedes 'limited statistical power per subcategory.' The paper should report confidence intervals or bootstrap intervals, provide a per-model format comparison, and qualify Key takeaway 1 accordingly; the current claim implies a broader model-independent conclusion that the data do not yet support.","section":"§IV-A, Fig. 3(b); Limitations"},{"comment":"The category-level difficulty analysis uses in-context accuracy computed only on questions that did not exceed the context window. At higher transition thresholds, only the 1M-context models contribute, and the contribution is unbalanced across models and categories. Key takeaway 3 ('Temporal, Multistep, and Correlation are consistently the hardest across all models') should therefore be re-examined on the intersection of bins where all models have answerable questions, or at least reported with the number of questions (n) underlying each point. Without this, the pattern may be an artifact of which questions remain answerable after context truncation rather than a stable property of the reasoning categories.","section":"§IV-C, Figs. 6–7"}],"minor_comments":[{"comment":"Please clarify whether 'VCD errors higher in 20 of 24 subcategories by more than 20%' means an absolute difference of >20 percentage points or a relative increase of >20%; the two readings give very different impressions.","section":"§IV-A, Fig. 2"},{"comment":"The table lists prompt templates but not the parameter-sampling procedure (how signals are selected, how time points are sampled, how ordinals are chosen). Publishing this logic or adding pseudocode would help reproducibility and make the semantic conventions easier to audit.","section":"Appendix VI-C"},{"comment":"Report exact context-exceeded counts per model in addition to percentages; the percentages in the text round to values that do not exactly multiply 360.","section":"Fig. 4"},{"comment":"There are numerous typographical/formatting artifacts in the PDF text (e.g., 'difficulty', 'A verage', 'Cla(de Sonnet'); these should be cleaned before camera-ready.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is a useful contribution and the programmatic ground-truth design is sound in principle. My recommendation is driven by two load-bearing gaps: (1) the generator's semantics for edges, simultaneous events, glitches, and FSM states are under-specified and unvalidated, and (2) the main empirical claims are supported by very small per-subcategory samples and a single model for the format comparison. These are fixable within the manuscript's scope (validation subset, CIs, qualified claims), so I do not see grounds for rejection. The open-source release will make the requested validation straightforward."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nHere's the short version: WaveformQA is the first benchmark I've seen that systematically probes LLM temporal reasoning directly on digital waveforms, and it does that well. The authors generate 360 questions with programmatically computed ground truth from real RISC-V traces, use deterministic seeding, and stratify across a 3×5 complexity grid. The headline result—event-time JSON beats raw VCD by 37–53% at a 15–30% token cost—is plausible and internally consistent. The model-level numbers cross-check arithmetically, and the paper is honest about its main limitation: 15 questions per subcategory caps statistical power.\n\nThe framework is genuinely useful. The 8-category taxonomy covers the kinds of tasks verification engineers actually do (counting edges, ordering events, correlating signals, FSM states). Using real open-source cores rather than synthetic toy traces is a plus. And the decision to treat context-exceeded answers separately from in-context accuracy is the right call—it's why the conclusion that transition count, not signal count, drives difficulty is credible.\n\nNow the soft spots, in rough proportion.\n\nFirst, the semantic validity concern is real. The paper describes a programmatic generator that computes ground truth for phrases like \"the 5th rising edge\" and \"ignoring glitches,\" but it never documents the boundary conventions—whether a transition at t_start counts, how simultaneous edges on different signals are ordered, what counts as a glitch in four-state logic. The stress-test note flags exactly this. I read the relevant sections and the ambiguity is there. It's not fatal: the labels are internally consistent by construction, and the benchmark would still rank models on the same tasks. But the 37–53% gain could shift if a human verification engineer reads some of these templates differently. A small human-validation sample—even 30–50 questions—would firm this up substantially.\n\nSecond, the statistics are thin. Cells of 15 questions mean a 3–4 answer difference moves a subcategory by 20+ points. The paper reports no confidence intervals or significance tests. Also, the 15 bins are non-independent re-windows of 13 traces, so the effective sample size is smaller than 360 suggests. The authors concede the statistical power point in the limitations, but the headline claim still deserves error bars.\n\nThird, the error decomposition in Figure 7 (wrong value vs. wrong signal vs. wrong order) is presented without stating the method. How were those categories assigned? That needs to be in the appendix or the code.\n\nFourth, a citation mismatch: the text says \"SV A-Eval [2]\" but reference [2] is AssertLLM. Minor, but sloppy.\n\nBottom line: this paper deserves a serious referee. It fills a real gap, the design is sound, and the limitations are mostly stated. The fixes are not structural. I'd send it to review with a request for the repository (commit-pinned), a documented semantics appendix, and a human-validation pilot.\n\nBest,\n\n[Your name]","headline":"WaveformQA fills a real gap—a systematic waveform temporal-reasoning benchmark—but the headline JSON-vs-VCD numbers rest on thin per-cell statistics and undocumented answer semantics.","tokens_in":14283,"tokens_out":1650,"would_cite":true,"duration_ms":15659,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WaveformQA shows LLMs answer waveform questions 37–53% more accurately from event-time JSON than from raw VCD traces.","keywords":["digital waveforms","temporal reasoning","LLM benchmark","VCD","event-time JSON","design verification","RISC-V traces","question answering"],"falsifier":"Have verification engineers independently answer a sample of the 360 questions from the same traces and compare their answers to the programmatic ground truth; if agreement is not near-perfect on subtle phrases like 'ignoring glitches' or 'the 5th rising edge,' the benchmark labels—and all format and difficulty comparisons built on them—would need revision.","tokens_in":13199,"feed_emoji":"⏱️","tokens_out":2992,"duration_ms":25659,"temperature":0.7,"pith_summary":"The paper introduces WaveformQA, a benchmark of 360 question-answer pairs built from simulated RISC-V processor waveforms, designed to measure whether LLMs can do precise temporal reasoning over digital signals. Its central empirical claim is that the same waveform presented as structured event-time JSON supports 37–53% higher accuracy than the standard VCD trace format, at a modest 15–30% token cost. The benchmark also establishes that in-context accuracy declines 8–12% as transition count grows from 5k to 30k, while signal count has little effect, and that event-ordering and multi-signal correlation are the hardest question types. If these results hold, the choice of waveform representation and context management are first-order levers for practical LLM-based waveform analysis, not just model selection.","feed_headline":"JSON waveforms beat raw VCD for LLM questions by 37–53%","feed_subtitle":"A 360-question temporal-reasoning benchmark shows representation choice can matter as much as model scale.","key_machinery":"The central comparison object is the event-time JSON waveform representation versus the standardized VCD format. VCD is compact, delta-encoded, and maps readable signal names to single-character identifiers, forcing the model to reconstruct signal state from sparse changes. The JSON representation groups each signal's changes with explicit naming, bit width, and hex values. This representational difference is the independent variable that carries the paper's main empirical argument, and the 360-question benchmark with programmatic ground truth is the measuring instrument behind all accuracy, scaling, and difficulty claims.","core_discovery":"WaveformQA is presented as the first benchmark specifically for LLM temporal reasoning over digital waveform traces, containing 360 programmatically verified questions across eight categories and 24 subcategories, generated from open-source RISC-V core simulations. The headline finding is representation-dependent: converting the same traces from VCD (IEEE 1364) to an event-time JSON that names signals explicitly, groups changes per signal, and carries width and radix metadata raises in-context accuracy by 37–53 percentage points on Gemini 2.5 Pro, despite using 15–30% more tokens. The authors attribute the gain to JSON's prevalence in pretraining corpora and to VCD's cryptic single-character","pith_inferences":["If the format gap generalizes beyond RISC-V traces, converting existing VCD dumps to structured JSON could be a cheap, high-impact preprocessing step in verification tools that use LLMs.","The error decomposition suggests agentic systems that first expand signal names and pre-index change times could close much of the remaining gap without waiting for larger models.","Because the benchmark's 360 questions are deterministically generated from open traces, it can be extended to visual waveform screenshots or time-grouped JSON, offering a controlled testbed for studying how modality and representation interact.","The near-total failure on the Ordering subcategory hints that current LLMs may not be performing genuine timeline reconstruction; counterfactual traces that require reading the whole timeline could reveal whether they actually process all transitions."],"forward_implications":["Event-time JSON is a more reliable input format for LLM waveform question answering than raw VCD, despite costing 15–30% more tokens.","Longer traces degrade reasoning even when they fit inside the context window, so context management and transition-count reduction are core problems for practical use.","Adding more signals does not consistently hurt accuracy, meaning token budget—not search width—is the main cost of scaling to larger designs.","LLMs are weakest at event ordering and multi-signal identification, not at extracting numeric values, pointing to a specific reasoning deficit.","Context-window overflow, not incorrect reasoning, is the dominant failure mode for smaller-context models on full traces."],"fun_headline_variants":["WaveformQA: LLMs fail temporal reasoning on raw VCD traces","JSON format boosts LLM waveform accuracy by up to 53 points","New benchmark: LLMs need JSON, not VCD, for chip timing questions","Study: LLM waveform reasoning depends heavily on data format","WaveformQA: 360-question test exposes LLM temporal reasoning limits"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's ground-truth answers are generated programmatically by the question generator, and if its counting of ordinal edges, handling of simultaneous transitions, or interpretation of phrases like 'ignoring glitches' differs from what a human verification engineer means by the question, every reported accuracy number shifts.","fun_headline_variants_meta":{"raw":{"variants":["WaveformQA: LLMs fail temporal reasoning on raw VCD traces","JSON format boosts LLM waveform accuracy by up to 53 points","New benchmark: LLMs need JSON, not VCD, for chip timing questions","Study: LLM waveform reasoning depends heavily on data format","WaveformQA: 360-question test exposes LLM temporal reasoning limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000835,"raw_usage":{"total_tokens":3473,"prompt_tokens":733,"completion_tokens":2740,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":2645}},"tokens_in":477,"tokens_out":2740,"duration_ms":17456,"temperature":1.0,"reasoning_tokens":2645,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T09:46:28.184437+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have verification engineers independently answer a sample of the 360 questions from the same traces and compare their answers to the programmatic ground truth; if agreement is not near-perfect on subtle phrases like 'ignoring glitches' or 'the 5th rising edge,' the benchmark labels—and all format and difficulty comparisons built on them—would need revision.","supporting_citations":[],"review_version":1}