REVIEW 3 major objections 4 minor 32 references
WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveforms
T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read WaveformQA shows LLMs answer waveform questions 37–53% more accurately from event-time JSON than from raw VCD traces.
desk verdict WaveformQA fills a real gap—a systematic waveform temporal-reasoning benchmark—but the headline JSON-vs-VCD numbers rest on thin per-cell statistics and undocumented answer semantics. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central comparison object is the event-time JSON waveform representation versus the standardized VCD format. VCD is compact, delta-encoded, and maps readable signal names to single-character identifiers, forcing the model to reconstruct signal state from sparse changes. The JSON representation groups each signal's changes with explicit naming, bit width, and hex values. This representational difference is the independent variable that carries the paper's main empirical argument, and the 360-question benchmark with programmatic ground truth is the measuring instrument behind all accuracy, scaling, and difficulty claims.
What would settle it
Have verification engineers independently answer a sample of the 360 questions from the same traces and compare their answers to the programmatic ground truth; if agreement is not near-perfect on subtle phrases like 'ignoring glitches' or 'the 5th rising edge,' the benchmark labels—and all format and difficulty comparisons built on them—would need revision.
Extended reading notes
Core claim
WaveformQA is presented as the first benchmark specifically for LLM temporal reasoning over digital waveform traces, containing 360 programmatically verified questions across eight categories and 24 subcategories, generated from open-source RISC-V core simulations. The headline finding is representation-dependent: converting the same traces from VCD (IEEE 1364) to an event-time JSON that names signals explicitly, groups changes per signal, and carries width and radix metadata raises in-context accuracy by 37–53 percentage points on Gemini 2.5 Pro, despite using 15–30% more tokens. The authors attribute the gain to JSON's prevalence in pretraining corpora and to VCD's cryptic single-character
Load-bearing premise
The benchmark's ground-truth answers are generated programmatically by the question generator, and if its counting of ordinal edges, handling of simultaneous transitions, or interpretation of phrases like 'ignoring glitches' differs from what a human verification engineer means by the question, every reported accuracy number shifts.
Editorial extensions
If this is right
- Event-time JSON is a more reliable input format for LLM waveform question answering than raw VCD, despite costing 15–30% more tokens.
- Longer traces degrade reasoning even when they fit inside the context window, so context management and transition-count reduction are core problems for practical use.
- Adding more signals does not consistently hurt accuracy, meaning token budget—not search width—is the main cost of scaling to larger designs.
- LLMs are weakest at event ordering and multi-signal identification, not at extracting numeric values, pointing to a specific reasoning deficit.
- Context-window overflow, not incorrect reasoning, is the dominant failure mode for smaller-context models on full traces.
Reading between the lines
- If the format gap generalizes beyond RISC-V traces, converting existing VCD dumps to structured JSON could be a cheap, high-impact preprocessing step in verification tools that use LLMs.
- The error decomposition suggests agentic systems that first expand signal names and pre-index change times could close much of the remaining gap without waiting for larger models.
- Because the benchmark's 360 questions are deterministically generated from open traces, it can be extended to visual waveform screenshots or time-grouped JSON, offering a controlled testbed for studying how modality and representation interact.
- The near-total failure on the Ordering subcategory hints that current LLMs may not be performing genuine timeline reconstruction; counterfactual traces that require reading the whole timeline could reveal whether they actually process all transitions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces WaveformQA, a question-answering benchmark for LLM temporal reasoning over digital waveforms. 360 questions are generated from VCD traces of five open-source RISC-V cores, with ground truth computed programmatically from the traces in a deterministic, seeded manner. Questions span eight categories and 24 subcategories, and are stratified across a 3x5 complexity grid. The paper compares two event-based representations (raw VCD and a structured JSON format) on Gemini 2.5 Pro, and evaluates four LLMs on the JSON format. The main empirical claims are that JSON yields a 37-53% in-context accuracy gain over VCD at a 15-30% token overhead; aggregate accuracy is dominated by context-window size; in-context accuracy declines 8-12% as transition count grows from 5k to 30k but is not consistently affected by signal count; and temporal ordering/multi-step/correlation questions are the hardest for all evaluated models.
Significance. If the ground-truth labels are semantically valid, WaveformQA fills a genuine gap: existing hardware benchmarks focus on HDL generation/debugging rather than trace-level temporal reasoning. The benchmark design is attractive in several respects: labels are computed programmatically with deterministic seeds and independent of LLM behavior, so there is no circularity or parameter fitting; traces come from real open-source RISC-V implementations; and the code/data release supports extension. The reported representation effect is large and practically relevant for EDA tooling. However, the validity of every quantitative claim rests on the correctness of the generator's interpretation of natural-language templates, which is not yet sufficiently documented or externally validated.
major comments (3)
- [§III-C / Appendix VI-C (Table IV)] The ground-truth generator computes answers directly from traces, but several semantic conventions that determine the labels are not specified and are not uniquely determined by the templates. (i) For ordinal-edge questions, it is not stated whether a transition exactly at t_start/t_end counts, whether the initial value at t=0 counts as an edge, or whether 'rising edge' in four-state logic includes X->1 or Z->1. (ii) For Before/After, Ordering, and Which Changes First/Last, the JSON representation stores per-signal changes only by timestamp; when two signals change at the same time no tie-break rule is given, leaving 'which changes last' and 'order by next change' ambiguous. (iii) 'Ignore Glitch' does not define what counts as a transient, and FSM state-sequence/next-state questions do not document how multi-bit state encodings map to displayed state names. Since every accuracy, format-c
- [§IV-A, Fig. 3(b); Limitations] The headline '37–53% accuracy gain' is measured on Gemini 2.5 Pro only and rests on 15 questions per subcategory. With n=15, a difference of three or four questions moves a subcategory percentage by 20–27 percentage points, and the 95% confidence interval for a proportion near 0.5 is roughly ±25 points. The Limitations paragraph itself concedes 'limited statistical power per subcategory.' The paper should report confidence intervals or bootstrap intervals, provide a per-model format comparison, and qualify Key takeaway 1 accordingly; the current claim implies a broader model-independent conclusion that the data do not yet support.
- [§IV-C, Figs. 6–7] The category-level difficulty analysis uses in-context accuracy computed only on questions that did not exceed the context window. At higher transition thresholds, only the 1M-context models contribute, and the contribution is unbalanced across models and categories. Key takeaway 3 ('Temporal, Multistep, and Correlation are consistently the hardest across all models') should therefore be re-examined on the intersection of bins where all models have answerable questions, or at least reported with the number of questions (n) underlying each point. Without this, the pattern may be an artifact of which questions remain answerable after context truncation rather than a stable property of the reasoning categories.
minor comments (4)
- [§IV-A, Fig. 2] Please clarify whether 'VCD errors higher in 20 of 24 subcategories by more than 20%' means an absolute difference of >20 percentage points or a relative increase of >20%; the two readings give very different impressions.
- [Appendix VI-C] The table lists prompt templates but not the parameter-sampling procedure (how signals are selected, how time points are sampled, how ordinals are chosen). Publishing this logic or adding pseudocode would help reproducibility and make the semantic conventions easier to audit.
- [Fig. 4] Report exact context-exceeded counts per model in addition to percentages; the percentages in the text round to values that do not exactly multiply 360.
- [General] There are numerous typographical/formatting artifacts in the PDF text (e.g., 'difficulty', 'A verage', 'Cla(de Sonnet'); these should be cleaned before camera-ready.
Circularity Check
No significant circularity: the benchmark's ground truths are computed from traces independently of the models, and the JSON-vs-VCD comparison is an empirical measurement, not a fitted or self-referential result.
full rationale
WaveformQA's central claims are empirical benchmark results, not derivations from fitted inputs. The 360 ground-truth labels are produced by a programmatic question generator that computes answers directly from the waveform traces (Section III-C: 'computes ground truth directly from the trace, and instantiates a natural-language question via Python f-strings'), independent of any LLM behavior. No parameter is fitted to the evaluation data, and no 'prediction' is constructed so as to equal its input by definition. The headline result—that event-time JSON outperforms VCD by 37–53%—is an observed accuracy difference on Gemini 2.5 Pro, not an analytic consequence of how the formats are defined; it could in principle have gone the other way. The authors' design of both the JSON format and the benchmark that favors it is a potential confound but not circularity, because the comparison is measured rather than stipulated. The paper does not rely on any load-bearing self-citation: the cited prior work concerns other benchmarks and is not used to justify the main result. The programmatic ground-truth semantics (e.g., ordinal edge counting, simultaneous transitions, glitch filtering) are a benchmark-validity and human-validation concern, not a circularity concern, since the labels are generated from the traces rather than from the models or from the claim being tested. The paper's own Limitations section acknowledges limited domain coverage and statistical power, which further supports treating the results as empirical rather than circular. No circular step can be exhibited, so the appropriate score is 0.
Assumptions & free parameters
free parameters (2)
- complexity-grid thresholds =
transition caps 5k/7k/15k/30k/60k; signal bins 0–1k/1–3k/3–5k
- question template set =
8 categories, 24 subcategories, one template instance per subcategory per bin
assumptions (3)
- domain assumption The programmatic ground-truth generator implements the intended semantics of each question template (ordinal edges, simultaneous events, four-state logic, 'ignore glitches'), and internal programmatic verification suffices to validate the labels.
- domain assumption Traces from five open-source RISC-V cores run with fibonacci/CoreMark/bubblesort/Dhrystone firmware are representative of real verification-relevant waveform complexity.
- domain assumption The unstated error-classification scheme (wrong value/signal/order/other) in Figure 7 reliably captures model failure causes.
Cite this review
Pith. "Pith review of WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveforms." pith.science (2026). https://pith.science/paper/FN4JQLOW
@misc{pith2026260720638,
author = {Pith},
title = {Pith review of: WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveforms},
year = {2026},
howpublished = {\url{https://pith.science/paper/FN4JQLOW}},
note = {Machine review of arXiv:2607.20638}
}
read the original abstract
Large Language Models (LLMs) have demonstrated strong capabilities in code generation and reasoning, yet their ability to perform temporal reasoning over digital waveform data remains largely unexplored. Although reasoning over digital waveforms is a critical bottleneck in design verification, existing benchmarks primarily evaluate hardware description language (HDL) code generation and use waveforms only as supplementary context. This paper presents WaveformQA, an open-source question-answering benchmark for evaluating LLM temporal reasoning over digital waveforms. The benchmark comprises 360 questions with programmatically generated ground truths across eight categories of varying difficulty, including questions targeting multi-signal correlation and event ordering. Waveforms are generated from open-source design implementations, ensuring reproducibility and grounding the benchmark in real hardware behavior. Evaluation of frontier LLMs reveals that while models achieve reasonable accuracy on simple queries, performance degrades due to context window limitations and reasoning difficulties on complex temporal and multi-step questions. In addition, we show that an event-time JSON representation of waveforms improves LLM reasoning accuracy versus the standardized value change dump (VCD) format. The open-source framework supports extending to new question categories and importing new waveform sources, enabling researchers to rapidly prototype temporal reasoning experiments.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[2]
AssertLLM: Generating Hardware Verification Assertions from Design Specifications via Multi- LLMs,
Z. Yan, Z. Xie et al., “AssertLLM: Generating Hardware Verification Assertions from Design Specifications via Multi- LLMs,” in Proceedings of the Asia and South Pacific Design Automation Conference (ASPDAC), 2025. [Online]. Available: https://dl.acm.org/doi/10.1145/3658617.3697756
arXiv 2025
-
[1]
ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design,
Z. Yu, C. Zhou, Y. Lin, H. Zhang, H. Ye, J. Cui, Z. Pan, J. Zhao, and Y. Ding, “ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design,”
-
[3]
H. Foster. Part 8: The 2018 Wilson Research Group Functional Verification Study. [Online]. Available: https: //blogs.sw.siemens.com/verificationhorizons/2019/01/29/ part-8-the-2018-wilson-research-group-functional-verification-study/
2018
-
[4]
gtkwave/gtkwave,
R. F. et al., “gtkwave/gtkwave,” https://github.com/gtkwave/gtkwave, Mar 16 2026. [Online]. Available: https://github.com/gtkwave/gtkwave
2026
-
[5]
Y. Bai, G. B. Hamad, C.-T. Ho, S. Suhaib, and H. Ren, “FVDebug: An LLM-Driven Debugging Assistant for Automated Root Cause Analysis of Formal Verification Failures,” 2025. [Online]. Available: https://arxiv.org/abs/2510.15906
arXiv 2025
-
[6]
PicoRV32 – A Size-Optimized RISC-V CPU,
C. Wolf, “PicoRV32 – A Size-Optimized RISC-V CPU,” https: //github.com/YosysHQ/picorv32, 2015–
2015
-
[7]
DarkRISCV – Open-Source RISC-V CPU in Verilog,
darklife, “DarkRISCV – Open-Source RISC-V CPU in Verilog,” https://github.com/darklife/darkriscv, 2018–
2018
-
[8]
Ibex–ASmall32-BitRISC-VCPU Core,
lowRISCandcontributors,“Ibex–ASmall32-BitRISC-VCPU Core,” https://github.com/lowRISC/ibex, 2018–
2018
Show all 32 references
-
[9]
SERV – Award-Winning Bit-Serial RISC-V Core,
O. Kindgren, “SERV – Award-Winning Bit-Serial RISC-V Core,” https://github.com/olofk/serv, 2018–
2018
-
[10]
biRISC-V – 32-Bit Dual-Issue RISC-V CPU,
ultraembedded, “biRISC-V – 32-Bit Dual-Issue RISC-V CPU,” https://github.com/ultraembedded/biriscv, 2019–
2019
-
[11]
Claude 4.5 Sonnet,
Anthropic, “Claude 4.5 Sonnet,” https://www.anthropic.com/ news/claude-sonnet-4-5, September 2025, accessed: Mar. 16, 2026
2025
-
[12]
Claude Sonnet 4.6,
Anthropic, “Claude Sonnet 4.6,” https://www.anthropic.com/ news/claude-sonnet-4-6, 2026, accessed: Mar. 16, 2026
2026
-
[13]
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities,
G. C. et al., “Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities,” 2025. [Online]. Available: https://arxiv.org/abs/2507.06261
2025 arXiv
-
[14]
Qwen3 Technical Report,
A. Y. et al., “Qwen3 Technical Report,” 2025. [Online]. Available: https://arxiv.org/abs/2505.09388
2025 arXiv
-
[15]
A Dataset for Answering Time-Sensitive Questions,
W. Chen, X. Wang, and W. Y. Wang, “A Dataset for Answering Time-Sensitive Questions,” CoRR, vol. abs/2108.06314, 2021. [Online]. Available: https://arxiv.org/abs/2108.06314
2021 arXiv
-
[16]
TempQuestions: A Benchmark for Temporal Question Answering,
Z. Jia, A. Abujabal, R. Saha Roy, J. Strötgen, and G. Weikum, “TempQuestions: A Benchmark for Temporal Question Answering,” in Companion Proceedings of the The Web Conference 2018, ser. WWW ’18. Republic and Canton of Geneva, CHE: International World Wide Web Conferences Steer...
2018
-
[17]
TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models,
Z. Chu, J. Chen, Q. Chen, W. Yu, H. Wang, M. Liu, and B. Qin, “TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models,” in Proc. ACL (Volume 1: Long Papers), 2024, pp. 1204–1228
2024
-
[18]
LTLBench: Towards Benchmarks for Evaluating Temporal Reasoning in Large Language Models,
W. Tang, K. Nuamah, and V. Belle, “LTLBench: Towards Benchmarks for Evaluating Temporal Reasoning in Large Language Models,” arXiv preprint arXiv:2407.05434, 2024. [Online]. Available: https://arxiv.org/abs/2407.05434
2024
-
[19]
TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Tasks,
M. Weng, D. Cao, W. Yang, Y. Sharma, and Y. Liu, “TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Tasks,”
-
[20]
TRAM: Benchmarking Temporal Reasoning for Large Language Models,
Y. Wang and Y. Zhao, “TRAM: Benchmarking Temporal Reasoning for Large Language Models,” 2024. [Online]. Available: https://arxiv.org/abs/2310.00835
2024 arXiv
-
[21]
Available: https://arxiv.org/abs/2602.13272
[Online]. Available: https://arxiv.org/abs/2602.13272
-
[22]
VerilogEval: Evaluating Large Language Models for Verilog Code Generation,
M. Liu, N. Pinckney, B. Khailany, and H. Ren, “VerilogEval: Evaluating Large Language Models for Verilog Code Generation,” 2023. [Online]. Available: https://arxiv.org/abs/2309.07544
2023 arXiv
-
[23]
OpenLLM-RTL: Open Dataset and Benchmark for LLM-Aided Design RTL Generation,
S. Liu, Y. Lu, W. Fang, M. Li, and Z. Xie, “OpenLLM-RTL: Open Dataset and Benchmark for LLM-Aided Design RTL Generation,” in Proc. IEEE/ACM Int. Conf. Comput.-Aided Design (ICCAD), 2025
2025
-
[24]
An Iteratively-refined Dataset for High-Level Synthesis Functional Verification through LLM-Aided Bug Injection,
L. J. Wan, H. Ye, J. Wang, M. Jha, and D. Chen, “An Iteratively-refined Dataset for High-Level Synthesis Functional Verification through LLM-Aided Bug Injection,” in 2024 IEEE LLM Aided Design Workshop (LAD), 2024, pp. 1–6. [Online]. Available: https://hanchenye.com/assets/pdf...
2024
-
[25]
MEIC: Re-thinking RTL Debug Automation using LLMs,
K. Xu, J. Sun, Y. Hu et al., “MEIC: Re-thinking RTL Debug Automation using LLMs,” in Proc. IEEE/ACM Int. Conf. Comput.-Aided Design (ICCAD), 2025
2025
-
[26]
Icarus Verilog — Icarus Verilog documentation,
S. Williams, “Icarus Verilog — Icarus Verilog documentation,”
-
[27]
Enhancing Large Language Models for Hardware Verification: A Novel SystemVerilog Assertion Dataset,
A. Menon, S. S. Miftah, S. Kundu, S. Kundu, A. Srivastava, A. Raha, G. T. Sonnenschein, S. Banerjee, D. Mathaikutty, and K. Basu, “Enhancing Large Language Models for Hardware Verification: A Novel SystemVerilog Assertion Dataset,” 2025. [Online]. Available: https://arxiv.org/...
2025 arXiv
-
[28]
1364-2005, 2005, section 18: Value Change Dump (VCD) files
IEEE Standard for Verilog Hardware Description Language, IEEE Std. 1364-2005, 2005, section 18: Value Change Dump (VCD) files. [Online]. Available: https://standards.ieee.org/ standard/1364-2005.html
2005
-
[29]
LLM Dataset Formats 101: A no‐bs guide for hugging face DEVS,
Tegridy, “LLM Dataset Formats 101: A no‐bs guide for hugging face DEVS,” Feb 2025. [Online]. Available: https://huggingface. co/blog/tegridydev/llm-dataset-formats-101-hugging-face VI. Appendix A. VCD versus JSON Representation Comparison To illustrate the representational tra...
2025
-
[30]
Verilator: Open-source SystemVerilog simulator and lint system,
W. Snyder, “Verilator: Open-source SystemVerilog simulator and lint system,” https://www.veripool.org/verilator/, 2024
2024
-
[2024]
Available: https://steveicarus.github.io/ iverilog/
[Online]. Available: https://steveicarus.github.io/ iverilog/
-
[2026]
Available: https://arxiv.org/abs/2601.21448
[Online]. Available: https://arxiv.org/abs/2601.21448
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.