Pith. sign in

REVIEW 3 major objections 4 minor 23 references

VCDiag: Classifying Erroneous Waveforms for Failure Triage Acceleration

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Machine learning on chip-simulation waveforms can rank the failure-causing module among the top three 94% of the time.

desk verdict Solid engineering paper with a real pipeline and honest reporting, but the headline accuracy numbers all rest on synthetic bugs from the authors' own generator, so treat them as internal validation, not field performance. read the letter →

arxiv 2506.03590 v5 pith:Q5ODPNBT submitted 2025-06-04 cs.LG

Minh Luu , Surya Jasper , Khoi Le , Evan Pan , Michael Quinn , Aakash Tyagi , Jiang Hu This is my paper
classification cs.LG
keywords functionalverificationfailuretriageVCDwaveformmachinelearningRTLdebuggingstatisticalcompressionsyntheticbuginjectionLightGBM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VCDiag is a machine-learning framework that turns failing simulation waveforms into a short ranked list of suspect RTL modules, so verification engineers can start debugging at the likely fault location instead of reading traces by hand. It works by extracting only signals tied to target modules, compressing each waveform with statistical summaries, and training a tree-based classifier on waveforms labeled by synthetic injected bugs. On the largest benchmark, the FabScalar processor, the best model ranks the true failure-causing module in its top three predictions 94% of the time, while shrinking raw waveform data by 123x. The paper argues that this makes learned failure triage practical for large designs and easy to adapt to any Verilog/SystemVerilog testbench.

What carries the argument

The central object is the Value Change Dump (VCD) file, the IEEE-standard waveform format that records every signal's value changes during simulation. VCDiag's load-bearing mechanism is a four-stage reduction pipeline: an AST-based crawler (using Verible) builds a module-to-signal lookup table to prune irrelevant signals; a fixed 2000-tick window around the failure is normalized by trimming and zero-padding; sktime's SummaryTransformer compresses each signal into one row of statistical features (mean, standard deviation, quantiles); and optional XGBoost feature ranking keeps the top 50–70% of signals, capped at 5000. The statistical compression is what makes the classification tractable, turning gigabytes of sparse temporal traces into a compact tabular dataset that classical supervised models can learn from quickly.

What would settle it

Run the trained classifier on a set of real, known-cause RTL failures, for example bugs fixed in an open-source processor's repository, and compare top-3 accuracy against the 94% reported on injected FabScalar bugs; a large drop on real failures would falsify the claim that these waveform signatures generalize beyond synthetic mutations.

Watch

Extended reading notes

Core claim

In the paper's own terms, the central discovery is that statistically compressed VCD waveforms retain enough temporal and structural information to classify a failing simulation to its root-cause module: on the 12-module FabScalar processor, LightGBM achieves 77% top-1 and 94% top-3 accuracy, and across the OpenTitan IP blocks the top-3 accuracy reaches 96–99%. The framework formulates the task as multivariate time-series classification, with a waveform $X \in \mathbb{R}^{T \times d}$ mapped by $f: \mathbb{R}^{T \times d} \to \{1,\dots,M\}$ to one of $M$ modules. Rather than feeding raw traces, VCDiag prunes signals via an AST-derived lookup table, truncates to a 2000-tick failure window, and reduces each signal to statistical features (mean, standard deviation, quantiles), which the paper shows is sufficient for accurate module identification. The paper further claims that tree-based gradient boosting models outperform distance- and bagging-based baselines on this representation, and that the compression lowers FabScalar's dataset from 308 GB to 2.5 GB without hurting accuracy.

Load-bearing premise

The accuracy numbers assume that the synthetically injected, single-module bugs produce waveform signatures representative of real chip bugs, an assumption the paper does not test on any real failing testcases.

Editorial extensions

If this is right

  • On FabScalar, ranking the true module in the top three 94% of the time means an engineer can ignore most of the design and focus debugging on three modules.
  • Because the input is only VCD data, the same pipeline transfers across Verilog/SystemVerilog designs and simulators without changing the ML stack.
  • The 123x compression and 36-hour parallel generation of 1,600 bug scenarios make a large, ML-ready failure dataset feasible on a single 32-core machine.
  • Accuracy stays comparable when the trace window is cut from 2000 to 200 ticks, so shorter simulations may be enough for training data.
  • Adding a new design requires only one automation script and a config file; adapting to additional OpenTitan IPs took hours once the first was integrated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If real RTL bugs produce waveform signatures similar to the injected mutations, VCDiag could be applied directly to industrial regression failures, but the paper gives no evidence for that resemblance, so this remains a conjecture.
  • The comparable accuracy at 200 ticks suggests the classifier largely exploits timeouts and early-failure signatures on FabScalar; bugs without such timing traces might need different features.
  • The single-module labeling scheme limits the method to one bug per failing test, so practical multi-bug triage would require the iterative 'onion-peeling' strategy the paper only sketches as future work.
  • Because signal names must stay consistent and parameter renames invalidate training data, the approach may need retraining whenever a design evolves across regression versions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes VCDiag, a framework that classifies failing VCD simulation waveforms to predict which RTL module is responsible for a failure, with the goal of accelerating failure triage in design verification. The pipeline consists of AST-based signal selection, parallel simulation and extraction, statistical compression of waveform data into tabular features, optional XGBoost signal ranking, and classical ML classifiers (KNN, Random Forest, XGBoost, LightGBM). The framework is evaluated on OpenTitan AES and other OpenTitan IPs as well as the FabScalar processor, using failures generated by a separate LLM-based synthetic bug injection tool, BugGen. On the largest benchmark, FabScalar, LightGBM achieves 77% top-1 and 94% top-3 accuracy (Table IV), and the reported data compression reduces raw waveform data from 308 GB to 2.5 GB, a 123x reduction. The authors also report end-to-end parallelization speedups and an exploratory result on accuracy versus simulation window length.

Significance. If the reported results transfer to real design failures, VCDiag would be a practical and useful contribution: it targets RTL simulation, uses the standard VCD format, avoids proprietary log parsing, and provides an open-source artifact with explicitly separated train/test splits and reproducible processing scripts. The reported compression ratio and parallelization are genuinely useful engineering results, and the use of multiple open RTL designs (OpenTitan, FabScalar) is a strength relative to much prior work that evaluates on proprietary or very small designs. The main significance is conditional on whether the synthetic-bug benchmark adequately represents real RTL failure signatures; the paper is transparent about this limitation but does not close the gap.

major comments (3)
  1. [Section IV-B and Section VI-B] The central accuracy claims in Table IV are measured entirely on failures generated by the authors' LLM-based synthetic bug injector. The paper does not provide any experiment on real failing regression tests, historical bug fixes, or independently authored bug reports, so the 77% top-1 and 94% top-3 numbers are demonstrated only for the distribution of BugGen mutations. Section IV-C itself notes that the generator requires plain-English descriptions and auto-reverts ineffective bugs, which may bias the injected set toward simple, localized mutations; Section VI-A additionally acknowledges that FabScalar stimulus diversity is limited by fixed program binaries. This is not an internal inconsistency, because the held-out test scenarios are legitimately separated from training, but it is a load-bearing external-validity gap for the paper's stated goal of accelerating real failure triage. Please add at least one evaluation on real failing waveforms or human-written bug fixes, or explicitly and pervasively scope all conclusions to synthetic-bug triage.
  2. [Section IV-B and Section VI-B] The abstract and Section VI-B claim that the signal-selection and statistical-compression pipeline preserves features essential for classification, but no ablation is reported that compares classification accuracy with and without these stages. The reader can see the final compressed feature set in Table V, but there is no experiment using, for example, all signals versus selected signals, or raw/rough CSV features versus statistically compressed features, on the same train/test split. Without such an ablation, the claim that a 123x reduction is achieved 'while preserving features essential for classification' is unsupported. I request an ablation study that quantifies the accuracy impact of each pipeline stage; if the accuracy is already high without compression, the compression claim should be reframed as a storage/efficiency result rather than a classification-preservation result.
  3. [Section II and Section VI-A] The comparison with BugMD in Section II and the statement in Section VI-A that 'our approach exceeds BugMD's reported results on FabScalar' is not a same-benchmark comparison. BugMD is evaluated on post-silicon CPU bugs with gate-level injection, a different abstraction level, a different bug population, and different metric definitions. The present paper uses RTL-level bug injection, a different feature representation, and a different simulator setup. As reported, the accuracy gap between VCDiag and BugMD may be explained by any of these differences rather than by the proposed method. Please either adapt BugMD to the same RTL-level synthetic-bug benchmark or clearly characterize the comparison as cross-study and not a direct superiority claim.
minor comments (4)
  1. [Table IV] All accuracy numbers are point estimates with no confidence intervals, standard deviations, or repeated-seed results; given the variability visible in the FabScalar accuracy versus simulation-window experiment, bootstrap or repeat-run intervals would make the headline numbers more reliable.
  2. [Table II] In the AES rows, two modules are both labeled 'FSM' with different training/test counts; the later text distinguishes CipherControlFSM and AESCounterFSM, so the table should use these distinct names to avoid ambiguity.
  3. [Section IV-B] Step 3 of the compression pipeline says statistical compression 'shrinks the data size but increases the features'; this trade-off is clear, but the interaction between the optional XGBoost signal count limit (5000 signals) and the number of statistical features per signal (n) is not defined, so the final feature dimensionality is not reproducible from the text alone.
  4. [Artifact Appendix] The artifact appendix says the tool is built entirely on open-source libraries, but the main experiments use Synopsys VCS and Cadence Xcelium (Table II); please state whether the artifact can reproduce results with an open-source simulator such as Icarus Verilog or Verilator, or list the commercial dependencies explicitly in the reproducibility statement.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: VCDiag's accuracy is an honest held-out evaluation on its own synthetic-bug benchmark; the only related concern is a companion-paper self-citation and an external-validity gap, not circularity.

full rationale

VCDiag is an empirical ML pipeline rather than an analytic derivation. The predictive chain is: inject a mutation into a chosen RTL module (Section IV-C), simulate to first failure, compress the VCD into statistical features (Section IV-B), train a classifier on training scenarios, and report accuracy on test scenarios generated independently. The paper states: 'Each module includes 80–120 training and 20 testing bug scenarios, injected and extracted independently for strict separation of training and testing data.' The test labels are not used in training, feature selection is described as 'based on relevance to training bug scenarios,' and the headline 77% top-1 / 94% top-3 numbers in Table IV are test-set measurements. Nothing in the paper equates a fitted parameter with a prediction, imports a uniqueness or ansatz result from the authors' prior work, or renames a known result as a new derivation. The only author-overlap citation is [18] (BugGen), used as the LLM bug-injection tool, and the artifact appendix explicitly bundles BugGen with VCDiag. That self-citation is not load-bearing as evidence for the accuracy claim because the generator's behavior and limitations are described in the paper itself, and the classification evaluation is self-contained. The real limitation—that all bugs are synthetic and no experiment connects LLM-injected mutations to real RTL failure signatures—is an external-validity gap, explicitly acknowledged by the synthetic-only evaluation. It is not a circular step under the definitions used here.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The framework adds no theoretical entities. The central claim depends on four domain assumptions about VCD signal coverage, statistical compression preserving discriminative information, the first-error snapshot being a sufficient signature, and synthetic LLM bugs representing real bugs. The hand-set processing thresholds are listed as free parameters.

free parameters (4)
  • simulation window length = 2000 ticks before failure, with 200 ticks also evaluated
    Fixed window chosen to cap VCD file size; Figure 8 shows comparable accuracy at 200 ticks, so the exact value is not critical, but it is a hand-set hyperparameter.
  • XGBoost signal retention threshold = 50%-70% of ranked signals
    Optional feature selection keeps only the top 50-70% of signals until all modules are included or the 5000-signal limit is reached; the threshold is chosen by hand.
  • signal processing limit = 5000 signals
    Cap on the number of signals kept after XGBoost selection, chosen for processing feasibility.
  • statistical feature set = mean, standard deviation, quantiles (exact list not fully specified)
    SummaryTransformer compresses each signal into a row using statistical metrics; the choice and count of statistics affects feature dimensionality and is not determined by theory.
assumptions (4)
  • domain assumption VCD signals attached to the target modules, as identified by AST lookup, contain the information needed to localize injected bugs.
    The pipeline prunes the signal list to module-local signals before feature extraction (Section IV-A); if failure signatures live in signals outside these modules, the classifier cannot see them.
  • domain assumption Statistical summaries of each signal over the simulation window preserve the class-discriminative temporal structure.
    Statistical compression (Section IV-B) collapses the time dimension; temporal ordering and event timing are captured only through aggregate statistics.
  • domain assumption Simulation halts at the first error, so the VCD ending at failure contains a representative behavioral signature of the buggy module.
    Per Section III, simulation halts upon detecting the first error, and both the VCD content and the module label assume this single snapshot is sufficient.
  • ad hoc to paper LLM-injected syntactic mutations are representative of real RTL bugs.
    All training and test labels come from the authors' BugGen pipeline (Section IV-C); the paper provides no evidence that this synthetic distribution matches real-world bug distributions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VCDiag: Classifying Erroneous Waveforms for Failure Triage Acceleration." pith.science (2026). https://pith.science/paper/Q5ODPNBT

@misc{pith2026250603590,
  author       = {Pith},
  title        = {Pith review of: VCDiag: Classifying Erroneous Waveforms for Failure Triage Acceleration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q5ODPNBT}},
  note         = {Machine review of arXiv:2506.03590}
}
read the original abstract

Failure triage in design functional verification is critical but time-intensive, relying on manual specification reviews, log inspections, and waveform analyses. While machine learning (ML) has improved areas like stimulus generation and coverage closure, its application to RTL-level simulation failure triage, particularly for large designs, remains limited. VCDiag offers an efficient, adaptable approach using VCD data to classify failing waveforms and pinpoint likely failure locations. In the largest experiment, VCDiag achieves over 94% accuracy in identifying the top three most likely modules. The framework introduces a novel signal selection and statistical compression approach, achieving over 120x reduction in raw data size while preserving features essential for classification. It can also be integrated into diverse Verilog/SystemVerilog designs and testbenches.

Figures

Figures reproduced from arXiv: 2506.03590 by the authors.

Figure 1
Figure 1. VCDiag framework a broader set of modern, open-source designs, demonstrating its scalability and applicability. III. PROBLEM FORMULATION A Verilog/SystemVerilog module defines the behavior of a hard￾ware block through internal logic and signal interactions. Modules communicate via well-defined interfaces and can be composed hier￾archically to build complex digital systems. During simulation, the complete design, ref… view at source ↗
Figure 2
Figure 2. Failure signature classification. By using VCDs as the sole input, VCDiag remains independent of the DUT and testbench, ensuring compatibility with any Verilog/Sys￾temVerilog design or simulator without modification. To maintain this generality, future integration of logs or design specs will require standardized outputs or novel feature extraction methods for ML training. IV. PROPOSED FRAMEWORK This section describ… view at source ↗
Figure 3
Figure 3. Pruning signals in hierarchical form. B. Parallel simulation, extraction, and processing Signal selection by itself is insufficient for designs with numerous modules. Even after pruning, VCD datasets can reach hundreds of gigabytes, making them difficult to process efficiently. To address this challenge, we developed a multi-stage data compression process: 1) VCDs to CSVs (raw): A fixed simulation window of 2000 tic… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: LLM attempts to get an accepted bug scenario. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: LightGBM Confusion Matrix for AES. outcomes and highlight opportunities to refine our training strategies to reduce dependency on handcrafted scenarios [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: LightGBM Confusion Matrix for FabScalar. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Multi-core completion time and speedup for 40 AES’ bug [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: LightGBM accuracy relative to simulation ticks before failure. [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 19 canonical work pages

  1. [1]

    The 2024 wilson research group functional verification study,

    H. D. Foster, “The 2024 wilson research group functional verification study,” 2024

    work page 2024

  2. [2]

    Fault diagnosis and logic debugging using boolean satisfiability,

    A. Smith, A. Veneris, M. Ali, and A. Viglas, “Fault diagnosis and logic debugging using boolean satisfiability,”IEEE Transactions on Computer- Aided Design of Integrated Circuits and Systems , vol. 24, no. 10, pp. 1606–1621, 2005

    work page 2005

  3. [3]

    A failure triage engine based on error trace signature extraction,

    Z. Poulos, Y .-S. Yang, and A. Veneris, “A failure triage engine based on error trace signature extraction,” in 2013 IEEE 19th International On-Line Testing Symposium (IOLTS) , 2013, pp. 73–78

    work page 2013

  4. [4]

    Fae: Autoencoder-based failure binning of rtl designs for verification and debugging,

    C.-H. Shen, A. C.-W. Liang, C. C.-H. Hsu, and C. H.-P. Wen, “Fae: Autoencoder-based failure binning of rtl designs for verification and debugging,” in 2019 IEEE International Test Conference (ITC) , 2019, pp. 1–10

    work page 2019

  5. [5]

    Clustering and classification of uvm test failures using machine learning techniques,

    A. Truong, D. Hellstr ¨om, H. Duque, and L. Viklund, “Clustering and classification of uvm test failures using machine learning techniques,” in Proceedings of the Design and Verification Conference (DVCON), San Jose, CA, USA , vol. 26, 2018

    work page 2018

  6. [6]

    Failure triage: The neglected debugging problem,

    S. Safarpour, B. Keng, Y .-S. Yang, and E. Qin, “Failure triage: The neglected debugging problem,” in Design and Verification Conference , 2012

    work page 2012

  7. [7]

    Bugmd: Automatic mismatch diagnosis for bug triaging,

    B. Mammo, M. Furia, V . Bertacco, S. Mahlke, and D. S. Khudia, “Bugmd: Automatic mismatch diagnosis for bug triaging,” in 2016 IEEE/ACM International Conference on Computer-Aided Design (IC- CAD). IEEE, 2016, pp. 1–7

    work page 2016

  8. [8]

    Ieee standard hardware description language based on the verilog(r) hardware description language,

    “Ieee standard hardware description language based on the verilog(r) hardware description language,” IEEE Std 1364-1995 , pp. 1–688, 1996

    work page 1995

Show all 23 references
  1. [9]

    Verible,

    “Verible,” https://github.com/chipsalliance/verible, 2024

    2024

  2. [10]

    cirosantilli, “Vcdvcd,” https://github.com/cirosantilli/vcdvcd, 2024

    2024

  3. [11]

    Data structures for statistical computing in python,

    W. McKinney, “Data structures for statistical computing in python,” in Proceedings of the 9th Python in Science Conference , S. van der Walt and J. Millman, Eds., 2010, pp. 51 – 56

    2010

  4. [12]

    Opentitan: Open source silicon root of trust (rot),

    “Opentitan: Open source silicon root of trust (rot),” https://github.com/ lowrisc/opentitan, 2024

    2024

  5. [13]

    Fab- scalar: Composing synthesizable rtl designs of arbitrary cores within a canonical superscalar template,

    N. K. Choudhary, S. V . Wadhavkar, T. A. Shah, H. Mayukh, J. Gandhi, B. H. Dwiel, S. Navada, H. H. Najaf-abadi, and E. Rotenberg, “Fab- scalar: Composing synthesizable rtl designs of arbitrary cores within a canonical superscalar template,” in 2011 38th Annual International Sy...

    2011

  6. [14]

    Xgboost: A scalable tree boosting system,

    T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , ser. KDD ’16. New York, NY , USA: Association for Computing Machinery, 2016, p. 785–794. [Online]. Avail...

    2016

  7. [15]

    sktime documentation,

    sktime developers, “sktime documentation,” https://www.sktime.net/en/ stable/, 2025, accessed: 2025-07-10

    2025

  8. [16]

    Large language models are few-shot testers: Exploring llm-based general bug reproduction,

    S. Kang, J. Yoon, and S. Yoo, “Large language models are few-shot testers: Exploring llm-based general bug reproduction,” in Proceedings of the 45th International Conference on Software Engineering , ser. ICSE ’23. IEEE Press, 2023, p. 2312–2323. [Online]. Available: https://d...

    2023

  9. [17]

    Automated bug generation in the era of large language models,

    A. R. Ibrahimzada, Y . Chen, R. Rong, and R. Jabbarvand, “Automated bug generation in the era of large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2310.02407

    2024 arXiv

  10. [18]

    Buggen: A self-correcting multi-agent llm pipeline for realistic rtl bug synthesis,

    S. Jasper, M. Luu, E. Pan, A. Tyagi, M. Quinn, J. Hu, and D. K. Houngninou, “Buggen: A self-correcting multi-agent llm pipeline for realistic rtl bug synthesis,” 2025. [Online]. Available: https://arxiv.org/abs/2506.10501

    2025 arXiv

  11. [19]

    API design for machine learning software: experiences from the scikit-learn project,

    L. Buitinck, G. Louppe, M. Blondel, F. Pedregosa, A. Mueller, O. Grisel, V . Niculae, P. Prettenhofer, A. Gramfort, J. Grobler, R. Layton, J. Van- derPlas, A. Joly, B. Holt, and G. Varoquaux, “API design for machine learning software: experiences from the scikit-learn project,...

    2013

  12. [20]

    Lightgbm: A highly efficient gradient boosting decision tree,

    G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.- Y . Liu, “Lightgbm: A highly efficient gradient boosting decision tree,” Advances in neural information processing systems , vol. 30, 2017

    2017

  13. [21]

    VCS: Functional Verification Solution,

    Synopsys, Inc., “VCS: Functional Verification Solution,” 2025

    2025

  14. [22]

    Xcelium Logic Simulation,

    Cadence Design Systems, Inc., “Xcelium Logic Simulation,” 2025

    2025

  15. [23]

    Multiclass receiver operating characteristic (roc),

    “Multiclass receiver operating characteristic (roc),” 2025, accessed: 2025-05-19. [Online]. Available: https://scikit-learn.org ARTIFACT APPENDIX A. Abstract This artifact supports two papers: (1) VCDiag: Classifying Er- roneous Waveforms for Failure Triage Acceleration , and ...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.