VeriPilot: An LLM-Powered Verilog Debugging Framework
Pith reviewed 2026-06-26 06:18 UTC · model grok-4.3
The pith
VeriPilot aligns internal variables between buggy Verilog and golden models then traces signals via CDFGs to give LLMs minimal suspicious regions for repair.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
VeriPilot uses golden reference models to align internal variable semantics between a buggy Verilog design and its correct version, then applies static analysis to produce control-data-flow graphs that identify a minimal set of suspicious code regions along with their correct counterparts; these structured insights are supplied to an LLM to guide localization and automated repair.
What carries the argument
LLM-based semantic alignment of internal variables combined with CDFG-derived step-by-step signal tracing that isolates minimal suspicious regions from a golden reference model.
If this is right
- Bug localization accuracy rises because the method compares internal signals rather than only final outputs.
- Repair effectiveness improves on designs with long dependency chains that defeat output-only prompting.
- Large codebases become tractable because the framework reduces the effective context the LLM must reason over.
- The same golden-model-plus-CDFG pattern can be reused across multiple LLM back-ends without retraining.
Where Pith is reading between the lines
- The technique could transfer to other hardware description languages if equivalent golden models and static-analysis front-ends are supplied.
- Combining the CDFG traces with existing formal verification tools might further shrink the suspicious region set.
- Design teams without ready golden models might first generate them via high-level synthesis or simulation to unlock the framework.
Load-bearing premise
Accurate golden reference models exist for the designs and the LLM can reliably match the semantics of internal variables between the buggy code and the reference.
What would settle it
Apply VeriPilot to the same benchmark designs but withhold the golden reference models and measure whether the repair success rate drops back to the 54.3 percent baseline.
Figures
read the original abstract
Verilog debugging remains one of the most time-consuming stages in digital circuit design. Recent advances in Large Language Models (LLMs) have enabled automated debugging; however, most existing approaches rely solely on test outputs and compiler feedback in an end-to-end manner, limiting their effectiveness on complex bugs. A key challenge is that the root cause of an error may be far removed from its observable outputs, making it difficult for LLMs to trace long dependency chains in code. This challenge is further exacerbated in large codebases, where long context lengths hinder efficient reasoning. To address these limitations, we propose VeriPilot, an LLM-powered debugging framework that leverages golden reference models to enable fine-grained bug localization and repair. VeriPilot goes beyond output-level comparison by aligning internal variable semantics between the Verilog design and its corresponding golden model through LLM-based analysis. It then performs step-by-step signal tracing using Control-Data-Flow Graphs (CDFGs) derived from static analysis, identifying a minimal set of suspicious code regions along with their correct counterparts from the golden model. These structured insights are subsequently provided to the LLM to guide reasoning and automated code repair. Experimental results on the Comprehensive Verilog Design Problems (CVDP) benchmark from NVIDIA demonstrate that VeriPilot improves the repair success rate of GPT-4o from 54.3\% to 85.71\%, significantly enhancing both bug localization accuracy and repair effectiveness for complex Verilog designs. The source code and benchmark are publicly available at Github https://github.com/YihanWn/VeriPilot.git.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces VeriPilot, an LLM-powered framework for Verilog debugging that aligns internal variables between a buggy design and a golden reference model via LLM analysis, performs CDFG-guided tracing to localize suspicious regions, and supplies the resulting structured insights to an LLM for automated repair. On the CVDP benchmark it reports raising GPT-4o repair success from 54.3% to 85.71%.
Significance. If the central performance claim is substantiated, the work would demonstrate a practical way to leverage golden models and static-analysis structure to improve LLM reasoning over long dependency chains in hardware designs. The public release of code and benchmark is a clear strength that supports reproducibility and follow-on work.
major comments (2)
- [Abstract and §4] Abstract and §4 (Experimental Results): the headline improvement (54.3% → 85.71%) is produced by a pipeline whose first two stages are LLM-based variable alignment and CDFG-guided tracing that depends on those alignments. No quantitative accuracy, precision, or failure-case analysis is supplied for the alignment step, so it is impossible to determine whether the measured gain can be attributed to the claimed mechanism rather than to noisy or incorrect alignments.
- [§4] §4 (Experimental Results): no ablation is reported that removes the golden-model path or the alignment/tracing components while keeping the final repair LLM fixed. Without such controls the contribution of each stage to the reported success rate remains unquantified.
minor comments (2)
- [Abstract] Abstract: the GitHub URL is given but the manuscript should state the exact commit or release tag used for the reported experiments.
- [§3] §3 (Method): the description of how CDFGs are extracted from Verilog and how variable alignments are injected into the repair prompt could be expanded with a short pseudocode or example to improve reproducibility.
Simulated Author's Rebuttal
We thank the referee for the constructive comments, which highlight opportunities to strengthen the experimental section. We address each major comment below and will incorporate the requested analyses into a revised manuscript.
read point-by-point responses
-
Referee: [Abstract and §4] Abstract and §4 (Experimental Results): the headline improvement (54.3% → 85.71%) is produced by a pipeline whose first two stages are LLM-based variable alignment and CDFG-guided tracing that depends on those alignments. No quantitative accuracy, precision, or failure-case analysis is supplied for the alignment step, so it is impossible to determine whether the measured gain can be attributed to the claimed mechanism rather than to noisy or incorrect alignments.
Authors: We agree that the manuscript lacks a dedicated quantitative evaluation of the LLM-based variable alignment step. In the revision we will add a new subsection reporting alignment accuracy, precision, and recall on a manually annotated subset of CVDP cases, together with a discussion of observed failure modes and their downstream effect on repair success. revision: yes
-
Referee: [§4] §4 (Experimental Results): no ablation is reported that removes the golden-model path or the alignment/tracing components while keeping the final repair LLM fixed. Without such controls the contribution of each stage to the reported success rate remains unquantified.
Authors: We concur that component-wise ablations are necessary to isolate contributions. The revised manuscript will include controlled experiments that disable the golden-model alignment and CDFG tracing stages while retaining the identical repair LLM and test-feedback input, thereby quantifying the incremental benefit of each stage. revision: yes
Circularity Check
No circularity; empirical results on external benchmark
full rationale
The paper proposes an LLM-powered Verilog debugging framework and reports an empirical performance gain (54.3% to 85.71% repair success) on the external CVDP benchmark. No equations, fitted parameters, or derivations are present. The method description relies on golden reference models and LLM variable alignment as design choices, but these do not reduce any claimed result to a self-definition or self-citation chain. The central claim rests on reported experimental outcomes rather than internal re-labeling of inputs. This matches the default expectation of a non-circular empirical systems paper.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Golden reference models exist and can be used for semantic alignment with the design under debug.
Reference graph
Works this paper leans on
-
[1]
Verigen: A large language model for verilog code generation.ACM Transactions on Design Automation of Electronic Systems, 29(3):1–31, 2024
Shailja Thakur, Baleegh Ahmad, Hammond Pearce, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri, and Siddharth Garg. Verigen: A large language model for verilog code generation.ACM Transactions on Design Automation of Electronic Systems, 29(3):1–31, 2024
2024
-
[2]
Software/hardware co-design for llm and its application for design verification
Lily Jiaxin Wan, Yingbing Huang, Yuhong Li, Hanchen Ye, Jinghua Wang, Xiaofan Zhang, and Deming Chen. Software/hardware co-design for llm and its application for design verification. In2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC), pages 435–441. IEEE, 2024
2024
-
[3]
Yang Zhao, Di Huang, Chongxiao Li, Pengwei Jin, Ziyuan Nan, Tianyun Ma, Lei Qi, Yansong Pan, Zhenxing Zhang, Rui Zhang, et al. Codev: Empowering llms for verilog generation through multi-level summariza- tion.arXiv preprint arXiv:2407.10424, 2024
- [4]
-
[5]
Mg-verilog: Multi-grained dataset towards enhanced llm- assisted verilog generation
Yongan Zhang, Zhongzhi Yu, Yonggan Fu, Cheng Wan, and Yingyan Ce- line Lin. Mg-verilog: Multi-grained dataset towards enhanced llm- assisted verilog generation. In2024 IEEE LLM Aided Design Workshop (LAD), pages 1–5. IEEE, 2024
2024
-
[6]
Hivegen–hierarchical llm-based verilog generation for scalable chip design
Jinwei Tang, Jiayin Qin, Kiran Thorat, Chen Zhu-Tian, Yu Cao, Yang Katie Zhao, and Caiwen Ding. Hivegen–hierarchical llm-based verilog generation for scalable chip design. In2025 IEEE International Conference on LLM-Aided Design (ICLAD), pages 30–36. IEEE, 2025
2025
-
[7]
From large to small: Transferring cuda optimization expertise via reasoning graph, 2025
Junfeng Gong, Zhiyi Wei, Junying Chen, Cheng Liu, and Huawei Li. From large to small: Transferring cuda optimization expertise via reasoning graph, 2025
2025
-
[8]
Verilogeval: Evaluating large language models for verilog code genera- tion
Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. Verilogeval: Evaluating large language models for verilog code genera- tion. In2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD), pages 1–8. IEEE, 2023
2023
-
[9]
Large language model for verilog generation with code- structure-guided reinforcement learning
Ning Wang, Bingkun Yao, Jie Zhou, Yuchen Hu, Xi Wang, Zhe Jiang, and Nan Guan. Large language model for verilog generation with code- structure-guided reinforcement learning. In2025 IEEE International Conference on LLM-Aided Design (ICLAD), pages 164–170. IEEE, 2025
2025
-
[10]
Verilogreader: Llm-aided hardware test gener- ation
Ruiyang Ma, Yuxin Yang, Ziqian Liu, Jiaxi Zhang, Min Li, Junhua Huang, and Guojie Luo. Verilogreader: Llm-aided hardware test gener- ation. In2024 IEEE LLM Aided Design Workshop (LAD), pages 1–5. IEEE, 2024
2024
-
[11]
Large language models (llms) for verification, testing, and design
Chandan Kumar Jha, Muhammad Hassan, Khushboo Qayyum, Sallar Ahmadi-Pour, Kangwei Xu, Ruidi Qiu, Jason Blocklove, Luca Collini, Andre Nakkab, Ulf Schlichtmann, et al. Large language models (llms) for verification, testing, and design. In2025 IEEE European Test Symposium (ETS), pages 1–10. IEEE, 2025
2025
-
[12]
Autochip: Automating hdl generation using llm feedback.arXiv preprint arXiv:2311.04887, 2023
Shailja Thakur, Jason Blocklove, Hammond Pearce, Benjamin Tan, Sid- dharth Garg, and Ramesh Karri. Autochip: Automating hdl generation using llm feedback.arXiv preprint arXiv:2311.04887, 2023
-
[13]
Rtlfixer: Automatically fixing rtl syntax errors with large language model
YunDa Tsai, Mingjie Liu, and Haoxing Ren. Rtlfixer: Automatically fixing rtl syntax errors with large language model. InProceedings of the 61st ACM/IEEE Design Automation Conference, pages 1–6, 2024
2024
-
[14]
Hanxian Huang, Zhenghan Lin, Zixuan Wang, Xin Chen, Ke Ding, and Jishen Zhao. Towards llm-powered verilog rtl assistant: Self-verification and self-correction.arXiv preprint arXiv:2406.00115, 2024
-
[15]
Meic: Re-thinking rtl debug automation using llms
Ke Xu, Jialin Sun, Yuchen Hu, Xinwei Fang, Weiwei Shan, Xi Wang, and Zhe Jiang. Meic: Re-thinking rtl debug automation using llms. InProceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design, pages 1–9, 2024
2024
-
[16]
Veridebug: A unified llm for verilog debugging via con- trastive embedding and guided correction
Ning Wang, Bingkun Yao, Jie Zhou, Yuchen Hu, Xi Wang, Zhe Jiang, and Nan Guan. Veridebug: A unified llm for verilog debugging via con- trastive embedding and guided correction. In2025 IEEE International Conference on LLM-Aided Design (ICLAD), pages 61–67. IEEE, 2025
2025
-
[17]
Location is key: Leveraging llm for functional bug local- ization in verilog design
Bingkun Yao, Ning Wang, Jie Zhou, Xi Wang, Hong Gao, Zhe Jiang, and Nan Guan. Location is key: Leveraging llm for functional bug local- ization in verilog design. In2025 62nd ACM/IEEE Design Automation Conference (DAC), pages 1–7. IEEE, 2025
2025
-
[18]
Springer, 2000
Janick Bergeron.Writing testbenches: functional verification of HDL models. Springer, 2000
2000
-
[19]
Automated program repair.Communications of the ACM, 62(12):56–65, 2019
Claire Le Goues, Michael Pradel, and Abhik Roychoudhury. Automated program repair.Communications of the ACM, 62(12):56–65, 2019
2019
-
[20]
A survey of learning-based automated program repair
Quanjun Zhang, Chunrong Fang, Yuxiang Ma, Weisong Sun, and Zhenyu Chen. A survey of learning-based automated program repair. ACM Transactions on Software Engineering and Methodology, 33(2):1– 69, 2023. 13
2023
-
[21]
A critical review on the evaluation of automated program repair systems.Journal of Systems and Software, 171:110817, 2021
Kui Liu, Li Li, Anil Koyuncu, Dongsun Kim, Zhe Liu, Jacques Klein, and Tegawend ´e F Bissyand ´e. A critical review on the evaluation of automated program repair systems.Journal of Systems and Software, 171:110817, 2021
2021
-
[22]
Understanding bug fix patterns in verilog
Sangeetha Sudakrishnan, Janaki Madhavan, E James Whitehead Jr, and Jose Renau. Understanding bug fix patterns in verilog. InProceedings of the 2008 international working conference on Mining software repositories, pages 39–42, 2008
2008
-
[23]
Cirfix: Automatically repairing defects in hardware design code
Hammad Ahmad, Yu Huang, and Westley Weimer. Cirfix: Automatically repairing defects in hardware design code. InProceedings of the 27th ACM International Conference on Architectural Support for Program- ming Languages and Operating Systems, pages 990–1003, 2022
2022
-
[24]
Deheng Yang, Jiayu He, Xiaoguang Mao, Tun Li, Yan Lei, Xin Yi, and Jiang Wu. Strider: Signal value transition-guided defect repair for hdl programming assignments.IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 43(5):1594–1607, 2023
2023
-
[25]
Rtl-repair: Fast symbolic repair of hardware design code
Kevin Laeufer, Brandon Fajardo, Abhik Ahuja, Vighnesh Iyer, Borivoje Nikoli´c, and Koushik Sen. Rtl-repair: Fast symbolic repair of hardware design code. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pages 867–881, 2024
2024
-
[26]
Verifix: Verified repair of programming assign- ments.ACM Transactions on Software Engineering and Methodology (TOSEM), 31(4):1–31, 2022
Umair Z Ahmed, Zhiyu Fan, Jooyong Yi, Omar I Al-Bataineh, and Abhik Roychoudhury. Verifix: Verified repair of programming assign- ments.ACM Transactions on Software Engineering and Methodology (TOSEM), 31(4):1–31, 2022
2022
-
[27]
Llm-aided efficient hardware design automation
Kangwei Xu, Ruidi Qiu, Zhuorui Zhao, Grace Li Zhang, Ulf Schlicht- mann, and Bing Li. Llm-aided efficient hardware design automation. arXiv preprint arXiv:2410.18582, 2024
-
[28]
Llm-assisted bug identification and correction for verilog hdl.ACM Transactions on Design Automation of Electronic Systems, 2025
Khushboo Qayyum, Chandan Kumar Jha, Sallar Ahmadi-Pour, Muham- mad Hassan, and Rolf Drechsler. Llm-assisted bug identification and correction for verilog hdl.ACM Transactions on Design Automation of Electronic Systems, 2025
2025
-
[29]
Evaluating llm-based communicative agents for verilog design
Parker Link and Benjamin Tan. Evaluating llm-based communicative agents for verilog design. In2025 26th International Symposium on Quality Electronic Design (ISQED), pages 1–8. IEEE, 2025
2025
-
[30]
Zhigang Fang, Renzhi Chen, Zhijie Yang, Yang Guo, Huadong Dai, and Lei Wang. Lintllm: An open-source verilog linting framework based on large language models.arXiv preprint arXiv:2502.10815, 2025
-
[31]
SysVCoder: An LLM-Driven Framework for Systematic Generation of System-Level Design
Jian Zuo, Junzhe Liu, Xianyong Wang, Yicheng Liu, Navya Goli, Tong Xu, Hao Zhang, Umamaheswara Rao Tida, Zhenge Jia, and Mengying Zhao. Complexvcoder: An llm-driven framework for systematic gener- ation of complex verilog code.arXiv preprint arXiv:2504.20653, 2025
work page internal anchor Pith review arXiv 2025
-
[32]
A generalize hardware debugging approach for large language models semi-synthetic, datasets.IEEE Transactions on Circuits and Systems I: Regular Papers, 72(2):623–636, 2024
Weimin Fu, Shijie Li, Yifang Zhao, Kaichen Yang, Xuan Zhang, Yier Jin, and Xiaolong Guo. A generalize hardware debugging approach for large language models semi-synthetic, datasets.IEEE Transactions on Circuits and Systems I: Regular Papers, 72(2):623–636, 2024
2024
-
[33]
Verimind: Agentic llm for automated verilog generation with a novel evaluation metric
Bardia Nadimi, Ghali Omar Boutaib, and Hao Zheng. Verimind: Agentic llm for automated verilog generation with a novel evaluation metric. arXiv preprint arXiv:2503.16514, 2025
-
[34]
Zhendong Mi, Renming Zheng, Haowen Zhong, Yue Sun, Seth Kneeland, Sayan Moitra, Ken Kutzer, and Zhaozhuo Xu Shaoyi Huang. Coopetitivev: Leveraging llm-powered coopetitive multi- agent prompting for high-quality verilog generation.arXiv preprint arXiv:2412.11014, 2024
-
[35]
Mohammad Shahidzadeh, Behnam Ghavami, Steve Wilton, and Lesley Shannon. Automatic high-quality verilog assertion generation through subtask-focused fine-tuned llms and iterative prompting.arXiv preprint arXiv:2411.15442, 2024
-
[36]
Hdldebugger: Streamlining hdl debugging with large language models.ACM Transactions on Design Automation of Electronic Systems, 30(6):1–26, 2025
Xufeng Yao, Haoyang Li, Tsz Ho Chan, Wenyi Xiao, Mingxuan Yuan, Yu Huang, Lei Chen, and Bei Yu. Hdldebugger: Streamlining hdl debugging with large language models.ACM Transactions on Design Automation of Electronic Systems, 30(6):1–26, 2025
2025
-
[37]
Charles Hong, Brendan Roberts, Huijae An, Alex Um, Advay Ratan, and Yakun Sophia Shao. hdl2v: A code translation dataset for enhanced llm verilog generation.arXiv preprint arXiv:2506.04544, 2025
-
[38]
On hardware security bug code fixes by prompting large language models.IEEE Transactions on Information Forensics and Security, 19:4043–4057, 2024
Baleegh Ahmad, Shailja Thakur, Benjamin Tan, Ramesh Karri, and Hammond Pearce. On hardware security bug code fixes by prompting large language models.IEEE Transactions on Information Forensics and Security, 19:4043–4057, 2024
2024
-
[39]
Effective simulation and debugging for a high-level hardware language using software compilers
Cl ´ement Pit-Claudel, Thomas Bourgeat, Stella Lau, Arvind, and Adam Chlipala. Effective simulation and debugging for a high-level hardware language using software compilers. InProceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pages 789–803, 2021
2021
-
[40]
Zhuorui Zhao, Bing Li, Grace Li Zhang, and Ulf Schlichtmann. Vfo- cus: Better verilog generation from large language model via focused reasoning.arXiv preprint arXiv:2511.02285, 2025
-
[41]
Mage: A multi-agent engine for automated rtl code generation
Yujie Zhao, Hejia Zhang, Hanxian Huang, Zhongming Yu, and Jishen Zhao. Mage: A multi-agent engine for automated rtl code generation. In2025 62nd ACM/IEEE Design Automation Conference (DAC), pages 1–7. IEEE, 2025
2025
-
[42]
Verilogcoder: Autonomous verilog coding agents with graph-based planning and abstract syntax tree (ast)-based waveform tracing tool
Chia-Tung Ho, Haoxing Ren, and Brucek Khailany. Verilogcoder: Autonomous verilog coding agents with graph-based planning and abstract syntax tree (ast)-based waveform tracing tool. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 300– 307, 2025
2025
-
[43]
Buggen: A self-correcting multi-agent llm pipeline for realistic rtl bug synthesis
Surya Jasper, Minh Luu, Evan Pan, Aakash Tyagi, Michael Quinn, Jiang Hu, and David Houngninou. Buggen: A self-correcting multi-agent llm pipeline for realistic rtl bug synthesis. In2025 ACM/IEEE 7th Symposium on Machine Learning for CAD (MLCAD), pages 1–9. IEEE, 2025
2025
-
[44]
Pyverilog: A python-based hardware design processing toolkit for verilog hdl
Shinya Takamaeda-Yamazaki. Pyverilog: A python-based hardware design processing toolkit for verilog hdl. InInternational Symposium on Applied Reconfigurable Computing, pages 451–460. Springer, 2015
2015
-
[45]
pyslang: Systemverilog frontend
Mike Popoloski. pyslang: Systemverilog frontend. https://github.com/ MikePopoloski/slang, 2022
2022
-
[46]
cocotb: Coroutine-based cosimulation library for writing vhdl and verilog testbenches in python
cocotb contributors. cocotb: Coroutine-based cosimulation library for writing vhdl and verilog testbenches in python. https://github.com/ cocotb/cocotb, 2024
2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.