Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

ScaleRTL: Scaling LLMs with Reasoning Data and Test-Time Compute for Accurate RTL Code Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ScaleRTL, a reasoning LLM trained on 3.5 billion tokens of RTL chain-of-thought data and extended with test-time self-correction, sets state-of-the-art results on VerilogEval and RTLLM.

desk verdict Base ScaleRTL is a solid data-scaling contribution; the test-time scaling headline is compromised by using the benchmark testbench as an oracle in the repair loop. read the letter →

arxiv 2506.05566 v2 pith:PCU7WAFV submitted 2025-06-05 cs.AR cs.AI

Chenhui Deng , Yun-Da Tsai , Guan-Ting Liu , Zhongzhi Yu , Haoxing Ren This is my paper
classification cs.ARcs.AI
keywords RTLcodegenerationVerilogchain-of-thoughtreasoningtest-timescalingLLMfine-tuninghardwaredesignself-correctionfunctionalcorrectness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that the main barrier to accurate RTL code generation is not model size but the scarcity of high-quality reasoning data, and that scaling reasoning tokens during both training and inference can break that barrier. It introduces ScaleRTL, a reasoning LLM fine-tuned on 62,000 long chain-of-thought traces totalling about 3.5 billion tokens, which is orders of magnitude more training data than prior RTL models used. A test-time scaling variant, ScaleRTL†, iteratively re-prompts the model to recheck and self-correct its reasoning whenever a generated solution fails the testbench. If the paper is right, RTL code generation follows the same test-time scaling behavior seen in math and general code, and hardware-design LLMs should be built around domain reasoning data rather than plain instruction-code pairs.

What carries the argument

The load-bearing mechanism is a two-stage scaling recipe. In the training stage, 62,000 curated RTL code scripts are converted into long chain-of-thought traces by prompting a general-purpose reasoning LLM to write a specification from each script and then solve it with step-by-step reasoning, producing a corpus of roughly 3.5 billion tokens of specification, CoT trace, and code triples on which a 32B reasoning model is fine-tuned. In the inference stage, called ScaleRTL†, whenever generated code fails the testbench the model's end-of-reasoning delimiter is replaced with a corrective prompt containing general RTL coding rules, and the extended trace is fed back so the model rethinks and self-corrects. This loop converts additional test-time compute into higher functional correctness and is the mechanism the paper credits for the remaining gains over ScaleRTL.

What would settle it

Search the released training corpus for near-duplicates of every VerilogEval and RTLLM problem using the same Jaccard and embedding filters used in the paper; if any benchmark problem appears at similarity above 0.8, the accuracy claims would be inflated. A cleaner test is to run ScaleRTL and ScaleRTL† on a newly written set of RTL design problems not present on the web before the model's training cutoff and compare the margin over baselines.

Watch

Extended reading notes

Core claim

The paper's central claim is that a reasoning-oriented LLM, fine-tuned on a billion-token corpus of RTL chain-of-thought traces and then given extra compute at inference to rethink and self-correct, achieves state-of-the-art functional correctness on the VerilogEval and RTLLM benchmarks. On the strongest prior RTL-specific baseline it reports pass@1 gains of up to 10.7 percentage points on VerilogEval and 11.7 percentage points on RTLLM, with pass@5 gains up to 18.4 and 12.7 percentage points. It argues these gains are not from model scale, since a 70B model trained on prior RTL data underperforms the 32B ScaleRTL, and not from generic reasoning ability, since general reasoning models without RTL fine-tuning lag behind. ScaleRTL also preserves general coding and instruction-following performance on HumanEval, MBPP, and IFEval, where prior RTL-specialized models degrade.

Load-bearing premise

The training data is clean: the contamination filter (5-gram Jaccard similarity above 0.8 plus an embedding-based semantic filter) removes every near-duplicate of the VerilogEval and RTLLM reference solutions from the 62,000 scripts, and if any benchmark problem slipped through, the reported gains would be inflated.

Editorial extensions

If this is right

  • RTL code generation responds to data scale: expanding the training corpus from under 200 million tokens to 3.5 billion tokens through long reasoning traces yields double-digit gains on functional correctness benchmarks.
  • Test-time scaling laws transfer to hardware description: accuracy improves with reasoning-token count in a roughly logarithmic curve before saturating across all three benchmark subsets.
  • Reasoning fine-tuning need not erode general ability: ScaleRTL matches or exceeds its base reasoning model on HumanEval, MBPP, and IFEval, while prior RTL-specialized models lose those capabilities.
  • The corrective-prompt loop beats naive 'Wait' token extension, which tends to make the model reaffirm its original wrong solution instead of revising it.
  • A 32B model with this recipe outperforms larger RTL models and, on RTLLM and VerilogEval-Human, also beats recent closed-source models that the authors exclude from the main comparison because of likely benchmark contamination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the same data-and-inference scaling recipe to other hardware-design languages and artefacts, such as testbenches, assertions, and synthesis scripts, where the bottleneck is also scarce reasoning data rather than raw code volume.
  • The reported gains depend on the contamination filter's completeness; a held-out set of newly written RTL problems would tell how much of the advantage survives without any chance of benchmark leakage.
  • Because the corrective prompt's RTL rules are generated offline, the method's ceiling may track the quality of those rules; automating rule mining from the model's own failures is a plausible next step.
  • The paper leaves open combining ScaleRTL with agentic debugging tools; the pass-rate results suggest the iterative loop already solves many problems that individual samples miss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces ScaleRTL, an RTL-specialized reasoning LLM obtained by fine-tuning DeepSeek-R1-Distill-Qwen-32B on a curated dataset of 62K RTL scripts with DeepSeek-R1-generated chain-of-thought traces totaling 3.5B tokens, and ScaleRTL†, a test-time variant that iteratively appends corrective prompts to extend the reasoning trace. The authors report state-of-the-art functional correctness on VerilogEval and RTLLM, with gains up to 18.4% in pass@5 and 10.7% in pass@1 over prior RTL-specific baselines, and claim a test-time scaling law for RTL generation. The base ScaleRTL result is a substantive contribution; the test-time scaling claims, however, are evaluated with a protocol that uses the benchmark testbench as an oracle inside the inference loop, which conflates repair with one-shot generation.

Significance. If the base fine-tuning results are taken alone, the paper makes a solid empirical contribution: ScaleRTL-32B outperforms prior RTL-specialized models on VerilogEval-Human (76.3 vs. 69.7 pass@1 for CraftRTL-70B) and RTLLM (55.5 vs. 49.0 pass@1 for CraftRTL-15B) without any test-time repair, and Table 3 shows that general coding and instruction-following abilities are largely preserved. The 3.5B-token reasoning dataset is an order of magnitude larger than previous RTL training corpora, and the data-curation pipeline is described in enough detail to be reproducible. However, the paper's headline claim of state-of-the-art performance via test-time scaling is not supported by the current evaluation: Algorithm 1 calls IsCorrect(S) on the benchmark testbench and only then decides whether to continue reasoning, so ScaleRTL†-32B's reported pass@1 is the result of an oracle-guided repair process, not a single model sample. The purported scaling law in Figure 3 is also partly forced by this selection mechanism. The central data-scaling result is sound and useful, but the test-time scaling claims need a fundamentally different evaluation protocol before they can be accepted.

major comments (4)
  1. [§III-B, Algorithm 1] The reported pass@1 for ScaleRTL† is not comparable to the pass@1 of any baseline. Algorithm 1 executes Inference(M, P), checks IsCorrect(S) against the benchmark testbench, and, if the solution fails, appends a corrective prompt and re-runs the model for up to two additional passes. The final output is therefore the result of up to three attempts guided by ground-truth feedback. Baselines in Table 2 are evaluated with n=10 independent completions and no such feedback. Reporting the final repaired solution as pass@1 conflates an oracle-guided repair process with one-shot generation and inflates the comparison; with an unbounded repair loop, any model with a nonzero per-attempt success probability can be pushed toward arbitrary accuracy. The pass@5 and pass@1 gains in Section IV-B for ScaleRTL† therefore do not establish the claimed state-of-the-art. The authors should either report ScaleRTL† under a protocol where all baselines have access to the same testbench oracle and repair budget, or explicitly frame and label ScaleRTL† as an agentic/oracle-guided repair method and separate those numbers from single-pass pass@k results.
  2. [§IV-C, Figure 3] The 'test-time scaling law' in Figure 3 is partly an artifact of the selection mechanism. The two 'enhanced' variants are obtained by continuing inference only when the benchmark testbench has already declared the previous solution incorrect, so longer traces are not sampled unconditionally; they are traces that survived a correctness filter. The two 'downgraded' variants are truncated at fixed points, again without a corresponding oracle-based selection. Comparing these two families produces a curve that is not a scaling law for unguided reasoning length but a mixture of (a) truncation effects and (b) testbench-conditioned repair. To support the claimed log-linear relationship, the authors should plot performance against reasoning-token budgets for samples drawn unconditionally, or at minimum compare all methods under the same oracle-based budget (e.g., total testbench calls) and report pass@k computed over the entire multi-attempt trajectory rather than only the final accepted solution.
  3. [§III-A] The contamination-control claim is load-bearing for the base ScaleRTL result and is not supported by the reported procedure. The paper states that scripts with Jaccard similarity above 0.8 to any golden solution are discarded and that an embedding-based semantic filter is then applied, and later asserts 'we ensure that our CoT dataset contains no specification or code overlapping with existing RTL benchmarks.' A Jaccard threshold of 0.8 on 5-grams is very permissive: a solution that shares a large fraction of a benchmark module's structure, or a specification that paraphrases the benchmark prompt, can easily fall below this threshold. The embedding filter is described only by citation and its operating point is not given. The authors should report the number of samples removed by each filtering stage, present a nearest-neighbor analysis between the final 62K scripts and the VerilogEval/RTLLM golden solutions and prompts, and describe the human or automated verification that supports the 'no overlap' statement. Without this, the possibility of benchmark leakage into the 3.5B-token training corpus remains open and directly affects the validity of the base fine-tuning numbers.
  4. [Appendix D, Table 5] The 'pass rate' metric in Appendix D is presented as a more meaningful indicator for the test-time scaling approach, but it introduces an additional evaluation inconsistency. Pass rate counts a problem as solved if any of n=10 trials succeeds, while for ScaleRTL† each 'trial' can itself include up to three oracle-guided repair passes. This metric is not defined for the baselines and cannot be compared with the pass@1/pass@5 numbers in Table 2. If the authors wish to use this metric, they should define it precisely for all methods, including the repair budget and how failures within a trial are counted, or remove it from the main argument.
minor comments (4)
  1. [Section V] The conclusion contains a typo: 'This is acheived' should be 'This is achieved.'
  2. [Section IV-B] The sentence 'test-time scaling plays a critical in accurate RTL code generation' is missing a noun; it should read 'plays a critical role in accurate RTL code generation.'
  3. [Figure 4] The example in Figure 4 is internally inconsistent about the timing of the done signal: the problem statement says 'signal done in the cycle immediately after the third byte of each message was successfully received', while the subsequent text says the model should assert done 'in the same cycle' as the third byte. The authors should clarify which timing is correct and ensure the figure's text matches the VerilogEval problem specification.
  4. [Section IV-D, Table 3] The text says ScaleRTL achieves results 'on par with, and in some cases surpassing, the original DeepSeek-R1-Distill-Qwen-32B on all benchmarks,' but Table 3 shows IFEval accuracy of 73.2 for ScaleRTL vs. 76.8 for the base model, which is not surpassing. The sentence should be adjusted to reflect the actual direction of the differences.

Circularity Check

1 steps flagged · score 6.0 of 10

The test-time scaling law and ScaleRTL†'s reported gains are partly forced by the testbench oracle inside Algorithm 1; the base fine-tuning result remains independent.

  1. fitted input called prediction [Section III-B, Algorithm 1; Section IV-C, Figure 3]
    "If the solution fails to pass the testbench, the end-of-reasoning token delimiter “ </think>” is then replaced with a corrective prompt: ... while t < Tand not IsCorrect(S) do"

    The loop extends the reasoning trace only when IsCorrect(S) is false, and stops as soon as the solution passes. Thus the longer reasoning traces plotted in Figure 3 are not sampled unconditionally from the model; they are selected by the same benchmark testbench used for evaluation. By construction, each additional iteration can only keep or improve the measured pass score, so the positive length-accuracy trend is an artifact of the stopping rule rather than an independent empirical scaling law. The same oracle feedback means ScaleRTL†'s reported pass@1 is the outcome of up to three testbench-guided repair attempts, not a single-pass sample comparable to the baseline pass@1 values.

full rationale

Base ScaleRTL is obtained by supervised fine-tuning on 62K DeepSeek-R1-generated CoT traces and is evaluated against external benchmarks, so the core fine-tuning contribution is not circular. The main circularity burden is concentrated in the test-time scaling claim. Algorithm 1 gates trace extension on the benchmark testbench, so the length-accuracy trend in Figure 3 is an artifact of selective continuation: only failed solutions receive extra tokens, and the loop terminates on success. This makes the claimed 'test-time scaling law' a property of the stopping rule rather than an independent scaling result, and it makes ScaleRTL†'s pass@1 a repair-process outcome rather than a single-sample pass@1. The paper's citations to VerilogEval and CraftRTL have overlapping authorship but are ordinary benchmarking references and are not load-bearing: the method does not depend on an unverified theorem from those papers. Overall, the fine-tuning contribution stands, but the headline test-time scaling result reduces in part to its own oracle feedback, giving a partial circularity score of 6.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on dataset construction choices (Jaccard threshold, filtering, token limits), the assumption that teacher-generated traces are correct, and the availability of a testbench oracle at inference time. No new physical or mathematical entities are introduced.

free parameters (5)
  • Jaccard contamination threshold = 0.8
    Chosen to discard web-scraped code near-duplicate of benchmark golden solutions; directly affects leakage risk and dataset size.
  • Maximum test-time scaling iterations = 2
    Algorithm 1 loops until correct or T; the paper applies two additional inference passes per problem. This choice affects both compute cost and reported accuracy.
  • Inference temperature = 0.2
    Sampling temperature used for all generations; affects pass@k estimates and the behavior of the corrective loop.
  • Token limits for reasoning traces = 16,384 / 32,768 / 49,152
    Initial and two extended token budgets for test-time scaling; chosen to allow progressively longer reasoning.
  • Corrective reasoning rules = Claude3.7-Sonnet generated rules
    The prompt content injected into the corrective prompt is a hand-generated component that influences whether the model self-corrects effectively.
assumptions (4)
  • domain assumption DeepSeek-R1-generated chain-of-thought traces are correct and representative of good RTL reasoning.
    Section III-A uses these traces to build the 3.5B token training corpus; teacher errors would be distilled into ScaleRTL.
  • domain assumption The benchmark testbench is available at inference time to check correctness and drive the iterative loop.
    Algorithm 1 calls IsCorrect(S) and the paper's test-time scaling protocol relies on this oracle, which is not available in all deployment settings.
  • domain assumption The 5-gram Jaccard and embedding-based filters remove all benchmark contamination from the training data.
    Section III-A relies on these filters to claim no overlap with VerilogEval or RTLLM; if leakage remains, the benchmark gains are not trustworthy.
  • domain assumption Syntax validation and rule-based filtering retain a corpus representative of real RTL designs.
    Section III-A assumes the cleaned 62K scripts are high-quality and diverse enough to teach general RTL reasoning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ScaleRTL: Scaling LLMs with Reasoning Data and Test-Time Compute for Accurate RTL Code Generation." pith.science (2026). https://pith.science/paper/PCU7WAFV

@misc{pith2026250605566,
  author       = {Pith},
  title        = {Pith review of: ScaleRTL: Scaling LLMs with Reasoning Data and Test-Time Compute for Accurate RTL Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PCU7WAFV}},
  note         = {Machine review of arXiv:2506.05566}
}
read the original abstract

Recent advances in large language models (LLMs) have enabled near-human performance on software coding benchmarks, but their effectiveness in RTL code generation remains limited due to the scarcity of high-quality training data. While prior efforts have fine-tuned LLMs for RTL tasks, they do not fundamentally overcome the data bottleneck and lack support for test-time scaling due to their non-reasoning nature. In this work, we introduce ScaleRTL, the first reasoning LLM for RTL coding that scales up both high-quality reasoning data and test-time compute. Specifically, we curate a diverse set of long chain-of-thought reasoning traces averaging 56K tokens each, resulting in a dataset of 3.5B tokens that captures rich RTL knowledge. Fine-tuning a general-purpose reasoning model on this corpus yields ScaleRTL that is capable of deep RTL reasoning. Subsequently, we further enhance the performance of ScaleRTL through a novel test-time scaling strategy that extends the reasoning process via iteratively reflecting on and self-correcting previous reasoning steps. Experimental results show that ScaleRTL achieves state-of-the-art performance on VerilogEval and RTLLM, outperforming 18 competitive baselines by up to 18.4% on VerilogEval and 12.7% on RTLLM.

Figures

Figures reproduced from arXiv: 2506.05566 by the authors.

Figure 1
Figure 1. Comparison of LLMs on RTL Coding benchmarks — [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of our approach, which consists of two phases — In the first phase, we fine-tune a model on a large-scale curated [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Impact of reasoning trace length on RTL coding performance — For each benchmark, we evaluate five [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: A test-time scaling example in VerilogEval-Human. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Prompt for specification generation Prompt to generate Verilog solution with reasoning Your goal is to solve a Verilog problem by completing one Verilog module. • Guidelines for writing the solution (Verilog module) and formatting: 1) The module should be completely se…
Figure 6
Figure 6. Figure 6: Prompt for generating Verilog solution with reasoning [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Prompt for general RTL rules generation To enable ScaleRTL to rethink and self-correct earlier reasoning steps, we find that incorporating general RTL coding rules into the corrective prompt serves as effective hints, encouraging the model to generate longer chain-of-t…

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SCALE: Self-Supervised Constraint-Aware Layout GEneration for Local P&R DRV Fixing at Advanced Nodes

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Self-supervised generation of DRC-annotated layouts plus VLM fine-tuning lifts agent solve rates for local DRV fixing from 63–85% to 88–97% on 100 real sub-2nm cases.

  2. A Progressive Approach to Synthesizable RTL Design Generation Using LLMs

    cs.AR 2026-07 conditional novelty 6.0 of 10

    VeriRefine boosts LLM-generated RTL correctness to 94.0% on RTLLM v2.0 and 98.1% on VerilogEval-Human v2 by refining and auditing a per-signal intermediate representation before code generation.

  3. Spec2RTL-Agent: Automated Hardware Code Generation from Complex Specifications Using LLM Agent Systems

    cs.AR 2025-06 conditional novelty 6.0 of 10

    On three NIST crypto standards (AES, DSS, HMAC), Spec2RTL-Agent generates RTL via a multi-agent pipeline from pseudocode to Python to synthesizable C++, reporting 3/3 correct designs with about 4.3 human interventions...

Reference graph

Works this paper leans on

53 extracted references · 26 canonical work pages · cited by 3 Pith papers

  1. [1]

    Nemotron-4 340b technical report

    Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprint arXiv:2406.11704, 2024

    arXiv 2024

  2. [2]

    Claude (oct 8 version)

    Anthropic. Claude (oct 8 version). https://www.anthropic.com/, 2023. Large language model

    work page 2023

  3. [3]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021

    arXiv 2021

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  5. [5]

    Replug: Retrieval-augmented black-box language models

    Shizhuo Chen, Yuxiang Sun, Shunyu Yao, Dian Yu, Zhou Yu, and Dong Zhao. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652, 2023

    arXiv 2023

  6. [6]

    Origen: Enhancing rtl code generation with code-to-code augmentation and self-reflection

    Fan Cui, Chenyang Yin, Kexing Zhou, Youwei Xiao, Guangyu Sun, Qiang Xu, Qipeng Guo, Yun Liang, Xingcheng Zhang, Demin Song, et al. Origen: Enhancing rtl code generation with code-to-code augmentation and self-reflection. In Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design , pages 1–9, 2024

    work page 2024

  7. [7]

    Competitive programming with large reasoning models

    Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, Oleg Mürk, Rhythm Garg, Rui Shu, Szymon Sidor, Vineet Kosaraju, and Wenda Zhou. Competitive programmi...

    arXiv 2025

  8. [8]

    Gemini: A family of highly capable multimodal models

    Google Gemini Team. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2025

    arXiv 2025

Show all 53 references
  1. [9]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  2. [10]

    Deepseek- r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek- r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025

    2025 arXiv

  3. [11]

    Verilogcoder: Autonomous verilog coding agents with graph-based planning and abstract syntax tree (ast)-based waveform tracing tool

    Chia-Tung Ho, Haoxing Ren, and Brucek Khailany. Verilogcoder: Autonomous verilog coding agents with graph-based planning and abstract syntax tree (ast)-based waveform tracing tool. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 300–307, 2025

    2025

  4. [12]

    Train- ing compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  5. [13]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 , 2024

    2024 arXiv

  6. [14]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024

    2024 arXiv

  7. [15]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  8. [16]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Tanaka. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022

    2022 arXiv

  9. [17]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, p...

    2023

  10. [18]

    S*: Test time scaling for code generation

    Dacheng Li, Shiyi Cao, Chengkun Cao, Xiuyu Li, Shangyin Tan, Kurt Keutzer, Jiarong Xing, Joseph E Gonzalez, and Ion Stoica. S*: Test time scaling for code generation. arXiv preprint arXiv:2502.14382 , 2025

    2025 arXiv

  11. [19]

    Verilogeval: Evaluating large language models for verilog code generation

    Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. Verilogeval: Evaluating large language models for verilog code generation. In 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD), pages 1–8. IEEE, 2023

    2023

  12. [20]

    Craftrtl: High-quality synthetic data generation for verilog code models with correct-by-construction non-textual representations and targeted code repair

    Mingjie Liu, Yun-Da Tsai, Wenfei Zhou, and Haoxing Ren. Craftrtl: High-quality synthetic data generation for verilog code models with correct-by-construction non-textual representations and targeted code repair. arXiv preprint arXiv:2409.12993 , 2024

    2024 arXiv

  13. [21]

    Rtlcoder: Fully open-source and efficient llm-assisted rtl code generation technique

    Shang Liu, Wenji Fang, Yao Lu, Jing Wang, Qijun Zhang, Hongce Zhang, and Zhiyao Xie. Rtlcoder: Fully open-source and efficient llm-assisted rtl code generation technique. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 2024

    2024

  14. [22]

    Starcoder 2 and the stack v2: The next generation

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173 , 2024

    2024 arXiv

  15. [23]

    Rtllm: An open-source benchmark for design rtl generation with large language model

    Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xie. Rtllm: An open-source benchmark for design rtl generation with large language model. In 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC) , pages 722–727. IEEE, 2024

    2024

  16. [24]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Xiang Lin, Zhou Liu, Yulia Tsvetkov, and Graham Neubig. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023

    2023 arXiv

  17. [25]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  18. [26]

    Gpt-3.5 models

    OpenAI. Gpt-3.5 models. https://platform.openai.com/docs/models/ gpt-3-5, 2023. Accessed: 2025-05-22

    2023

  19. [27]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  20. [28]

    Betterv: Controlled verilog generation with discriminative guidance

    Zehua Pei, Hui-Ling Zhen, Mingxuan Yuan, Yu Huang, and Bei Yu. Betterv: Controlled verilog generation with discriminative guidance. arXiv preprint arXiv:2402.03375, 2024

    2024 arXiv

  21. [29]

    Code llama: Open foundation models for code

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  22. [30]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 , 2024

    2024 arXiv

  23. [31]

    Chain of thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022

    2022 arXiv

  24. [32]

    Magicoder: Source code is all you need

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120, 2023

    2023 arXiv

  25. [33]

    Mg-verilog: Multi-grained dataset towards enhanced llm-assisted verilog generation

    Yongan Zhang, Zhongzhi Yu, Yonggan Fu, Cheng Wan, and Yingyan Ce- line Lin. Mg-verilog: Multi-grained dataset towards enhanced llm-assisted verilog generation. In 2024 IEEE LLM Aided Design Workshop (LAD) , pages 1–5. IEEE, 2024

    2024

  26. [34]

    Codev: Empowering llms for verilog generation through multi-level summarization

    Yang Zhao, Di Huang, Chongxiao Li, Pengwei Jin, Ziyuan Nan, Tianyun Ma, Lei Qi, Yansong Pan, Zhenxing Zhang, Rui Zhang, et al. Codev: Empowering llms for verilog generation through multi-level summarization. arXiv preprint arXiv:2407.10424 , 2024

    2024 arXiv

  27. [35]

    Deepseek-coder- v2: Breaking the barrier of closed-source models in code intelligence

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder- v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931 , 2024. APPENDIX A PROMPTS FOR REASONING...

    2024 arXiv

  28. [36]

    This should be completely self-contained, providing all the contextual information one needs to understand and solve the problem

  29. [37]

    Assume common Verilog knowledge, but ensure that any specific context, variables, or code snippets pertinent to this problem are explicitly included

  30. [38]

    Do not include the code snippet in the problem

  31. [39]

    Here is an example: • Guidelines for the problem description format: The problem description section should be enclosed within <PROBLEM> </PROBLEM> tags

    The problem should be designed for the programmer to solve with one Verilog module. Here is an example: • Guidelines for the problem description format: The problem description section should be enclosed within <PROBLEM> </PROBLEM> tags. Below shows an example: Output: <PROBLE...

  32. [40]

    Your new problem should not be directly solved by the original code snippet

  33. [41]

    If you do have a reset method that is synchronous to a clock, make sure to add the clock signal to the problem module input

    You can also change the bit-width requirement, how to reset internal signals (if applicable), and whether the solution needs a clock signal (combinatorial versus sequential logic). If you do have a reset method that is synchronous to a clock, make sure to add the clock signal ...

  34. [42]

    Add new constraints and requirements to the original problem, adding approximately 10 additional words

  35. [43]

    Replace a commonly used requirement in the programming task with a less common and more specific one

  36. [44]

    If the original problem can be solved with only a few logical steps, please add more reasoning steps. Code snippet for inspiration: {code} Output: Figure 5: Prompt for specification generation Prompt to generate Verilog solution with reasoning Your goal is to solve a Verilog p...

  37. [45]

    The module should be completely self-contained , fulfilling all the requirements needed to solve the problem

  38. [46]

    Ensure the correctness of the syntax and functionality

  39. [47]

    The solution must be valid Verilog code

  40. [48]

    Do not include or import outside the module and define everything inside the module

  41. [49]

    • Below shows an example: Problem description: Build a counter that counts from 0 to 999, inclusive, with a period of 1000 cycles

    The solution section should be enclosed within <answer> </answer> tags. • Below shows an example: Problem description: Build a counter that counts from 0 to 999, inclusive, with a period of 1000 cycles. ... Output: <think> Okay, so I need to design a counter that goes from 0 t...

  42. [50]

    Potential Pitfalls Analysis : Identify the key challenges and tricky aspects of this problem that could lead to incorrect code generation. This could include: • Complex syntax requirements • Subtle logic implementation details • Problem interpretation challenges • Timing-sensi...

  43. [51]

    Critical Implementation Areas : Point out the specific aspects of the problem that are most prone to errors, with clear explanations of what makes these areas challenging

  44. [52]

    Problem Complexity Assessment : Analyze the inherent complexity of the problem and identify areas where careful attention to detail is required for correct implementation

  45. [53]

    General Verilog Coding Rules : Based on the potential challenges identified, formulate concise, general Verilog coding rules that would help prevent common mistakes. These rules should: • Be widely applicable to Verilog coding, not just this specific example • Focus on best pr...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.