Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

CoopetitiveV: Leveraging LLM-powered Coopetitive Multi-Agent Prompting for High-quality Verilog Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a coopetitive multi-agent prompting loop, with a prosecutor agent that critiques a research agent's repair strategy, raises LLM Verilog generation to 99.2% pass@10 on VerilogEval-Machine and 99.1% on VerilogEval-Human.

desk verdict The method is a plausible multi-agent extension, but the evaluation oracle is self-referential—the pipeline writes and revises its own testbenches—so the near-perfect pass rates don't demonstrate functional correctness. read the letter →

arxiv 2412.11014 v2 pith:YNTI4OQF submitted 2024-12-15 cs.LG cs.AIcs.ARcs.PLcs.SE

Zhendong Mi , Renming Zheng , Haowen Zhong , Yue Sun , Seth Kneeland , Sayan Moitra , Ken Kutzer , Zhaozhuo Xu Shaoyi Huang This is my paper · ORCID
classification cs.LGcs.AIcs.ARcs.PLcs.SE
keywords Veriloggenerationmulti-agentLLMscoopetitivepromptingcoderepairhardwaredesignautomationtestbenchpass@kevaluationerrorpropagation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes CoopetitiveV, a prompting framework in which several large language model agents cooperate to write Verilog code and then compete to correct it. The central claim is that adding a prosecutor agent, which challenges and refines the strategy of a research agent before revision agents apply it, fixes two failure modes: single-agent degeneration, where an agent repeats its own wrong fixes, and cooperation-only error propagation, where one agent's bad advice flows downstream. If true, the framework lifts pass@10 on VerilogEval-Machine to 99.2% and on VerilogEval-Human to 99.1% with GPT-4, near-saturating these benchmarks. This matters because hardware designers could delegate more RTL module generation and debugging to LLM agents, provided the verification signal driving corrections is trustworthy.

What carries the argument

The load-bearing mechanism is the prosecutor agent inserted between the research agent and the revision agents. In the first correction round, the research agent's strategy goes directly to the revision agents, forming a cooperation-only loop. If simulation still fails, in later rounds the prosecutor agent must score, challenge, and refine that strategy into a more comprehensive one before the revision agents apply it, creating the competitive component. The whole loop is wrapped around Icarus Verilog simulation driven by AutoBench-generated testbenches, which supply the pass/fail signal that decides whether another round starts.

What would settle it

Run CoopetitiveV with GPT-4 on VerilogEval and RTLLM but grade the generated modules with the datasets' official, independently written testbenches instead of AutoBench-generated ones. The central claim would be falsified if pass@10 on VerilogEval-Machine drops by more than a few points relative to the reported 99.2%.

Watch

Extended reading notes

Core claim

The paper's central discovery is that competition, not just cooperation, among LLM agents is what prevents correction loops from stalling or compounding errors in Verilog generation. In the proposed pipeline, one agent generates code, another generates a testbench, a research agent analyzes simulation failures and proposes fixes, a prosecutor agent must critique and improve that proposal, and two revision agents apply the final guidance to code and testbench separately. The paper reports that this coopetitive loop, with GPT-4 or Claude-3.5, reaches 99.2% pass@10 on VerilogEval-Machine and 99.1% pass@10 on VerilogEval-Human, plus 100% syntax and 99.9% functionality pass@5 on RTLLM. It frames the result as evidence that the prosecutor's oversight interrupts the propagation of faulty correction strategies that plagues cooperation-only multi-agent pipelines.

Load-bearing premise

The entire correction loop is driven by AutoBench-generated testbenches, and those testbenches are produced from the module description together with the candidate code; if they are not independent of the code under test, the reported pass@k scores overstate true functional correctness.

Editorial extensions

If this is right

  • If the reported scores hold, LLM-based RTL generation reaches near-ceiling performance on VerilogEval and RTLLM, leaving testbench quality rather than code quality as the main remaining bottleneck.
  • The framework should reduce the number of generation attempts needed to obtain correct Verilog, because pass@5 and pass@10 are high even when pass@1 is moderate.
  • The competition mechanism can be applied to any iterative code repair loop to break chains of bad suggestions, not just hardware description languages.
  • The gap between machine and human VerilogEval scores shrinks from tens of points in the baselines to a few points, suggesting the correction loop generalizes beyond synthetic problems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if AutoBench-generated testbenches were replaced by independent held-out testbenches, the reported pass rates would likely drop because the oracle and the code under test come from overlapping generation processes.
  • Editorial inference: the same prosecutor-plus-revision loop could be tested on Python or C++ repair, where official unit tests exist, to see whether competition helps beyond Verilog-specific syntax quirks.
  • Editorial inference: a per-round ablation would likely show that the prosecutor's benefit concentrates in the second correction round, since the paper only reports overall gains and not round-by-round behavior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CoopetitiveV, a multi-agent LLM prompting framework for Verilog code generation. The system includes a code generation agent, an AutoBench-based testbench generation and simulation step, a research agent that proposes correction strategies, a prosecutor agent that critiques and refines those strategies, and separate revision agents for code and testbench. The authors report state-of-the-art pass@k numbers on VerilogEval-Machine, VerilogEval-Human, Verilog-V2, and RTLLM, including 99.2% and 99.1% pass@10 for CoopetitiveV+GPT-4, and claim that the framework mitigates degeneration and error propagation while improving code error correction.

Significance. If the results were valid, CoopetitiveV would be a useful contribution to agentic RTL code generation, with a clearly described multi-agent architecture, a broad comparison across models and datasets, and an ablation of the prosecutor agent. The paper also makes a reasonable distinction between single-agent degeneration and cooperation-only error propagation. However, the evaluation's validity hinges on an oracle that is generated and revised by the pipeline itself. Because the testbench is not independent of the implementation, the headline accuracy numbers cannot be interpreted as functional correctness against a fixed specification, and the comparison with baselines evaluated on official testbenches is not apples-to-apples.

major comments (4)
  1. [Section 2.3.1] The functional oracle is endogenous: the displayed equation TB0, Pass0, E0 = A(Code0, m) shows that AutoBench generates the testbench from the module description and the candidate code, and Figure 4 instructs the testbench generation agent to 'generate testbench based on the RTL code.' A module can therefore pass by satisfying whatever behaviors the generated testbench happens to check, rather than the full specification. Since Table 1 baselines are evaluated on the fixed official testbenches of VerilogEval and RTLLM, the comparison is not apples-to-apples.
  2. [Sections 2.3.5, 2.3.6, 2.4] The correction loop revises the testbench together with the code. The research agent produces both code and testbench strategies (S_c, S_t), the revision agents L_c and L_t apply them in parallel, and the prosecutor refines both; Section 2.4 terminates when Pass_{k+1}=True is obtained from Icarus Verilog on the jointly revised pair (Code_{k+1}, TB_{k+1}). Consequently a buggy implementation can be certified if the testbench is edited to match its behavior. The reported pass@k therefore measures self-consistency between generated code and generated testbench, not correctness against a fixed specification.
  3. [Section 3.2, Table 1] The headline claims (99.2% and 99.1% pass@10 for CoopetitiveV+GPT-4) are computed under this circular oracle. The pattern of much higher pass@5/pass@10 than pass@1 is consistent with the pipeline searching over both code and testbench until a self-consistent pair passes. The paper does not report any verification of the final testbenches against reference testbenches or by independent human inspection; without such evidence, the central claim of 'higher quality Verilog code generation' is unsupported. The ablation in Section 3.3 suffers from the same issue and does not establish that the prosecutor improves functional correctness.
  4. [Section 5, Limitations] The limitations paragraph only notes the restriction to Verilog and does not acknowledge the dependence of the evaluation oracle on the method's own testbench generation and revision. This omission is material because the validity of every reported pass@k number is at stake. The paper should explicitly state that all pass/fail signals come from AutoBench-generated testbenches and should provide a separate evaluation on fixed reference testbenches before claiming state-of-the-art performance.
minor comments (5)
  1. [Abstract] The phrase 'the agents cannot collaborate with each other to form the generation pipeline, but also create' appears to be a typo; it should read 'can not only collaborate' or 'not only can collaborate.'
  2. [Section 3.2] In the paragraph explaining the pass@1 versus pass@5/pass@10 pattern, '20 trails' should be '20 trials.'
  3. [Table 1] The 'Open source' column marks CoopetitiveV as 'Y', but no repository or artifact link is provided in the paper; please clarify how the implementation can be accessed or reproduced.
  4. [Equation (1)] The pass@k estimator is defined with n=20, but it is not stated whether pass/fail is determined after the full correction loop or on the initial code generation. Since the correction loop is stochastic and may alter the testbench, the meaning of 'passed trials' should be made explicit.
  5. [Figure 4] The figure caption and prompt text contain duplicated fragments such as 'Only use cooperation framework may keep propagatingthe error.' Please clean up the figure so the reader can follow the pipeline without confusion.

Circularity Check

2 steps flagged · score 6.0 of 10

The functional-correctness oracle is endogenous: testbenches are generated and revised from the candidate code, so pass@k measures self-consistency, not conformance to a fixed specification.

  1. self definitional [Section 2.3.1, equation (TB0, Pass0, E0) = A(Code0, m)]
    "Specifically, the AutoBench, denoted as A, not only synthesizes appropriate verification environments by generating the testbench but also performs code compilation through the embedded Iverilog... We formulate the Autobench generation as well as the code and testbench syntax verification process as follows: TB0, Pass0, E0 = A(Code0, m)."

    The pass/fail signal Pass0 is defined by a testbench TB0 that is generated from the candidate Code0 and the module information m. Therefore a 'pass' means only that the code is consistent with the testbench the pipeline itself wrote for it, not that the code satisfies the fixed benchmark specification. Bugs not exercised by the self-generated testbench are invisible, so the pass labels are functions of the candidate code rather than outputs of an independent oracle.

  2. fitted input called prediction [Sections 2.3.5, 2.3.6, and 2.4 (TB_{k+1} and Pass_{k+1})]
    "Testbench revision agent Lt dedicated to testbench refinement. When Passk = False in the previous simulation step, the dual learner agents generate the corrected Verilog code Codek+1 and the refined testbench TBk+1. ... Passk+1, Ek+1 = Iv(Codek+1, TBk+1)."

    The correction loop revises the testbench alongside the code until the pair passes Iverilog, so the final 'Pass' is a fixed point of jointly generating code and testbench, not a check against a fixed specification. A functionally incorrect module can be reported as passing if the LLM-written testbench is modified to avoid the faulty path. Since the headline pass@k values are computed from these endogenous Pass values and baselines are scored on fixed testbenches, the comparison is not apples-to-apples and the claimed quality improvement reduces to the pipeline fitting its own testbench.

full rationale

The paper's central claim of 'higher quality Verilog code generation' rests on pass@k scores whose oracle is the pipeline's own AutoBench/Iverilog testbench loop. Section 2.3.1 defines the oracle as TB0, Pass0, E0 = A(Code0, m), making the initial pass label a function of the candidate code. Sections 2.3.5 and 2.3.6 then introduce testbench revision agents that refine TB_{k+1} alongside Code_{k+1}, and Section 2.4 accepts the loop when Iv(Code_{k+1}, TB_{k+1}) returns Pass=True. The loop can therefore terminate by making code and testbench jointly self-consistent: a buggy implementation is deemed correct once the LLM-written testbench no longer exercises the faulty behavior. The reported 99.2% VerilogEval-Machine, 99.1% VerilogEval-Human, and 99.9% RTLLM functionality figures are thus not evidence of conformance to the fixed specifications used to score the baselines in Table 1. The limitation section does not acknowledge this self-referential evaluation. No self-citation chain or imported uniqueness argument is involved; the circularity is in the evaluation oracle. Because the multi-agent framework could in principle be measured against fixed official testbenches, the circularity is partial but load-bearing for the main quality claim.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new physical entities are introduced; the agents are roles within an LLM prompting framework. The free parameters are iteration limits and sample counts, and the key axioms concern the validity of AutoBench-based evaluation and the comparability of baselines.

free parameters (2)
  • max correction iterations k = 2
    The upper limit for correction rounds is set to 2 in experiments without a sensitivity study; this choice can affect the pass@k results.
  • number of samples n = 20
    pass@k is estimated with n=20 samples per problem, which is a standard but arbitrary choice.
assumptions (3)
  • domain assumption AutoBench-generated testbenches are valid oracles for functional correctness
    The entire evaluation pipeline (Section 2.3.1) treats Pass from Icarus simulation of AutoBench-generated testbenches as ground truth.
  • domain assumption The prosecutor agent's critique reduces error propagation
    The paper assumes the competitive mechanism improves outcomes; the ablation supports it only for GPT-3.5 on two datasets.
  • domain assumption Baseline numbers from prior work were obtained under comparable evaluation conditions
    Table 1 compares CoopetitiveV against published baselines without stating whether the same testbench generation pipeline was used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoopetitiveV: Leveraging LLM-powered Coopetitive Multi-Agent Prompting for High-quality Verilog Generation." pith.science (2026). https://pith.science/paper/YNTI4OQF

@misc{pith2026241211014,
  author       = {Pith},
  title        = {Pith review of: CoopetitiveV: Leveraging LLM-powered Coopetitive Multi-Agent Prompting for High-quality Verilog Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YNTI4OQF}},
  note         = {Machine review of arXiv:2412.11014}
}
read the original abstract

Recent advances in agentic LLMs have demonstrated great capabilities in Verilog code generation. However, existing approaches either use LLM-assisted single-agent prompting or cooperation-only multi-agent learning, which will lead to: (i) Degeneration issue for single-agent learning: characterized by diminished error detection and correction capabilities; (ii) Error propagation in cooperation-only multi-agent learning: erroneous information from the former agent will be propagated to the latter through prompts, which can make the latter agents generate buggy code. In this paper, we propose an LLM-based coopetitive multi-agent prompting framework, in which the agents cannot collaborate with each other to form the generation pipeline, but also create a healthy competitive mechanism to improve the generating quality. Our experimental results show that the coopetitive multi-agent framework can effectively mitigate the degeneration risk and reduce the error propagation while improving code error correction capabilities, resulting in higher quality Verilog code generation. The effectiveness of our approach is validated through extensive experiments. On VerilogEval Machine and Human dataset, CoopetitiveV+GPT-4 achieves 99.2% and 99.1% pass@10 scores, respectively. While on RTLLM, CoopetitiveV+GPT-4 obtains 100% syntax and 99.9% functionality pass@5 scores.

Figures

Figures reproduced from arXiv: 2412.11014 by the authors.

Figure 1
Figure 1. (a) Degeneration issue in single-agent based [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. LLM-based multi-agent learning for Verilog generation with (a) Cooperation-only mechanism; (b) [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The proposed LLM-powered multi-agent prompting framework for Verilog generation [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Verilog generation task 2 Method 2.1 Task Description While LLM research has made considerable progress in software programming languages such as Python and JavaScript, its application to hard￾ware description languages like Verilog remains relatively underexplored. In…
Figure 5
Figure 5. Figure 5: Comparison of without and with prosecutor [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VerilogDB: The Largest, Highest-Quality Dataset with a Preprocessing Framework for LLM-based RTL Generation

    cs.AR 2025-07 conditional novelty 4.0 of 10

    A new pipeline and dataset of 20,392 synthesis-checked Verilog modules for LLM fine-tuning is presented, claimed to be the largest high-quality dataset of its kind.

Reference graph

Works this paper leans on

39 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  2. [2]

    Junzhe Chen, Xuming Hu, Shuodi Liu, Shiyu Huang, Wei-Wei Tu, Zhaofeng He, and Lijie Wen. 2024. Llmarena: Assessing capabilities of large language models in dynamic multi-agent environments. arXiv preprint arXiv:2402.16499

    arXiv 2024

  3. [3]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond \'e , Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 34 others. 2021. https://api.semanticscholar.org/CorpusID:235755472 Evaluating lar...

    arXiv 2021

  4. [4]

    Matthew T Dearing, Yiheng Tao, Xingfu Wu, Zhiling Lan, and Valerie Taylor. 2024. Lassi: An llm-based automated self-correcting pipeline for translating parallel scientific codes. arXiv preprint arXiv:2407.01638

    arXiv 2024

  5. [5]

    Abhimanyu Dubey and 1 others. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    arXiv 2024

  6. [6]

    Shuzheng Gao, Cuiyun Gao, Wenchao Gu, and Michael Lyu. 2024. Search-based llms for code optimization. arXiv preprint arXiv:2408.12159

    arXiv 2024

  7. [7]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, and 1 others. 2024. Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196

    arXiv 2024

  8. [8]

    Chia-Tung Ho, Haoxing Ren, and Brucek Khailany. 2025. Verilogcoder: Autonomous verilog coding agents with graph-based planning and abstract syntax tree (ast)-based waveform tracing tool. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 300--307

    work page 2025

Show all 39 references
  1. [9]

    Dong Huang, Qingwen Bu, and Heming Cui. 2023 a . Codecot and beyond: Learning to program and test like a developer. arXiv preprint arXiv:2308.08784

    2023 arXiv

  2. [10]

    Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui. 2023 b . Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010

    2023 arXiv

  3. [11]

    Hanxian Huang, Zhenghan Lin, Zixuan Wang, Xin Chen, Ke Ding, and Jishen Zhao. 2024. https://api.semanticscholar.org/CorpusID:270216110 Towards llm-powered verilog rtl assistant: Self-verification and self-correction . ArXiv, abs/2406.00115

    2024 arXiv

  4. [12]

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118

    2023 arXiv

  5. [14]

    Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, and 1 others. 2023 a . Chipnemo: Domain-adapted llms for chip design. arXiv preprint arXiv:2311.00176

    2023 arXiv

  6. [15]

    Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. 2023 b . Verilogeval: Evaluating large language models for verilog code generation. In ICCAD, pages 1--8. IEEE

    2023

  7. [16]

    Shang Liu, Wenji Fang, Yao Lu, Qijun Zhang, Hongce Zhang, and Zhiyao Xie. 2024 b . Rtlcoder: Outperforming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution. In 2024 IEEE LLM Aided Design Workshop (LAD), pages 1--5. IEEE

    2024

  8. [17]

    Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xie. 2024. Rtllm: An open-source benchmark for design rtl generation with large language model. In ASP-DAC, pages 722--727. IEEE

    2024

  9. [18]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36

    2024

  10. [19]

    Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2023. Is self-repair a silver bullet for code generation? In ICLR

    2023

  11. [20]

    OpenAI, Josh Achiam, and 1 others. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  12. [21]

    Hammond Pearce, Benjamin Tan, and Ramesh Karri. 2020. Dave: Deriving automatically verilog from english. In Proceedings of the 2020 ACM/IEEE Workshop on Machine Learning for CAD, pages 27--32

    2020

  13. [22]

    Zehua Pei, Hui-Ling Zhen, Mingxuan Yuan, Yu Huang, and Bei Yu. 2024. Betterv: Controlled verilog generation with discriminative guidance. arXiv preprint arXiv:2402.03375

    2024 arXiv

  14. [23]

    Ruidi Qiu, Grace Li Zhang, Rolf Drechsler, Ulf Schlichtmann, and Bing Li. 2024. Autobench: Automatic testbench generation and evaluation using llms for hdl design. In Proceedings of the 2024 ACM/IEEE International Symposium on Machine Learning for CAD, pages 1--10

    2024

  15. [24]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, and 1 others. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

    2023 arXiv

  16. [25]

    Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, and 1 others. 2023. Pangu-coder2: Boosting large language models for code with ranking feedback. arXiv preprint arXiv:2307.14936

    2023 arXiv

  17. [26]

    Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. 2023. Repository-level prompt generation for large language models of code. In ICML, pages 31693--31715. PMLR

    2023

  18. [27]

    Hugo Touvron and 1 others. 2023. https://arxiv.org/abs/2307.09288 Llama 2: Open foundation and fine-tuned chat models . Preprint, arXiv:2307.09288

    2023 arXiv

  19. [28]

    Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O'Sullivan, and Hoang D Nguyen. 2025. Multi-agent collaboration mechanisms: A survey of llms. arXiv preprint arXiv:2501.06322

    2025 arXiv

  20. [29]

    Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh. 2024. Improving llm code generation with grammar augmentation. arXiv preprint arXiv:2403.01632

    2024 arXiv

  21. [30]

    Hanbin Wang, Zhenghao Liu, Shuo Wang, Ganqu Cui, Ning Ding, Zhiyuan Liu, and Ge Yu. 2024 a . Intervenor: Prompting the coding ability of large language models with the interactive chain of repair. In Findings of ACL, pages 2081--2107

    2024

  22. [31]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, and 1 others. 2024 b . A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345

    2024

  23. [32]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560

    2022 arXiv

  24. [33]

    Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang, and Jian-Guang Lou. 2022. Large language models meet nl2code: A survey. arXiv preprint arXiv:2212.09420

    2022 arXiv

  25. [34]

    Che Zhang, Zhenyang Xiao, Chengcheng Han, Yixin Lian, and Yuejian Fang. 2024. https://api.semanticscholar.org/CorpusID:267759831 Learning to check: Unleashing potentials for self-correction in large language models . ArXiv, abs/2402.13035

    2024 arXiv

  26. [35]

    Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023 a . Self-edit: Fault-aware code editor for code generation. arXiv preprint arXiv:2305.04087

    2023 arXiv

  27. [36]

    Tianyi Zhang, Tao Yu, Tatsunori Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida Wang. 2023 b . Coder reviewer reranking for code generation. In ICML, pages 41832--41846. PMLR

    2023

  28. [37]

    Yang Zhao, Di Huang, Chongxiao Li, Pengwei Jin, Ziyuan Nan, Tianyun Ma, Lei Qi, Yansong Pan, Zhenxing Zhang, Rui Zhang, and 1 others. 2024 a . Codev: Empowering llms for verilog generation through multi-level summarization. arXiv preprint arXiv:2407.10424

    2024 arXiv

  29. [38]

    Yujie Zhao, Hejia Zhang, Hanxian Huang, Zhongming Yu, and Jishen Zhao. 2024 b . Mage: A multi-agent engine for automated rtl code generation. arXiv preprint arXiv:2412.07822

    2024 arXiv

  30. [39]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  31. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.