Pith. sign in

REVIEW 4 major objections 6 minor 27 references

QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces QHackBench, a 49-challenge PennyLane benchmark from QHack 2023–2024, and reports that current LLMs solve under half the tasks, with RAG giving mixed results while multi-agent refinement raises execution success.

desk verdict Useful PennyLane benchmark, but the paper's RAG claim is contradicted by its own table and the contamination threat it flags is unexamined. read the letter →

arxiv 2506.20008 v2 pith:UNP6HTOB submitted 2025-06-24 cs.AI cs.PLcs.SE

Abdul Basit , Minghao Shao , Muhammad Haider Asif , Nouhaila Innan , Muhammad Kashif , Alberto Marchisio , Muhammad Shafique This is my paper
classification cs.AIcs.PLcs.SE
keywords quantumcodegenerationPennyLanebenchmarklargelanguagemodelsretrieval-augmentedmulti-agentsystemsQHackprogramming
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces QHackBench, a benchmark of 49 real PennyLane coding challenges drawn from the QHack 2023 and 2024 hackathons, and uses it to measure how well large language models can write quantum code. Across five model families under plain prompting and retrieval-augmented generation (RAG), the best average accuracy is 49.4% (DeepSeek V3, non-RAG) and the strongest overall performer is LLaMa 4 at 47.0% with RAG. The authors find that RAG does not reliably help: four of five models score lower with retrieval, and two models (Claude 3.7 and Qwen 3) fail completely under RAG. A multi-agent pipeline in which a validator inspects runtime errors and asks the builder to regenerate code solves more challenges and raises execution success rates, particularly when paired with GPT-o3-Mini and DeepSeek V3. If the benchmark is sound, it gives the community a reusable yardstick for AI-assisted PennyLane programming and a way to improve reliability through iterative debugging.

What carries the argument

The load-bearing object is QHackBench itself: a curated set of 49 real hackathon challenges, each with a problem statement, a notebook template, and a hidden test function, plus an automated evaluation loop that injects generated code into a notebook executor and classifies outcomes (pass, syntax error, no output, incomplete, runtime error). On top of this sits a retrieval channel (vector search over a curated PennyLane code corpus) and a two-agent refinement loop in which a validator reads execution logs and reformulates the query before the builder regenerates code, with at most five attempts. The evaluation reports Pass@$k$, execution success rate, score, and error-type breakdown.

What would settle it

Run the same models on QHack 2025 challenges or a held-out set of newly written PennyLane tasks of matched difficulty; if accuracy collapses relative to the 2023 results (for example, DeepSeek V3's RAG accuracy dropping from 60.7% to 4.8%), the benchmark's absolute scores are inflated by memorized solutions.

Watch

Extended reading notes

Core claim

The paper's central claim is that QHackBench is the first benchmark for PennyLane-based LLM code generation, built from 28 QHack 2023 and 21 QHack 2024 challenges organized by theme, point value, and difficulty. Using an automated pipeline that executes generated code against hidden test suites, the paper establishes a baseline showing large performance heterogeneity: LLaMa 4 is the most consistent model (44.0% non-RAG, 47.0% RAG) and DeepSeek V3 is the best non-RAG model (49.4%), with most failures being 'no output' rather than syntax errors. The paper further claims that a RAG-augmented PennyLang corpus does not consistently improve accuracy and can be actively harmful, while a multi-agent validator-builder loop improves execution success and convergence speed on unsolved problems.

Load-bearing premise

The benchmark's accuracy numbers assume the QHack hidden test suites are a non-leaky ground truth and that the models have not memorized the public QHack 2023 solutions during pretraining, an assumption the authors themselves flag as possibly violated.

Editorial extensions

If this is right

  • QHackBench gives PennyLane researchers a standardized, reproducible way to compare LLMs on quantum coding, with difficulty tiers that track challenge point values.
  • Even the strongest evaluated model solves only about half the challenges, so current LLMs remain far from reliable for PennyLane code generation, especially on advanced tasks.
  • RAG's benefit is conditional: it slightly helps LLaMa 4 but degrades DeepSeek V3, GPT-4.1, Claude 3.7, and Qwen 3, so retrieval quality and model-retrieval compatibility must be assessed together.
  • The multi-agent validator-builder loop is a practical reliability gain, solving challenges that plain prompting and RAG cannot and converging in fewer iterations.
  • The sharp year-over-year accuracy drops (for example, DeepSeek V3 under RAG falling from 60.7% in 2023 to 4.8% in 2024) indicate both rising challenge difficulty and possible memorization of the public 2023 solutions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the authors flag possible contamination from the public 2023 set, the cleaner signal is the 2024 set and the relative gains of multi-agent refinement; absolute 2023 accuracy numbers may be inflated.
  • A direct test of contamination is to run the same models on QHack 2025 or newly written PennyLane challenges of matched difficulty; a large drop would confirm memorization.
  • Replacing binary pass/fail with partial credit or circuit-equivalence checks would separate semantic correctness from mere execution success and give a finer-grained ranking.
  • Adding a relevance threshold with a fallback to plain prompting when retrieved context is weak could repair the Qwen-3-style RAG failures documented in the paper's case study.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces QHackBench, a benchmark dataset of 49 PennyLane coding challenges drawn from the QHack 2023 and 2024 hackathons, together with an evaluation pipeline that measures functional correctness, execution success, and iteration counts under vanilla prompting, retrieval-augmented generation (RAG), and a multi-agent iterative refinement setup. Five general-purpose LLMs (Claude 3.7, DeepSeek V3, GPT-4.1, LLaMa 4, Qwen 3) are evaluated with and without RAG, and a multi-agent pipeline is demonstrated on a subset of 2024 challenges using o3-mini and DeepSeek V3. The paper claims that RAG-enhanced models perform approximately as well as standard prompting, that multi-agent refinement improves execution success, and that QHackBench provides a reusable yardstick for PennyLane code generation. The main reported finding from Table II is that RAG lowers accuracy in four of the five tested models and drives two models to 0.0% accuracy, while the multi-agent results are presented only as heatmaps without numeric accuracy or iteration statistics.

Significance. If validated, QHackBench would fill a genuine gap: there is no established PennyLane-specific benchmark for LLM code generation, and the QHack challenge format (public problem statements, templates, and hidden tests) is a sensible basis for a functional-correctness benchmark. The paper provides a detailed challenge inventory (Table I), a structured difficulty mapping, and an automated execution harness using Papermill and ChromaDB, which are useful methodological elements. The honest acknowledgment of a possible data-contamination threat in Section V is a point in the paper's favor. However, the central empirical contributions—the RAG comparison and the multi-agent improvement—are currently not supported by the reported evidence: the main accuracy table contradicts the abstract's RAG claim, the multi-agent results lack numeric reporting, and the acknowledged contamination risk is not analyzed. The benchmark itself is a reasonable contribution that could be made solid with corrected claims and additional analysis, but as it stands the manuscript overstates what its data show.

major comments (4)
  1. [Abstract and Section V, Table II] The abstract and conclusion state that RAG-enhanced models 'approximately generate similar results as the standard prompting' and that 'RAG boosts correctness,' but Table II shows the opposite for four of the five models: Claude 3.7 RAG 0.0% vs non-RAG 31.0%, DeepSeek V3 RAG 18.5% vs non-RAG 49.4%, GPT-4.1 RAG 26.8% vs non-RAG 34.5%, and Qwen 3 RAG 0.0% vs non-RAG 10.7%; only LLaMa 4 is slightly higher with RAG (47.0% vs 44.0%). This is a load-bearing contradiction that needs to be resolved either by revising the claims to reflect that RAG generally degrades or has no consistent benefit, or by providing the promised analysis of cases where RAG helps (e.g., complex algorithms). The current framing is misleading.
  2. [Section V, Figure 6] The multi-agent evaluation is presented only as heatmaps in Figure 6, with no numeric table of success rates, execution success rates, mean or median iterations, or Pass@k values. The claims that the multi-agent approach 'consistently outperforms both baselines' and 'further enhancing execution success rates' cannot be verified from the figure alone. Additionally, the heatmap covers only 18 of the 21 2024 challenges without explaining the omission. A table reporting per-method success counts, accuracy, and iteration statistics for the full challenge set is required to support the central multi-agent claim.
  3. [Section V, contamination discussion] Section V explicitly acknowledges that 'some models may have inadvertently memorized parts of the 2023 challenge set during pretraining,' and uses this to explain the large 2023-to-2024 accuracy drops. Because the QHack problem statements, templates, and official solutions are public (Refs. [26] and [27]), the benchmark scores may reflect memorization rather than quantum coding ability. No contamination analysis is provided: there is no comparison against a held-out or rephrased challenge set, no n-gram overlap measurement, and no discussion of whether RAG's retrieved PennyLang snippets overlap with QHack solutions. This threat undermines the benchmark's validity as a reusable yardstick and confounds both the RAG and multi-agent comparisons. The paper needs to either provide such an analysis or substantially qualify the scores as upper bounds.
  4. [Section V (all results)] No repeated runs or statistical measures are reported. With 49 binary challenges, a one-challenge difference changes accuracy by approximately two percentage points, so differences such as LLaMa 4's RAG-versus-non-RAG gap (47.0% versus 44.0%) and GPT-4.1's gap (26.8% versus 34.5%) are within the noise of a single run. The paper should report standard errors, repeated sampling (e.g., with temperature variation), or at least a sensitivity analysis, and should hedge conclusions accordingly. Without this, the comparative claims in Section V (such as 'LLaMa 4 demonstrates balanced performance, with a slight preference for RAG') are not statistically grounded.
minor comments (6)
  1. [General] There are several typos and inconsistencies: 'Out evaluation' in Section V should be 'Our evaluation'; 'LLaMa 4' appears with inconsistent capitalization; 'GPT-o3-Mini' and 'o3-mini' are used interchangeably; and the Conclusion mentions 'GPT-4o, GPT-o3-Mini, and DeepSeek-V3' while Table II does not include GPT-4o or an o3-mini baseline.
  2. [Section VI-A, Table III] The error rates in Table III are not clearly defined—it is ambiguous whether the percentages are over all attempts, all challenges, or only failing runs. The text should state the denominator explicitly so the reader can interpret, for example, the 55.4% 'NoOut' rate for LLaMa 4.
  3. [Section V] The paper defines Pass@k and mentions it in Section IV-C, but no Pass@k values are reported anywhere. Either report Pass@1, Pass@3, and Pass@5 results or remove the metric from the methodology to avoid a gap between definition and results.
  4. [Section VI-A] The treatment of 'No Output' as a model failure may conflate genuine model suppression with harness-side failures to extract code from a chat response that includes explanations. Clarify how code is extracted from the model output and whether 'NoOut' includes cases where the model returned text without a code block.
  5. [Figure 4 and Section IV-B] The text says 'Figure 4 shows a clear alignment between point values and difficulty levels,' but the figure appears to be a histogram of challenge point values, not a demonstration of alignment. Also, the sentence 'no tutorial-level challenges were included initially' is confusing because 2023 has eight tutorial challenges; rephrase for clarity.
  6. [References] Reference [16] is a PennyLane blog post about QHack 2022, which does not support the claim about QHack 2023–2024 challenges; consider replacing it with the official QHack 2023 and 2024 challenge repositories that are already listed as Refs. [26] and [27].

Circularity Check

0 steps flagged · score 2.0 of 10

QHackBench is externally grounded in QHack hidden tests, so the benchmark numbers are not circular; the only circularity-adjacent items are a same-group RAG corpus and an acknowledged contamination risk, which are validity concerns rather than self-referential reductions.

full rationale

The paper's central empirical claims are benchmark accuracy numbers computed by executing generated code against QHack 2023-2024 hidden test suites (Table II, Fig. 5). That ground truth is external to the paper: no parameter is fitted to those test outcomes and then renamed as a prediction, and no claimed result is defined in terms of an output it is supposed to predict. The RAG condition uses the PennyLang dataset [22], a corpus authored by the same group, but the correctness criterion is still the external QHack checker, so the RAG comparison does not reduce to the corpus construction. The paper itself flags a contamination threat in Section V: 'some models may have inadvertently memorized parts of the 2023 challenge set during pretraining,' and Section VII concedes that binary pass/fail misses semantic errors. Those are threats to external validity and to the RAG-versus-non-RAG contrast, not circularity in the derivation chain: memorization would inflate absolute scores and confound the comparison, but it does not make the evaluation equivalent to its inputs by construction. The 'first benchmark' novelty claim is a scope claim, not a derived result. Hence no specific circular step is present; score 2 reflects the minor self-citation of PennyLang and the acknowledged, uncontrolled contamination risk rather than any circular reduction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on assumptions about test validity, model stability, and no prior memorization. It introduces no physical or mathematical entities, and the hand-chosen hyperparameters are limited to retrieval top-k, the iteration cap, and difficulty bin thresholds. No parameter is fitted to data.

free parameters (4)
  • retrieval top-k = 3
    ChromaDB retrieval uses MMR with top-k=3 (Section IV-C2). This hand-chosen value changes the RAG prompt content and is never swept.
  • maximum refinement attempts = 5
    The multi-agent loop stops after five iterations (Section III-C). This cap affects pass@5 and the reported multi-agent success rates.
  • difficulty bin thresholds = Easy 0-100, Intermediate 200-300, Advanced 400-600
    Challenge difficulty labels in Table I are derived from point thresholds chosen by the authors; these bins drive the difficulty analysis in Figure 4.
  • embedding model for retrieval = unspecified
    Algorithm 2 uses an embedding model E, but the paper never names it, making the RAG configuration underspecified and not reproducible.
assumptions (4)
  • domain assumption QHack official hidden test suites are a valid and sufficient ground truth for code correctness.
    The benchmark computes accuracy by executing generated code against predefined test cases (Section III-A), with no validation that hidden tests capture all required behavior.
  • ad hoc to paper LLM outputs that fail to execute are genuine model failures and not artifacts of prompt parsing or harness.
    Section VI-A attributes the dominant 'no output' failure mode to insufficient quantum understanding, but alternative explanations such as output formatting, parsing, or API truncation are not tested.
  • domain assumption QHack challenge data were not memorized by the evaluated models.
    The 2024 performance drop is attributed to possible data contamination in Section V; if contamination is present, the accuracy numbers are inflated.
  • domain assumption Public model APIs used via OpenAI, Anthropic, and DeepSeek are stable enough for benchmarking.
    No repeated runs or seeds are reported, and stochastic API outputs are treated as deterministic scores (Section IV-C4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges." pith.science (2026). https://pith.science/paper/UNP6HTOB

@misc{pith2026250620008,
  author       = {Pith},
  title        = {Pith review of: QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UNP6HTOB}},
  note         = {Machine review of arXiv:2506.20008}
}
read the original abstract

Recent advances in Large Language Models (LLMs) have demonstrated strong potential in code generation, yet their effectiveness in quantum computing remains underexplored. This paper benchmarks LLMs for PennyLane-based quantum code generation using real-world challenges from the Quantum Hackathon (QHack). We introduce QHackBench, a novel benchmark dataset derived from QHack competitions, and evaluate model performance under vanilla prompting and Retrieval-Augmented Generation (RAG). Our structured evaluation framework assesses functional correctness, syntactic validity, and execution success across varying challenge difficulties. Results indicate that RAG-enhanced models, supplemented with an augmented PennyLane dataset, approximately generate similar results as the standard prompting, particularly in complex quantum algorithms. Additionally, we introduce a multi-agent evaluation pipeline that iteratively refines incorrect solutions, further enhancing execution success rates. To foster further research, we commit to publicly releasing QHackBench, along with our evaluation framework and experimental results, enabling continued advancements in AI-assisted quantum programming.

Figures

Figures reproduced from arXiv: 2506.20008 by the authors.

Figure 1
Figure 1. An overview of our contributions. In this study, we report only binary outcomes—each challenge is either solved correctly or not. Our experiments show that when the model’s queries were supplemented with relevant PennyLane documentation via RAG, the number of challenges solved were approximately the same as the baseline approach. Furthermore, integrating a multi-agent refinement pipeline further improved the executi… view at source ↗
Figure 2
Figure 2. Composition analysis of the PennyLang dataset (3,347 samples). Our experimental results demonstrate that integrating a multi-agent framework significantly improves execution success rates compared to standalone prompting or RAG-based approaches. By systematically evaluating LLM performance in a multi-agent setting, we provide the first in-depth analysis of iterative refinement in quantum code generation. III. QHackB… view at source ↗
Figure 3
Figure 3. Overview of the QHackBench. ⃝1 Base query. The QHack challenge description and template are provided to LLMs (GPT-4.1, Claude 3.7, ...) for initial code generation. ⃝2 Retrieval-augmented query If enabled, relevant PennyLang code snippets are retrieved, embedded, and incorporated into the query to enhance accuracy. ⃝2 Multi-agent pipeline: a validator agent, implemented with the same LLM, inspects run-time logs and … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Difficulty distribution among all the challenges. [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: Per–challenge performance heat-map for all models on the complete QHack 2023 & 2024 challenge suites. Each row corresponds to a model–prompting method [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Top Panel: Matrix depicting the correctness of results given by each of the [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 18 canonical work pages

  1. [26]

    Qhack 2023 coding challenges

    Xanadu AI, “Qhack 2023 coding challenges.” [Online]. Available: https://github.com/XanaduAI/qhack 2023 coding challenges

    work page 2023

  2. [27]

    Qhack 2024 coding challenges

    Xanadu AI, “Qhack 2024 coding challenges.” [Online]. Available: https://github.com/XanaduAI/QHack2024-coding-challenges 7

    work page 2024

  3. [1]

    Quantum supremacy using a programmable superconducting processor,

    F. Arute et al., “Quantum supremacy using a programmable superconducting processor,” Nature, 2019

    work page 2019

  4. [2]

    Computational advantage in hybrid quantum neural networks: Myth or reality?

    M. Kashif, A. Marchisio, and M. Shafique, “Computational advantage in hybrid quantum neural networks: Myth or reality?” in DAC, 2025

    work page 2025

  5. [3]

    Demonstrating quantum advantage in hybrid quantum neural networks for model capacity,

    M. Kashif and S. Al-Kuwari, “Demonstrating quantum advantage in hybrid quantum neural networks for model capacity,” inICRC, 2022, pp. 36–44

    work page 2022

  6. [4]

    PennyLane: Automatic differentiation of hybrid quantum-classical computations,

    V. Bergholm et al. , “PennyLane: Automatic differentiation of hybrid quantum-classical computations,” arXiv:1811.04968, 2018

    arXiv 2018

  7. [5]

    Quantum machine learning,

    J. Biamonte et al., “Quantum machine learning,” Nature, 2017

    work page 2017

  8. [6]

    A survey on quantum machine learning: Current trends, challenges, opportunities, and the road ahead,

    K. Zaman, A. Marchisio, M. A. Hanif, and M. Shafique, “A survey on quantum machine learning: Current trends, challenges, opportunities, and the road ahead,” arXiv preprint arXiv:2310.10315, 2023

    arXiv 2023

Show all 27 references
  1. [7]

    Optimizing low-energy carbon IIOT systems with quantum algorithms: Performance evaluation and noise robustness,

    K. Dave et al., “Optimizing low-energy carbon IIOT systems with quantum algorithms: Performance evaluation and noise robustness,”IEEE IoT, 2025

    2025

  2. [8]

    Qnn-vrcs: A quantum neural network for vehicle road cooperation systems,

    N. Innan, B. K. Behera, S. Al-Kuwari, and A. Farouk, “Qnn-vrcs: A quantum neural network for vehicle road cooperation systems,” IEEE Transactions on Intelligent Transportation Systems, 2025

    2025

  3. [9]

    Next-generation quantum neural networks: Enhancing efficiency, security, and privacy,

    N. Innan et al. , “Next-generation quantum neural networks: Enhancing efficiency, security, and privacy,” inIOLTS, 2025

    2025

  4. [10]

    Survey of different large language model architectures: Trends, benchmarks, and challenges,

    M. Shao et al. , “Survey of different large language model architectures: Trends, benchmarks, and challenges,”IEEE Access, 2024

    2024

  5. [11]

    Qiskit Code Assistant: Training LLMs for generating Quantum Computing Code,

    N. Dupuis et al. , “Qiskit Code Assistant: Training LLMs for generating Quantum Computing Code,” arXiv preprint arXiv:2405.19495, 2024

    2024 arXiv

  6. [12]

    Qiskit HumanEval: An Evaluation Benchmark for Quantum Code Generative Models,

    S. Vishwakarma et al., “Qiskit HumanEval: An Evaluation Benchmark for Quantum Code Generative Models,” arXiv:2406.14712, 2024

    2024 arXiv

  7. [13]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Junet al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  8. [14]

    Starcoder: May the source be with you!

    R. Li, A. Q. Jiang, C. Qian et al., “Starcoder: May the source be with you!” arXiv preprint arXiv:2305.06161, 2023

    2023 arXiv

  9. [15]

    Program synthesis with large language models,

    J. Austin et al. , “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, 2021

    2021 arXiv

  10. [16]

    QHack 2022 - the one-of-a-kind celebration of quantum computing,

    L. Watson, “QHack 2022 - the one-of-a-kind celebration of quantum computing,” PennyLane Blog, uRL: https://pennylane.ai/blog/2022/03/ qhack-2022-the-one-of-a-kind-celebration-of-quantum-computing

    2022

  11. [17]

    Introducing qiskit code assistant,

    D. G. Almeida, J. Cruz-Benito, and R. Davis, “Introducing qiskit code assistant,” IBM Quantum Computing Blog (Oct 9, 2024), uRL: https://ww w.ibm.com/quantum/blog/qiskit-code-assistant

    2024

  12. [18]

    Pennycoder: Efficient domain-specific llms for pennylane- based quantum code generation,

    A. Basit et al., “Pennycoder: Efficient domain-specific llms for pennylane- based quantum code generation,” arXiv preprint arXiv:2507.19562, 2025

    2025

  13. [19]

    Cirq: A python framework for creating, editing, and invoking noisy intermediate scale quantum (nisq) circuits,

    C. Gidney and M. Newman, “Cirq: A python framework for creating, editing, and invoking noisy intermediate scale quantum (nisq) circuits,” QST, 2021

    2021

  14. [20]

    Accelerate quantum software development on amazon braket with claude-3,

    Y. Kharkov et al. , “Accelerate quantum software development on amazon braket with claude-3,” AWS Quantum Technologies Blog (18 Sep 2024), uRL: https://aws.amazon.com/blogs/quantum-computing/accelerate-quant um-software-development-on-amazon-braket-with-claude-3/

    2024

  15. [21]

    Retrieval-augmented generation for knowledge-intensive nlp tasks,

    P. Lewis et al. , “Retrieval-augmented generation for knowledge-intensive nlp tasks,” arXiv preprint arXiv:2005.11401, 2020

    2005 arXiv

  16. [22]

    Pennylang: Pioneering llm-based quantum code generation with a novel pennylane-centric dataset,

    A. Basit et al., “Pennylang: Pioneering llm-based quantum code generation with a novel pennylane-centric dataset,”arXiv:2503.02497, 2025

    2025 arXiv

  17. [23]

    Wooldridge, An Introduction to MultiAgent Systems

    M. Wooldridge, An Introduction to MultiAgent Systems . John Wiley & Sons, 2009

    2009

  18. [24]

    Rgd: Multi-llm based agent debugger via refinement and generation guidance,

    H. Jin, Z. Sun, and H. Chen, “Rgd: Multi-llm based agent debugger via refinement and generation guidance,” 2024

    2024

  19. [25]

    Coast: Enhancing the code debugging ability of llms through communicative agent based data synthesis,

    W. Yang et al., “Coast: Enhancing the code debugging ability of llms through communicative agent based data synthesis,” in Findings of the Association for Computational Linguistics: NAACL 2025 , 2025

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.