{"id":"8725c47a-1e40-4f2c-bc4c-2c0207ff0037","arxiv_id":"2508.20134","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"QAgent, a multi-agent LLM system, increases OpenQASM generation pass rates by up to 71.6% over static few-shot baselines, but the evaluation has potential data overlap and missing error bars.","lead":"This paper presents QAgent, a multi-agent AI system that writes OpenQASM quantum circuit code from plain-language descriptions. The system combines example retrieval, self-correction, and tool use to improve code correctness, which could make quantum programming easier for non-experts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RAG database and test benchmark are both derived from QCircuitNet with no described split; if retrieved examples overlap test solutions, the 71.6% improvement measures recall, not generation.","rationale":"The paper presents a plausible multi-agent system and provides useful qualitative error analyses, but the quantitative claim depends entirely on the evaluation protocol. The most load-bearing assumption is that the RAG database does not contain test-adjacent examples. The reader identified the same concern; my stress-test confirms it is concrete and testable. The paper gives no evidence of a split, and internal evidence (Section 5.1 and Section 6) suggests the database and benchmark share the same origin. This does not by itself prove fraud or even failure, but it makes the central number unreliable until checked. The right response is to keep the verdict conditional on a retrieval-overlap audit and code release. I do not see a different concern that is more central: statistical issues (no error bars) and limited task scope are secondary because even perfect statistics would not fix a systematic overlap. I recommend UNCHANGED because the reader's conditional verdict already captures the appropriate stance.","tokens_in":11727,"tokens_out":5124,"duration_ms":54261,"concrete_test":"Inspect the released repository (github.com/fuzhenxiao/QCoder) and compare the RAG example database against the Level-1 QCircuitNet benchmark solutions. For each test problem and qubit count, retrieve the top-1 example and compute normalized edit distance to the ground-truth QASM. If any retrieved example is identical or within ~5% edit distance, leakage is confirmed and the 71.6% claim is invalid. A complementary behavioral test would be running QAgent on a held-out Phase Estimation instance with an oracle/phase absent from the database and checking whether pass@1 collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('71.6% enhancement' and 'fully automate OpenQASM programming') rests on an evaluation where the RAG retrieval database is never separated from the test benchmark. Section 2.2 says the Prompt Agent 'retrieves a set of similar example QASM code from a quantum algorithm database'; Section 5.1 evaluates on QCircuitNet; Section 6 says the released dataset is 'modified from [24]'. The paper never describes a train/test split, deduplication, or exclusion of test-adjacent examples from the RAG store. Because QCircuitNet provides per-algorithm Python generators and test cases, the retrieved 'similar examples' can be exact or near-exact copies of the target program (e.g., the Phase Estimation program in §7.3 with CU_0 repetitions 1,2,4,8, and W-state angle constants). If so, the Dynamic Coder performs retrieval-augmented copying rather than generation, so the reported pass@3 and the 71.6% gain over Static do not support autonomous OpenQASM programming. The ablation in §3.2.4 removes components but never removes the RAG database itself, and averaging over five repetitions cannot correct systematic overlap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QAgent, a multi-agent LLM system for generating OpenQASM programs from natural-language descriptions. The system combines a Dynamic-few-shot Coder that retrieves similar QASM examples via RAG and imitates them, a Tools-augmented Coder that plans and calls predefined quantum-programming functions, and a hybrid fallback that escalates from the former to the latter on failure. The authors evaluate syntax and functional pass@k on a set of single-algorithm benchmarks (Bernstein-Vazirani, Deutsch-Jozsa, Grover, Phase Estimation, W-state) and on composite multi-algorithm tasks, across three Qwen model sizes. They report that QAgent improves QASM code-generation accuracy by 71.6% over a static few-shot LLM baseline, and they present ablations of reflection rounds, candidate number, model size, and prompt components.","tokens_in":12051,"tokens_out":4546,"duration_ms":52091,"significance":"If the central claims are substantiated, QAgent would be a useful engineering contribution: it demonstrates a plausible architecture for combining RAG, few-shot prompting, reflection, and tool use for quantum code generation, with a pluggable base LLM and a hybrid dynamic/tools routing strategy. The ablation of reflection rounds and candidate number, and the explicit separation of syntax vs. functional correctness, are useful empirical details. The planned public release of code and data is also a positive step. However, the paper's main quantitative claim — a 71.6% improvement and 'fully automated' OpenQASM programming — is not currently established because the evaluation pipeline has a major potential leakage path: the RAG retrieval database and the test benchmark are both derived from QCircuitNet with no described split. If retrieved examples are identical or near-identical to test programs, the reported pass rates reflect retrieval-copying rather than generation. The lack of error bars, significance tests, and a validated static baseline further weaken the quantitative conclusions. These issues are addressable with additional experiments and analysis, but they are load-bearing","major_comments":[{"comment":"Potential RAG/test leakage undermines the central claim. The Prompt Agent retrieves 'a set of similar example QASM code from a quantum algorithm database' (§2.2); the evaluation uses the QCircuitNet dataset (§5.1); and the released dataset is 'modified from [24]' (§6). The paper never defines a split between the RAG store and the test benchmark, and never excludes test-adjacent examples from retrieval. Because QCircuitNet provides Python-based QASM generators and test cases for each algorithm, retrieved examples for bv, dj, gr, pe, and ws can be exact or near-exact copies of the target program (e.g., the pe and ws examples in §7.3 have fixed constants). Under these conditions, pass@3 and the 71.6% gain measure retrieval-copy success, not autonomous OpenQASM generation. The authors must specify a split, deduplicate or remove test instances from the RAG store, and report results with the R","section":"§2.2, §5.1, §6"},{"comment":"The headline '71.6%' improvement is not backed by a clear computation. No numerical table reports per-model, per-algorithm, or per-level pass rates; Figures 2 and 4 are bar charts without error bars, confidence intervals, or significance tests. The manuscript does not state whether 71.6% is the average over algorithms, over model sizes, or the best-case improvement, nor whether it refers to syntax or functional pass@k. The authors should provide a table with means and variances, state the aggregation formula, and add significance tests or confidence intervals. Five repetitions averaged together cannot support a precise percentage without variance reporting.","section":"§3 and §5.4"},{"comment":"The Static baseline is asserted to 'reproduce the setup from prior work [24]', but the paper gives no validation that its static few-shot results match the original QCircuitNet static results. If the static prompts, example selection, or sampling parameters differ from [24], the comparison is not controlled and the claimed improvement over 'previous static LLM-based approaches' may be inflated. The authors should run the exact static configuration from [24] as a control and report the comparison, or otherwise justify that the baseline is equivalent to prior work.","section":"§3.1 and §5.4"},{"comment":"The ablation study removes the few-shot prompt, the analysis component, and reflection, but never removes the RAG database. The '0-shot' condition presumably removes the few-shot examples, yet it is unclear whether retrieval is still active and whether the Prompt Agent still injects retrieved code into the prompt. Given that RAG is a central mechanism, the contribution of the retrieval memory itself must be ablated. Without this, the paper cannot distinguish between the value of retrieved examples and the value of the generation/reflection loop.","section":"§3.2.4"},{"comment":"The inference-budget specifications are internally inconsistent. Section 2.2 states the Dynamic Coder by default performs 'one candidate generation and up to three reflection rounds'; Section 2.4 says the default budget is '3 candidate, 5 reflection rounds'; Section 5.4 says the default budget is '3 candidate and 5 reflection rounds' for both Dynamic and Tools. Section 3.2.2 then 'fixes the default candidate number to 1'. This inconsistency affects reproducibility and the interpretation of pass@k: pass@3 results reported elsewhere depend on which budget was actually used. Please reconcile these numbers and report the exact default configuration used for each experiment.","section":"§2.2, §2.4, §5.4"}],"minor_comments":[{"comment":"Typos and small editorial issues: §5.1 'foundamental', §7.3 'W rongIQFT' and 'inaccuate', and inconsistent hyphenation 'Dynamic-few-shot' vs 'Dynamic'. The paper would also benefit from a careful pass for keyword capitalization ('T ools', 'V alue').","section":"General"},{"comment":"The bar charts would be much easier to interpret with numerical labels and error bars. The caption of Figure 4(a) says 'p-best' but it is not defined in the caption; the reader has to infer from the text.","section":"Figures 2–4"},{"comment":"In the pass@k formula, n_t is described as 'total number of generated samples', but it is not stated whether n_t is per task, per prompt, or aggregated over the five repetitions. Please clarify the sampling procedure used to compute the unbiased estimator.","section":"§5.4, Eq. (1)"},{"comment":"Reference [3] (Renner and Wolf, 'Quantum advantage in cryptography') is listed as appearing in AIAA Journal 61(5), 1895–1910, which appears to be an incorrect venue. Please verify.","section":"Reference [3]"}],"recommendation":"major_revision","confidential_remarks":"The leakage concern is serious enough that I would recommend the editor ask for a specific description of the RAG/test split and for re-run results with a deduplicated retrieval store. If the authors cannot provide such a split because the RAG database was built directly from QCircuitNet without a train/test partition, then the rejection threshold would be crossed, because the central 'fully automated' claim would be unverifiable. The paper is otherwise within the scope of the journal and contains useful ideas. The code availability statement promises public release after publication; at review time, no code is accessible, so verification is limited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is one of the first to target OpenQASM specifically with a multi-agent system, which is a genuinely new application. The hybrid Dynamic/Tools fallback is sensible, and the ablations are thoughtful—they show which components matter, and the error examples in the supplement are informative. The system is clearly described, and the code is promised for release.\n\nThe main issue is evaluation integrity. The RAG database is built from the same QCircuitNet dataset used for testing, and the paper never describes a train/test split, deduplication, or exclusion of test-adjacent examples. Since QCircuitNet includes per-algorithm generators and test cases, retrieved examples could be exact or near-exact copies of the target program—look at the Phase Estimation schedule with CU_0 repetitions 1,2,4,8 and the W-state angle constants in §7.3. If so, the Dynamic Coder is doing retrieval-augmented copying, not generation, and the 71.6% improvement over Static measures recall, not generalization. The ablation removes components but never removes the RAG database itself, so it cannot diagnose this. There are also smaller problems: no error bars or significance tests, hyperparameters tuned without sensitivity analysis, and the static baseline is not validated against the cited QCircuitNet numbers.\n\nThat said, the approach is not wrong. The architecture is plausible, and the system might genuinely help non-experts prototype circuits. But the central performance claim is not supportable on the evidence presented. The paper deserves a serious referee, but it needs a rigorous evaluation protocol with a proper RAG/test split, error bars, and a comparison against at least one competing agent. For readers working on LLM-based quantum code generation, this is a useful case study in how RAG can inflate results. For the broader community, the leakage gap is a cautionary tale. I would not cite it in its current form, but I would send it to peer review and let the authors address the concerns.","headline":"QAgent applies the standard LLM-agent stack to OpenQASM and reports big gains, but the evaluation has a load-bearing leakage gap: the RAG store and test set are both from QCircuitNet with no described split, so the 71.6% claim should not be taken at face value.","tokens_in":12475,"tokens_out":2694,"would_cite":false,"duration_ms":26777,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a multi-agent LLM system can take a natural-language description of a quantum problem and return a working OpenQASM program without human intervention; across three model sizes it reports a 71.6% accuracy gain over st","keywords":["OpenQASM","LLM agents","retrieval-augmented generation","few-shot learning","chain-of-thought reflection","quantum circuit generation","NISQ","multi-agent system"],"falsifier":"Construct a held-out set of OpenQASM tasks whose algorithms are absent from the example database, run QAgent with retrieval forced off and on, and inspect the retrieved examples for identity or near-identity with the target. If pass@k on genuinely unseen algorithms falls to the static-baseline level, or if near-duplicate examples are routinely retrieved for benchmark tasks, the central claim of autonomous generation fails.","tokens_in":11663,"feed_emoji":"🤖","tokens_out":8159,"duration_ms":84713,"temperature":0.7,"pith_summary":"This paper is trying to show that programming quantum computers in OpenQASM can be automated for non-experts by a multi-agent LLM system. Its central claim is that combining two coding strategies — retrieving and imitating known example circuits for short structured tasks, and composing predefined quantum tools via a planning agent for complex parameterized ones — with iterative self-reflection raises the rate of correct generated code by 71.6% relative to prior static few-shot LLM prompting, across several LLM sizes. If true, the practical consequence is that a researcher without quantum expertise can describe a quantum problem in natural language and obtain a compilable, functionally verified circuit, lowering the entry barrier to NISQ computing.","feed_headline":"LLM agent system boosts correct OpenQASM code by 71.6%","feed_subtitle":"Natural-language descriptions become working quantum programs via retrieved examples plus tool planning, beating static prompting.","key_machinery":"The central mechanism is the two-coder hybrid. A Dynamic-few-shot Coder retrieves similar example QASM programs via retrieval-augmented generation and writes code by imitation; a Tools-augmented Coder plans a sequence of composable predefined tools (gate application, oracle generation, formatting) and writes code by execution. Each pipeline runs generation, automated syntax/functional testing, and chain-of-thought reflection, and a fallback route sends easy tasks to imitation, escalating failures to planning.","core_discovery":"QAgent's claim is that generation quality in OpenQASM is governed by matching the strategy to the task structure. For short, well-defined algorithms (Bernstein-Vazirani, Deutsch-Jozsa, Grover), a Dynamic-few-shot Coder retrieves similar examples and builds a composite prompt with analysis; for longer, parameter-sensitive algorithms (phase estimation, W-state), a Tools-augmented Coder plans a sequence of predefined gate and oracle operations. The two are combined in a fallback hierarchy, and the paper reports that the full system beats the static baseline by 71.6% on average and beats either component alone. Ablations show few-shot prompting, analysis, and chain-of-thought reflection all cont","pith_inferences":["Because the example database and the benchmark tasks come from the same dataset family, the strongest test of the 71.6% claim would be to hold out algorithms absent from the retrieval database; without that split, part of the gain could be retrieval of near-duplicates.","The same hybrid architecture could transfer to Qiskit or PennyLane code generation, where static LLM baselines also struggle with parameterized circuits.","The differing value of reflection suggests an adaptive budget policy: spend candidates on imitation-path tasks and reflection rounds on planning-path tasks, rather than fixed rounds.","A cost-per-correct-program measurement would clarify whether the accuracy gain justifies the inference cost, since the paper reports accuracy but not compute."],"forward_implications":["A non-expert can describe a quantum problem in natural language and receive a compilable, functionally verified OpenQASM program, lowering the barrier to using NISQ devices.","For templated algorithms like Deutsch-Jozsa and Grover, the limiting factor is the quality of the retrieved example set, not the base LLM; for parameterized algorithms like phase estimation and W-state, tool planning is the strategy that carries the gain.","Reflection rounds add little in the imitation path and much in the planning path, so the system's default budget could be reallocated by path.","On composite tasks that mention two to four algorithms, accuracy falls sharply as the task count grows, making robust decomposition of multi-algorithm requests the next capability needed."],"supporting_citations":[{"why":"Supplies the benchmark of fundamental quantum problems, its Python-based OpenQASM generators and test cases, and the static few-shot baseline that QAgent is compared against.","marker":"[24]"},{"why":"Defines OpenQASM 3, the target language whose syntax and semantics generated programs must satisfy.","marker":"[7]"},{"why":"Defines the pass@k estimator used to measure syntax and functional success rates.","marker":"[33]"}],"fun_headline_variants":["LLM agents adapt to quantum tasks, boost code accuracy by 71.6%","Multi-agent LLM system writes OpenQASM with 71.6% higher accuracy","Task-matched LLM agents boost quantum code accuracy by 71.6%","LLM multi-agent system automates OpenQASM, up 71.6% in accuracy","Adaptive LLM agents for quantum programming: 71.6% better code"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the RAG database does not contain near-duplicates of the benchmark problems; if retrieved examples are effectively the answer keys, then the reported gains measure retrieval rather than code generation.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents adapt to quantum tasks, boost code accuracy by 71.6%","Multi-agent LLM system writes OpenQASM with 71.6% higher accuracy","Task-matched LLM agents boost quantum code accuracy by 71.6%","LLM multi-agent system automates OpenQASM, up 71.6% in accuracy","Adaptive LLM agents for quantum programming: 71.6% better code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000567,"raw_usage":{"total_tokens":2532,"prompt_tokens":760,"completion_tokens":1772,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":1660}},"tokens_in":504,"tokens_out":1772,"duration_ms":12987,"temperature":1.0,"reasoning_tokens":1660,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:46:59.140945+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a held-out set of OpenQASM tasks whose algorithms are absent from the example database, run QAgent with retrieval forced off and on, and inspect the retrieved examples for identity or near-identity with the target. If pass@k on genuinely unseen algorithms falls to the static-baseline level, or if near-duplicate examples are routinely retrieved for benchmark tasks, the central claim of autonomous generation fails.","supporting_citations":[{"cited_title":"ACM Transactions on Quantum Computing 3(3), 1–50 (2022)","cited_arxiv_id":null,"evidence_quote":"Defines OpenQASM 3, the target language whose syntax and semantics generated programs must satisfy."}],"review_version":1}