{"id":"39caee7d-5b00-42b7-9147-7835ae23dc1a","arxiv_id":"2508.20907","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A Qiskit code assistant trained with DPO and GRPO rewards (unit-test pass rate on simulators/hardware) achieves the highest reported score on Qiskit-HumanEval-hard.","lead":"IBM researchers trained a 14-billion-parameter coding model to write Qiskit quantum programs, using unit tests executed on quantum simulators and hardware as the training reward. Their best model beat much larger open-source models on a hard quantum-coding benchmark, suggesting small specialized models can excel in niche technical domains.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"QHE-hard margin of 1.33 pts over runner-up is within sampling noise; no variance or significance test reported for Table 2.","rationale":"The central claim is that DPO-GRPO attains the highest QHE-hard score among evaluated models. This relies on a single greedy-decoding run in Table 2. The 1.33-point margin over Intern-S1 on 151 problems is less than one-third of the standard error of the difference, so the result is within sampling noise. Without per-problem outcomes or repeated runs, the observed ranking cannot be distinguished from chance. This is more fundamental than the synthetic-data representativeness concern: even with perfect data, the headline number is not established. The paper is otherwise solid—execution-based reward is a sensible signal, the ablations are suggestive, and the authors are transparent about simulator usage—but the statistical grounding is missing. The fix is straightforward: release per-problem results or run multiple seeds and report confidence intervals. This aligns with the reader's CONDITIONAL verdict, but the condition should explicitly include statistical verification. I therefore keep the verdict unchanged while sharpening the rationale.","tokens_in":18101,"tokens_out":7374,"duration_ms":75960,"concrete_test":"Obtain per-problem pass/fail outcomes for all models in Table 2 on QHE-hard. Run McNemar's exact test on the paired 2x2 table comparing DPO-GRPO vs Intern-S1, and DPO-GRPO vs Qwen3-Coder-480B. Report p-values and a 95% confidence interval for the difference in pass rates. Additionally, rerun greedy decoding with at least 5 different random seeds (or temperature-0 runs with different sampling orders) and report the spread. If the confidence interval includes zero, the claim of 'highest score' is statistically unsupported and should be softened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4/Table 2 compares greedy-decoding pass@1 on QHE-hard with n=151 problems. DPO-GRPO scores 28.48%, Intern-S1 27.15%, Qwen3-Coder-480B 25.82%. The standard error of a proportion at p=0.28, n=151 is roughly 3.7 percentage points; the SE of the difference between two independent proportions is about 5.2 points. The observed 1.33-point margin is well under 0.3 SE, so the claim that DPO-GRPO surpasses these baselines is statistically indistinguishable from a tie. No error bars, confidence intervals, or significance tests are provided for these numbers. The central conclusion—that a 14B model with quantum-verified rewards beats models up to 30x larger—rests on a difference that could easily arise from a single evaluation seed. This is the most load-bearing issue because even if the synthetic-data distribution is perfectly representative, the headline number is not shown to be real.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a post-training pipeline for a Qiskit code assistant built on Qwen2.5-Coder-14B. The pipeline consists of extended pretraining on Qiskit code, weight merging with an instruct model, SFT with distilled quantum-physics explanations, and then DPO and/or GRPO using a 'quantum verifiable reward' defined as the unit-test pass rate on quantum simulators or QPUs. A synthetic prompt/unit-test generation pipeline supports both offline preference data and online GRPO rollouts. The authors evaluate on Qiskit-HumanEval (QHE) and Qiskit-HumanEval-hard (QHE-hard). The central reported result is that Qwen2.5-Coder-14B-Qiskit-Instruct-DPO-GRPO reaches 28.48% pass@1 on QHE-hard with greedy decoding, which the paper claims surpasses all evaluated open-source baselines, including models up to 30x larger.","tokens_in":18407,"tokens_out":5663,"duration_ms":64167,"significance":"If the headline result is statistically reliable, the paper makes a useful and timely contribution: it demonstrates a concrete recipe for using execution-based verifiable rewards in a specialized quantum-programming domain, provides a synthetic data pipeline, and ablates SFT, DPO, and GRPO. The experimental design is well structured, and the use of Qiskit-HumanEval as a benchmark is appropriate. The paper is not circular in the logical sense: using unit-test pass rate both as a training reward and as an evaluation metric is standard practice in code-generation RL. The main weaknesses are that the headline margin on QHE-hard is small relative to the finite-benchmark sampling error, and that transfer from the synthetic GRPO training distribution to the human-curated benchmark is not analyzed.","major_comments":[{"comment":"The central claim that DPO-GRPO 'attains the highest score' on QHE-hard rests on a 1.33 percentage point margin over Intern-S1 (28.48% vs 27.15%) with n=151 problems and greedy decoding. The standard error of each proportion is about 3.7 points, and the standard error of the difference is about 5.2 points; the observed margin is therefore well under 0.3 SE. No confidence intervals, bootstrap, or significance tests are reported for Table 2, so the headline comparison is statistically indistinguishable from a tie. Please report uncertainty for all Table 2 entries, e.g., Wilson intervals and a McNemar or bootstrap test for the DPO-GRPO vs Intern-S1 comparison, and ideally repeat greedy evaluation with different seeds or report pass@k with error bars to support the 'surpasses' claim.","section":"Section 4, Table 2"},{"comment":"The GRPO reward is optimized on 4.5k synthetic prompt/unit-test pairs generated by the same pipeline used to create training data, while QHE/QHE-hard are Qiskit-curated benchmarks covering similar API and task families. The paper does not quantify the overlap or distribution shift between the synthetic training prompts and the benchmark, nor does it evaluate on an independent benchmark such as QHackBench. Without an overlap/leakage analysis or an out-of-distribution evaluation, the reported QHE-hard gain could reflect specialization to the synthetic template distribution rather than a general improvement in executable Qiskit code generation. Please add a quantitative overlap analysis and at least one held-out benchmark.","section":"Sections 3.3 and 4"}],"minor_comments":[{"comment":"The abstract and introduction emphasize rewards 'provided by the quantum hardware,' but Section 4 states that the QHE/QHE-hard evaluation uses only simulators, and Section 2.2 says about 10% of training problems require runtime access. Please qualify the wording to 'quantum hardware or simulator' throughout, or report the exact fraction of examples actually run on QPUs.","section":"Abstract/Introduction, Section 2.2"},{"comment":"The text says the figure plots only mean values and points to Appendix C for variance. Please show uncertainty in the main figure as well, since several qualitative claims ('GRPO models outperform both instruct and DPO models') are made from mean curves without statistical tests.","section":"Section 4, Figure 5"},{"comment":"The DPO negative-selection heuristic (choosing the rejected sample with maximum embedding cosine similarity to the accepted one) is interesting but no analysis is provided of how sensitive final results are to this choice or to the embedding model. A short sensitivity study or at least a motivation with examples would help.","section":"Section 3.3"},{"comment":"Typo: 'a wild range of tasks' should be 'a wide range of tasks.'","section":"Section 4, first paragraph"},{"comment":"The qualitative note that 'the SFT model provides a more detailed explanation' is subjective and not tied to any metric. If retained, it should be framed as an observation, not a result.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The main barrier to acceptance is statistical: the headline 1.33-point margin on QHE-hard is within the sampling error of a 151-problem benchmark, so the central 'surpasses baselines up to 30x larger' claim is not currently supported. The second concern, about synthetic-to-benchmark distribution shift, is also worth taking seriously because the benchmark and the training-data pipeline originate from the same organization. These issues are fixable with additional analysis and independent evaluation; I would be willing to look at a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The stress-test note gets the main thing right. The 1.33-point margin on QHE-hard (28.48 vs 27.15) with n=151 problems is within sampling noise; a two-proportion SE is around 5 points, so the claim that the 14B model beats Intern-S1 or Qwen3-Coder-480B is not supported by Table 2 alone. No error bars or significance tests are given for that table, and Figure 5's variance trace doesn't fix it because that's pass@k, not greedy pass@1. The paper's headline statement overshoots the evidence.\n\nThat said, there is substance here. The contribution is a concrete recipe: synthetic prompt/unit-test pairs, offline DPO with bucket A/B, online GRPO with a unit-test pass-rate reward over simulators (plus a small real-runtime fraction), on Qwen2.5-Coder-14B. The ablation is well-structured and the training details are all present—LoRA ranks, betas, group sizes, compute. The authors state their own limitation (easy-to-intermediate problems) and don't oversell the SFT reasoning style. The appendix examples, including a known ZNE failure, are useful for reproduction.\n\nSoft spots are real but mostly standard. Reward/eval alignment is not a logical circle; unit-test pass rate is exactly the execution-based reward. The bigger question is distribution shift: 4.5k synthetic template prompts versus 151 human-written QHE-hard problems. The paper offers no evidence about transfer, so I'd read the gain as suggestive, not established. The 'quantum hardware' phrasing also is doing more work than the experiments: evaluation is simulator-only and only ~10% of training problems touch the runtime. That should be fixed by the authors.\n\nThe benchmark being co-created by the same group is a mild self-reference, but not circular; QHE and QHE-hard are public and this is clearly the target benchmark. The bigger missing item is artifacts: no training data, preference sets, model weights, or eval code. For a model paper, that's the difference between a repeatable result and a one-off experiment.\n\nNet: it deserves a serious referee. The statistical gap should motivate CI/bootstraps/multiple seeds and artifacts, not desk rejection. I'd bring it to a reading group as a case study in execution-based RL for specialized code LLMs, with significance testing as the discussion point. I'd cite it for the pipeline if I needed a quantum-code-assistant baseline, but I wouldn't rely on its head-to-head numbers.","headline":"Solid engineering recipe for execution-based RL on Qiskit code; the QHE-hard win is within sampling noise, but the pipeline and ablations merit referee time.","tokens_in":18886,"tokens_out":3448,"would_cite":true,"duration_ms":32639,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Post-training a 14B Qiskit code model with rewards from executing unit tests on quantum systems lets it beat open-source models up to 30 times larger on Qiskit-HumanEval-hard.","keywords":["Qiskit","quantum code generation","post-training LLMs","DPO","GRPO","verifiable rewards","Qiskit-HumanEval-hard","quantum hardware execution"],"falsifier":"Take the DPO-GRPO model and evaluate it on a held-out set of human-written Qiskit problems whose unit tests are not generated by the paper's template pipeline; if its pass-rate advantage over Qwen2.5-Coder-14B-Instruct shrinks toward zero or reverses, the claim that quantum-verifiable rewards generalize beyond synthetic prompts is refuted.","tokens_in":18048,"feed_emoji":"⚛️","tokens_out":4647,"duration_ms":45474,"temperature":0.7,"pith_summary":"This paper argues that large language models writing Qiskit quantum code improve most when their training signal comes from actually executing the generated code on quantum simulators or hardware. The authors build a synthetic pipeline that creates quantum problems paired with unit tests, then use the pass rate of those tests as a reward in both offline preference optimization (DPO) and online reinforcement learning (GRPO). Their best 14B-parameter model, combining DPO and GRPO, reaches the highest score among all evaluated models on the harder Qiskit-HumanEval benchmark, surpassing far larger open-source models. The significance is that hardware-grounded, execution-based rewards may be a practical way to specialize language models for domains where correctness is only knowable by running code.","feed_headline":"14B Qiskit model beats 30x bigger rivals on hard benchmark","feed_subtitle":"Quantum-verified rewards plus DPO and GRPO push Qiskit-HumanEval-hard score to 28.48%, top among all tested models.","key_machinery":"The load-bearing mechanism is the quantum verifiable reward: Definition 1 in the paper, defined as the percentage of unit tests that pass when generated code runs on quantum hardware or a quantum simulator. Around it sits the synthetic data pipeline that manufactures prompt/unit-test pairs from templates covering Qiskit features such as estimator and sampler primitives, transpilation, and error correction. The pipeline feeds two training paths: DPO uses accepted/rejected code pairs selected so rejected samples are cosinesimilar to accepted ones, while GRPO uses grouped rollouts whose normalized advantage is computed from the quantum verifiable reward plus a small formatting reward.","core_discovery":"The central claim is that a quantum verifiable reward—the percentage of unit tests passed when generated Qiskit code is executed on a quantum processing unit or a realistic simulator—can serve as a direct post-training signal for LLMs. Using a synthetic data pipeline that generates prompt/unit-test pairs and validates code in a sandboxed Qiskit environment, the authors create accepted/rejected pairs for DPO and an online GRPO reward computed from unit-test pass rates. The best model, Qwen2.5-Coder-14B-Qiskit-Instruct-DPO-GRPO, achieves 28.48% on Qiskit-HumanEval-hard with greedy decoding, the highest among all evaluated models, including open-source baselines up to 30 times larger. The paper","pith_inferences":["If the 4.5k synthetic GRPO prompts are not representative of human-written Qiskit problems, the reported 28.48% gain may partly reflect template-specific overfitting; a held-out test on non-synthetic problems would separate genuine improvement from memorization.","The reward only measures unit-test pass rate, not circuit depth, gate count, or hardware efficiency, so extending the reward to include quality or cost metrics could push the model beyond mere executability and toward optimized quantum programs.","Because about 90% of the synthetic problems use simulators, the approach is scalable today; shifting more of the reward computation to real QPU execution could reveal whether simulator-verified rewards fully transfer to noisy hardware.","The same offline-plus-online recipe—synthetic prompt/test generation, DPO on hard negatives, GRPO on execution rewards—may transfer to other scientific coding domains where correctness is determined by running code, such as numerical simulation or instrument control."],"forward_implications":["A 14B-parameter specialized model can outperform open-source models orders of magnitude larger on a hard domain benchmark when post-trained with execution-based quantum rewards.","DPO and GRPO are complementary: DPO improves one-sample pass rates, while GRPO teaches the model to supply missing imports and produce executable code through self-play, which is essential on Qiskit-HumanEval-hard.","Quantum verification can be automated through a synthetic template pipeline, making hardware-in-the-loop training feasible without large human-annotated preference datasets.","The Qiskit variants retain competitive scores on standard HumanEval, suggesting domain specialization through quantum-verified post-training does not destroy general coding ability."],"supporting_citations":[{"why":"Supplies the DPO algorithm used for offline preference alignment from accepted/rejected quantum-verified code.","marker":"[Rafailov et al., 2023]"},{"why":"Supplies the GRPO algorithm that integrates the quantum verifiable reward through grouped Monte Carlo rollouts.","marker":"[Shao et al., 2024]"},{"why":"Defines the Qiskit-HumanEval benchmark and its evaluation methodology, the primary measure of the model's performance.","marker":"[Vishwakarma et al., 2024]"},{"why":"Provides the Qiskit SDK and AerSimulator environment used for executing and verifying generated code.","marker":"[Javadi-Abhari et al., 2024]"},{"why":"Supplies the base and instruct Qwen2.5-Coder-14B models that receive extended pretraining and post-training.","marker":"[Hui et al., 2024]"},{"why":"Defines HumanEval and the pass@k metric used for all benchmark evaluations.","marker":"[Chen et al., 2021]"},{"why":"The Qiskit-HumanEval dataset used to measure standard Qiskit code generation performance.","marker":"[Qiskit, 2024]"},{"why":"The Qiskit-HumanEval-hard dataset, the harder no-import benchmark where the best model attains 28.48%.","marker":"[Qiskit, 2025]"}],"fun_headline_variants":["Quantum-verified rewards make 14B Qiskit model beat 30x rivals","Quantum hardware verifies code: small model tops huge rivals","14B model with quantum-verified rewards beats 30x bigger LLMs","Quantum-verified rewards: secret sauce for Qiskit code LLMs","DPO+GRPO with quantum rewards: 14B model tops benchmark"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The 4.5k synthetic prompt/unit-test pairs used for GRPO are representative of the human-written Qiskit-HumanEval-hard benchmark, so maximizing unit-test pass rate on them produces genuine improvement rather than template-specific overfitting.","fun_headline_variants_meta":{"raw":{"variants":["Quantum-verified rewards make 14B Qiskit model beat 30x rivals","Quantum hardware verifies code: small model tops huge rivals","14B model with quantum-verified rewards beats 30x bigger LLMs","Quantum-verified rewards: secret sauce for Qiskit code LLMs","DPO+GRPO with quantum rewards: 14B model tops benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000449,"raw_usage":{"total_tokens":2071,"prompt_tokens":681,"completion_tokens":1390,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":425,"completion_tokens_details":{"reasoning_tokens":1292}},"tokens_in":425,"tokens_out":1390,"duration_ms":9888,"temperature":1.0,"reasoning_tokens":1292,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:42:09.305882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the DPO-GRPO model and evaluate it on a held-out set of human-written Qiskit problems whose unit tests are not generated by the paper's template pipeline; if its pass-rate advantage over Qwen2.5-Coder-14B-Instruct shrinks toward zero or reverses, the claim that quantum-verifiable rewards generalize beyond synthetic prompts is refuted.","supporting_citations":[],"review_version":1}