Pith. sign in

REVIEW 4 major objections 6 minor 30 references

Automatic Qiskit Code Refactoring Using Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Structured migration taxonomies from release notes materially improve an LLM's ability to identify and fix outdated Qiskit API usage.

desk verdict Useful incremental contribution with a self-aligned benchmark; the direction of the effect is plausible, but the headline numbers need corroboration before being trusted. read the letter →

arxiv 2506.14535 v1 pith:J6I7NGFE submitted 2025-06-17 cs.SE cs.AIcs.ET

José Manuel Suárez , Luis Mariano Bibbó , Joaquin Bogado , Alejandro Fernandez This is my paper
classification cs.SEcs.AIcs.ET
keywords QiskitcodemigrationlargelanguagemodelsrefactoringtaxonomyquantumsoftwareengineeringAPIevolutionprompt
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that giving a large language model a compact, structured taxonomy of Qiskit migration scenarios, extracted from official release notes, materially improves its ability to spot outdated API usage and propose workable refactors. In the reported 25-snippet test set, where 21 snippets contained migration issues and 81 lines needed changes, adding the taxonomy raised correct refactoring suggestions from 29 to 50 lines, lifting precision from 0.32 to 0.55 and recall from 0.35 to 0.62. The authors argue that a taxonomy is condensed enough to fit the model's context window without retrieval or chunking, letting the model spend its attention on the actual code. If correct, this offers a low-cost path to partially automated migration for fast-evolving quantum software frameworks.

What carries the argument

The load-bearing object is the version-specific migration taxonomy, a markdown table of 43 scenarios built from Qiskit's release notes and documentation, with fields for category, migration flow, summary, artifacts, example source and target code, difficulty, impact, and references. The prompt asks the model to return a markdown table mapping line numbers to scenario IDs, artifacts, and refactoring suggestions, and to mark unmatched findings with an asterisk. The taxonomy does the work by compressing an entire version's breaking changes into a form that fits the model's context window; the structured output does the work by making every suggestion attributable to a named scenario.

What would settle it

Take a corpus of real open-source Qiskit projects with known migration commits from versions 0.41 to 0.45 up to 0.46, run the same taxonomy prompt on the pre-migration files, and score line-level suggestions against the actual committed changes: the claim predicts precision and recall near or above the reported 0.55 and 0.62, with no larger false-positive rate on files that needed no migration.

Watch

Extended reading notes

Core claim

The paper's central claim is that exposing gpt-4-0613 to a compact, structured taxonomy of Qiskit migration scenarios, rather than relying only on its memorized knowledge, is what makes LLM-assisted migration work. On 25 synthetic snippets, 21 containing migration issues totaling 81 lines needing changes, the taxonomy raised correct refactoring suggestions from 29 to 50 lines, lifting precision from 0.32 to 0.55 and recall from 0.35 to 0.62, while cutting incorrect suggestions from 61 to 40 and missed lines from 52 to 31. Scenario identification also improved from 10 to 12 of 21 correctly identified refactor-needed scenarios. The authors interpret this as evidence that domain-specific migration knowledge, encoded as a taxonomy, materially improves both the identification and the resolution of version-drift problems in quantum code.

Load-bearing premise

The measured benefit of the taxonomy rests on 25 synthetic snippets, hand-written by the authors to embed scenarios from the same taxonomy placed in the prompt; if those snippets are not representative of real Qiskit code, the gain may not transfer.

Editorial extensions

If this is right

  • For a new Qiskit release, a taxonomy can be produced from release notes and attached to the prompt, giving the model version-specific guidance without fine-tuning or retrieval.
  • The structured table output lets developers audit each suggested change against a scenario ID, making LLM refactoring suggestions reviewable rather than opaque.
  • Taxonomy guidance cuts both false positives (from 61 to 40 incorrect suggestions) and false negatives (from 52 to 31 missed lines) in the reported test set, shifting migration work from hunting for breakages to verifying flagged lines.
  • The negative test cases show the method still over-flags: 3 of 4 compatible snippets received unnecessary migration suggestions, so human review remains necessary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • On a real-codebase corpus, the measured gain may shrink because real imports and usages do not map onto taxonomy examples as neatly as synthetic snippets crafted around the same taxonomy.
  • A strong test would generate a fresh taxonomy for Qiskit 2.0, a version released after the model's training data, and check whether the precision and recall gains persist when the model cannot rely on remembered documentation.
  • The same prompt-plus-taxonomy recipe could be applied to other Python SDKs that publish versioned release notes, treating release-note-derived taxonomies as a general migration aid.
  • The line-level scoring rubric could be automated, turning the manual expert review into a cheaper evaluation harness for comparing taxonomy quality and prompt variants.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a methodology for using large language models (LLMs) to refactor Qiskit code across versions, guided by a taxonomy of migration scenarios extracted from official Qiskit documentation and release notes. The authors evaluate their approach on 25 synthetic Python snippets, comparing the performance of gpt-4-0613 with and without the taxonomy included in the prompt. They report that the taxonomy improves scenario identification and line-level refactoring suggestions, with the key quantitative result being an increase in correct refactoring suggestions from 29/81 to 50/81 lines. The paper contributes the taxonomy, the prompts, and a publicly available repository.

Significance. If the result holds, the paper offers a practical and inexpensive way to assist Qiskit migration using LLMs, and the public release of prompts, taxonomy, and scripts supports reproducibility. The direction of the effect—access to structured migration knowledge improving LLM performance—is plausible and consistent with related work. However, the significance is currently tempered by three issues: the benchmark snippets are constructed from the same taxonomy that serves as the intervention, the manual scoring lacks inter-rater reliability, and Table II contains an internal inconsistency in the scenario-level confusion matrix. These issues are fixable but affect the strength of the central empirical claim.

major comments (4)
  1. [Section IV, first paragraph] The benchmark construction is coupled with the intervention. The paper states that the 25 snippets were 'constructed to expose the model to known patterns from the taxonomy,' and the taxonomy is exactly the context added in the treatment condition. This means the with-taxonomy model can succeed by retrieving or copying example refactorings from the taxonomy's example columns, which is supported by the Discussion's admission of 'reuse of placeholder variables copied verbatim from the taxonomy examples.' I would like to see either a holdout set of real-world Qiskit code or release-note-derived snippets that were not used when building the taxonomy, or an explicit evaluation of whether the model's correct answers depend on verbatim pattern matching.
  2. [Section V, Table II] Table II contains a numerical inconsistency in the scenario-identification confusion matrix. The text states that 21 of 25 snippets required refactoring and 4 were negative. With TP=12 and TN=1, the remaining 3 of the 4 negative snippets imply FP=3, not the reported FP=9; conversely, with 21 positive snippets and TP=12, FN should be 9, not 3. The reported recall values also do not follow: 12/(12+3)=0.80, not 0.85, and 10/(10+3)=0.77, not 0.76. Please correct the matrix and the corresponding precision/recall values, and clarify whether 'scenario' means snippets, individual migration issues, or something else.
  3. [Section IV, manual evaluation paragraph] The line-level scoring is the basis for the headline 50-vs-29 improvement, but the paper reports that the two experts evaluated independently 'without formal inter-rater reliability measures.' The rubric categories (OK, OK-, X+, X) require judgment, and there is no reported agreement statistic or resolution procedure. Please provide inter-rater agreement (e.g., Cohen's kappa) on a subset, or otherwise justify that the scoring is sufficiently reliable to support the claimed precision/recall gains.
  4. [Section VI, 'It is also worth mentioning...'] The abstract and introduction make claims about 'LLMs' in general, but the experiments use a single OpenAI model (gpt-4-0613). The Discussion correctly acknowledges this as a limitation, but the central claim would be strengthened substantially by testing at least one additional model, ideally a code-specialized one, even on a subset of the snippets. In addition, no confidence intervals or significance tests are reported for the 81-line and 21-scenario comparisons; given the small scenario-level counts, I would like to see a bootstrap or a matched-pair test to establish that the observed differences are not noise.
minor comments (6)
  1. [Abstract] The last sentence contains a typo: 'to asses the capabilities' should be 'to assess the capabilities.'
  2. [Author block] The ordinal suffixes in the author affiliations are incorrect: '2rd' and '3nd' should be '2nd' and '3rd.'
  3. [Figure 1 caption] The caption contains a grammatical error: 'The were a total of 50 invocations' should read 'There were a total of 50 invocations.'
  4. [Section II, Related Work] The model name 'Qween' in the paragraph on d'Aloisio et al. appears to be a typo for 'Qwen' or a similarly named model; please correct it.
  5. [Section III] The paper says the taxonomy was generated with 'ChatGPT 4.1,' while the evaluation uses 'gpt-4-0613'; please clarify whether the model that generated the taxonomy is the same as the model evaluated, since this affects the interpretation of the intervention.
  6. [Reference [20]] The GitHub repository URL contains a space ('qiskit experiments tlisc'), which will break the link; please provide the correct URL.

Circularity Check

1 steps flagged · score 6.0 of 10

The central evidence is a self-aligned benchmark: the test snippets were constructed from the same taxonomy used as the prompt, and the paper admits verbatim copying from taxonomy examples, so the reported gains partly reduce to pattern recall.

  1. self definitional [Section IV (Methodology), synthetic snippet design; Section VI (Discussion), admitted copy behavior]
    "A total of 25 Python code snippets were manually crafted to emulate realistic usage patterns while embedding specific migration scenarios relevant to Qiskit version 0.46. Each snippet consisted of between 9 to 30 lines of code and was designed to reflect real-world practices such as varied import styles, inline comments, and multi-line constructs. All snippets targeted deprecated features or modules, or were constructed to expose the model to known patterns from the taxonomy."

    The experimental target is defined by the intervention: the snippets and their expected refactorings are built from the same taxonomy that is placed in the with-taxonomy prompt. A correct answer can therefore be obtained by retrieving the taxonomy's example code rather than by performing migration reasoning; the paper later concedes 'the reuse of placeholder variables copied verbatim from the taxonomy examples.' The reported improvement from 29 to 50 correct lines thus partly reduces, by construction, to the model echoing the provided taxonomy, so it does not independently establish generalizable LLM migration assistance.

full rationale

The paper's core empirical claim is that providing the migration taxonomy improves LLM refactoring (50 vs 29 correct lines, Table II). The main circular step is the benchmark construction: Section IV states the 25 snippets 'were constructed to expose the model to known patterns from the taxonomy,' while the with-taxonomy prompt contains that same taxonomy with example code. The expected outputs (scenario IDs, artifacts, and refactorings) are the taxonomy's own entries, so the model can succeed by copying examples; the Discussion acknowledges verbatim reuse of placeholder variables from taxonomy examples. This makes the headline improvement partially an artifact of the test set being derived from the intervention. The comparison is not wholly circular: refactoring suggestions were manually checked against Qiskit API correctness, a without-taxonomy baseline was run, and the snippets include negative cases. However, the manual scoring used no formal inter-rater reliability, and Table II's scenario-level confusion matrix is internally inconsistent (TP 12 + FN 3 = 15, not the 21 refactoring scenarios stated in Section V; TN 1 + FP 9 = 10, not the 4 negative snippets), so the quantitative precision/recall gains are unvalidated upper bounds. The taxonomy's provenance is self-cited [19], but it is grounded in official Qiskit documentation and expert verification, so self-citation alone is not the load-bearing issue. Overall, the central demonstration is partially circular via benchmark construction, yielding a score of 6.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. It relies on assumptions about the correctness of the taxonomy, the representativeness of the synthetic snippets, the validity of expert scoring, and the model's prior knowledge. These assumptions are reasonable but unquantified.

assumptions (4)
  • domain assumption The 43-scenario taxonomy for Qiskit 0.46, generated with ChatGPT 4.1 and manually verified by the authors, is correct and complete.
    Used as the ground-truth prompt content and as the basis for the expected answers in Sections IV and V.
  • domain assumption The 25 hand-crafted snippets, embedding intended migration scenarios, represent realistic Qiskit code usage.
    Section IV states snippets were 'manually crafted to emulate realistic usage patterns'; no independent corpus is used.
  • domain assumption Expert color-coded manual scoring is a valid correctness measure.
    Section IV describes the rubric; no inter-rater reliability or automated checks are reported.
  • domain assumption GPT-4's prior training includes enough Qiskit documentation to make the no-taxonomy condition meaningful.
    Section IV acknowledges the model's training data versions are 'not transparent'.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatic Qiskit Code Refactoring Using Large Language Models." pith.science (2026). https://pith.science/paper/J6I7NGFE

@misc{pith2026250614535,
  author       = {Pith},
  title        = {Pith review of: Automatic Qiskit Code Refactoring Using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J6I7NGFE}},
  note         = {Machine review of arXiv:2506.14535}
}
read the original abstract

As quantum software frameworks evolve, developers face increasing challenges in maintaining compatibility with rapidly changing APIs. In this work, we present a novel methodology for refactoring Qiskit code using large language models (LLMs). We begin by extracting a taxonomy of migration scenarios from the different sources of official Qiskit documentation (such as release notes), capturing common patterns such as migration of functionality to different modules and deprecated usage. This taxonomy, along with the original Python source code, is provided as input to an LLM, which is then tasked with identifying instances of migration scenarios in the code and suggesting appropriate refactoring solutions. Our approach is designed to address the context length limitations of current LLMs by structuring the input and reasoning process in a targeted, efficient manner. The results demonstrate that LLMs, when guided by domain-specific migration knowledge, can effectively assist in automating Qiskit code migration. This work contributes both a set of proven prompts and taxonomy for Qiskit code migration from earlier versions to version 0.46 and a methodology to asses the capabilities of LLMs to assist in the migration of quantum code.

Figures

Figures reproduced from arXiv: 2506.14535 by the authors.

Figure 1
Figure 1. Architecture of the code repository used for the experiments. There are [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Evaluation criteria for the refactoring suggestions provided by the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 15 canonical work pages

  1. [1]

    Automatic Library Migration Using Large Language Models: First Results

    Aylton Almeida, Laerte Xavier, and Marco Tulio Valente. Automatic Library Migration Using Large Language Models: First Results. In Pro- ceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pages 427–433, October 2024. arXiv:2408.16151 [cs]

    work page Pith review arXiv 2024

  2. [2]

    PennyLang: Pioneering LLM- Based Quantum Code Generation with a Novel PennyLane-Centric Dataset, March 2025

    Haider Asif, Abdul Basit, Nouhaila Innan, Muhammad Kashif, Alberto Marchisio, and Muhammad Shafique. PennyLang: Pioneering LLM- Based Quantum Code Generation with a Novel PennyLane-Centric Dataset, March 2025. arXiv:2503.02497 [cs]

    arXiv 2025

  3. [3]

    AI-Driven Refactoring: A Pipeline for Identifying and Correcting Data Clumps in Git Repositories

    Nils Baumgartner, Padma Iyenghar, Timo Schoemaker, and Elke Pul- verm¨uller. AI-Driven Refactoring: A Pipeline for Identifying and Correcting Data Clumps in Git Repositories. Electronics, 13(9):1644, January 2024. Number: 9 Publisher: Multidisciplinary Digital Publishing Institute

    work page 2024

  4. [4]

    An Empirical Study on the Code Refactoring Capability of Large Language Models, November

    Jonathan Cordeiro, Shayan Noei, and Ying Zou. An Empirical Study on the Code Refactoring Capability of Large Language Models, November

    work page

  5. [5]

    Explor- ing LLM-Driven Explanations for Quantum Algorithms

    Giordano d’Aloisio, Sophie Fortz, Carol Hanna, Daniel Fortunato, Avner Bensoussan, E ˜naut Mendiluze Usandizaga, and Federica Sarro. Explor- ing LLM-Driven Explanations for Quantum Algorithms. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , ESEM ’24, pages 475–481, New York, NY , USA, October...

    work page 2024

  6. [6]

    Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves, April 2024

    Yihe Deng, Weitong Zhang, Zixiang Chen, and Quanquan Gu. Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves, April 2024. arXiv:2311.04205 [cs]

    arXiv 2024

  7. [7]

    Qiskit Code Assistant: Training LLMs for generating Quantum Com- puting Code, May 2024

    Nicolas Dupuis, Luca Buratti, Sanjay Vishwakarma, Aitana Viudes Forrat, David Kremer, Ismael Faro, Ruchir Puri, and Juan Cruz-Benito. Qiskit Code Assistant: Training LLMs for generating Quantum Com- puting Code, May 2024. arXiv:2405.19495 [quant-ph]

    arXiv 2024

  8. [8]

    A Survey On Large Language Models For Code Generation

    Nam Huynh and Beiyu Lin. A Survey On Large Language Models For Code Generation. March 2025

    work page 2025

Show all 30 references
  1. [9]

    Wood, Jake Lishman, Julien Gacon, Simon Martiel, Paul D

    Ali Javadi-Abhari, Matthew Treinish, Kevin Krsulich, Christopher J. Wood, Jake Lishman, Julien Gacon, Simon Martiel, Paul D. Nation, Lev S. Bishop, Andrew W. Cross, Blake R. Johnson, and Jay M. Gam- betta. Quantum computing with Qiskit, June 2024. arXiv:2405.08810 [quant-ph]

    2024 arXiv

  2. [10]

    Code generation for classical-quantum software systems modeled in UML

    Luis Jim ´enez-Navajas, Ricardo P´erez-Castillo, and Mario Piattini. Code generation for classical-quantum software systems modeled in UML. Softw Syst Model , January 2025

    2025

  3. [11]

    The Impact of Prompt Programming on Function- Level Code Generation, December 2024

    Ranim Khojah, Francisco Gomes de Oliveira Neto, Mazen Mohamad, and Philipp Leitner. The Impact of Prompt Programming on Function- Level Code Generation, December 2024. arXiv:2412.20545 [cs]

    2024 arXiv

  4. [12]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, page...

    2022

  5. [13]

    Retrieval- augmented generation for knowledge-intensive nlp tasks, 2021

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ¨uttler, Mike Lewis, Wen tau Yih, Tim Rockt ¨aschel, Sebastian Riedel, and Douwe Kiela. Retrieval- augmented generation for knowledge-intensive nlp tasks, 2021

    2021

  6. [14]

    InstructCoder: Instruction Tuning Large Language Models for Code Editing, February 2024

    Kaixin Li, Qisheng Hu, Xu Zhao, Hui Chen, Yuxi Xie, Tiedong Liu, Qizhe Xie, and Junxian He. InstructCoder: Instruction Tuning Large Language Models for Code Editing, February 2024. arXiv:2310.20329 [cs]

    2024 arXiv

  7. [15]

    Nation, Abdullah Ash Saki, Sebastian Brandhofer, Luciano Bello, Shelly Garion, Matthew Treinish, and Ali Javadi-Abhari

    Paul D. Nation, Abdullah Ash Saki, Sebastian Brandhofer, Luciano Bello, Shelly Garion, Matthew Treinish, and Ali Javadi-Abhari. Bench- marking the performance of quantum computing software, February

  8. [16]

    Quantum Computing in the NISQ era and beyond

    John Preskill. Quantum Computing in the NISQ era and beyond. Quantum, 2:79, August 2018. arXiv:1801.00862 [quant-ph]

    2018 arXiv

  9. [17]

    MQT Bench: Benchmarking Software and Design Automation Tools for Quantum Computing

    Nils Quetschlich, Lukas Burgholzer, and Robert Wille. MQT Bench: Benchmarking Software and Design Automation Tools for Quantum Computing. Quantum, 7:1062, July 2023. arXiv:2204.13719 [quant- ph]

    2023 arXiv

  10. [18]

    A Systematic Survey of Prompt Engineer- ing in Large Language Models: Techniques and Applications, February

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A Systematic Survey of Prompt Engineer- ing in Large Language Models: Techniques and Applications, February

  11. [19]

    Taxonomy of migration scenarios for qiskit refactoring using llms

    Jos ´e Manuel Su ´arez, Lu´ıs Mariano Bibb ´o Bibb ´o, Joaqu´ın Bogado, and Alejandro Fernandez. Taxonomy of migration scenarios for qiskit refactoring using llms. February 2025. Accepted for publication - JAIIO 54

    2025

  12. [20]

    Software associated to automatic qiskit code refac- toring using large language models

    Jos ´e Manuel Su ´arez, Joaquin Bogado, Luis Mariano Bibb ´o, and Ale- jandro Fern ´andez. Software associated to automatic qiskit code refac- toring using large language models. https://github.com/jwackito/qiskit experiments tlisc, June 2025

    2025

  13. [21]

    arXiv:2402.07927 [cs]

    arXiv

  14. [22]

    Zhang, Mark Harman, Don Syme, Joost Noppen, and Detlef Nauck

    Lukas Twist, Jie M. Zhang, Mark Harman, Don Syme, Joost Noppen, and Detlef Nauck. Llms love python: A study of llms’ bias for programming languages and libraries, 2025

    2025

  15. [23]

    TRANSAGENT: An LLM-Based Multi-Agent System for Code Translation, October 2024

    Zhiqiang Yuan, Weitong Chen, Hanlin Wang, Kai Yu, Xin Peng, and Yiling Lou. TRANSAGENT: An LLM-Based Multi-Agent System for Code Translation, October 2024. arXiv:2409.19894 [cs]

    2024 arXiv

  16. [24]

    RefactoringMiner 2.0

    Nikolaos Tsantalis, Ameya Ketkar, and Danny Dig. RefactoringMiner 2.0. IEEE Transactions on Software Engineering, 48(3):930–950, March

  17. [25]

    Migrating Code At Scale With LLMs At Google, April

    Celal Ziftci, Stoyan Nikolov, Anna Sj ¨ovall, Bo Kim, Daniele Codecasa, and Max Kim. Migrating Code At Scale With LLMs At Google, April

  18. [28]

    On Refactoring Quantum Programs, June 2023

    Jianjun Zhao. On Refactoring Quantum Programs, June 2023. arXiv:2306.10517 [cs]

    2023 arXiv

  19. [30]

    arXiv:2504.09691 [cs]

    arXiv

  20. [2022]

    Conference Name: IEEE Transactions on Software Engineering

  21. [2024]

    arXiv:2411.02320 [cs]

    arXiv

  22. [2025]

    arXiv:2409.08844 [quant-ph]

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.