REVIEW 5 major objections 5 minor 35 references
Taxonomy of migration scenarios for Qiskit refactoring using LLMs
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read LLMs can generate a Qiskit migration taxonomy that largely matches—and sometimes exceeds—one built by expert developers, in far less time.
desk verdict A modest but honest first cut at using LLMs to build a Qiskit migration taxonomy, with a real reproducibility pipeline, yet the central 'high correspondence' claim is currently unsupported by any measurement. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a paired taxonomy-generation pipeline with two arms. The manual arm produces a taxonomy from Qiskit release notes and changelogs, with each scenario described by category, migration flow, summary, artifacts, source and target code examples, difficulty, and SE/QSE impact. The automatic arm feeds the same documentation into a system prompt with few-shot examples, instructing an LLM to emit rows in the same markdown schema. Comparative analysis of the two taxonomies—overlap, gaps in either direction, and richness of examples—is what grounds the inference. The classification dimensions themselves are the objects that make the comparison meaningful.
What would settle it
Have several independent expert teams each build a taxonomy of the same Qiskit version migrations, measure their inter-rater agreement, then run the LLM prompts against the validated consensus taxonomy; if overlap drops below a pre-agreed threshold (for example, recall under 80%) or if most LLM-only scenarios are rejected as hallucinations, the claim of suitability fails.
Extended reading notes
Core claim
The paper's central finding is that when LLM prompts include the official Qiskit documentation, the automatically generated taxonomy shows high correspondence with the manual expert taxonomy, and in several cases the LLM output surpasses it in descriptive precision, completeness of code examples, and detection of scenarios the experts did not list. From this the authors infer that using LLMs for automatic taxonomy generation of quantum software engineering migration scenarios is plausible and suitable, substantially reducing the time required. The effect is clearest for major version updates, where documentation is extensive and the API changes are disruptive.
Load-bearing premise
The comparison assumes the manually built taxonomy is a correct and complete baseline, but the paper does not report how many experts built it, whether they agreed with each other, or how scenarios were validated.
Editorial extensions
If this is right
- LLM-generated taxonomies can serve as a fast first pass, letting experts spend their time validating and refining rather than building the catalog from scratch.
- The unified taxonomy becomes a reusable evaluation harness: migration tools can be scored per scenario, difficulty level, and QSE specificity.
- Feeding official documentation directly into the prompt is the main quality lever, and context-length limits can be handled with retrieval augmentation or summarization.
- Prioritizing tool development can now target the high-difficulty and quantum-specific scenarios that no classical refactoring tool covers.
Reading between the lines
- The paper's evidence for agreement is qualitative; a quantitative score (recall/precision or inter-rater agreement against multiple experts) would convert 'high correspondence' into a measurable claim.
- LLM-only scenarios flagged as 'extra' could be either genuine gaps in the manual taxonomy or hallucinations; the paper does not distinguish, so a follow-up study should have experts rate them.
- Because Qiskit is Python-based and LLMs are known to be strongest at Python, the result may not transfer to quantum languages with C-like syntax such as Q#, which the paper itself acknowledges needs validation.
- The manual taxonomy is treated as ground truth, yet its own reliability is unreported; a repeatability study with several independent expert taxonomies would make the comparison sound.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a feasibility study in which the authors construct a taxonomy of refactoring scenarios for migrating Qiskit code between versions, first manually from official documentation and then automatically by prompting several LLMs (gemma-3-27b-it, DeepSeek-R1-Distill-Qwen-32B, GPT-4o, GPT-4o-mini, DeepSeek-V3, DeepSeek-R1). The taxonomies are compared qualitatively in Section 4, and the authors conclude that LLMs are 'plausible and suitable' for automatic taxonomy generation and that they would 'substantially reduc[e] the required time.' The manuscript also provides a GitHub repository with scripts for documentation extraction, verification, and model access, and discusses future extensions such as RAG and full migration tooling.
Significance. If the central claim were backed by quantitative evidence, the contribution would be useful: a reproducible taxonomy of Qiskit migration scenarios could support benchmarking of automatic refactoring tools and guide QSE tooling. The paper has real strengths: it uses official Qiskit documentation as the source for both taxonomies, releases scripts to reproduce extraction and prompting, tests multiple open and commercial models, and candidly discusses hallucinations, context-length constraints, and the need for future validation. However, the evidence offered in the current version is primarily informal; the claimed correspondence is not measured, the manual baseline is not validated, and the promised unified taxonomy is not included. The main skeptical concern about the paper is justified: the headline inference depends on an unquantified comparison and an unmeasured time-saving claim, so it cannot currently be independently checked.
major comments (5)
- [§4, final paragraph] The paper's central inference—that LLM use is 'plausible and suitable' and would lead to 'substantially reducing the required time'—rests on qualitative statements such as 'high correspondence' and 'models behave the best.' No matching protocol, agreement metric, row counts, per-model breakdown, or repeated-run variance is reported. Moreover, because the prompt enforces the same column structure and a few-shot example as the manual taxonomy, structural alignment is partly by construction; the substantive question is scenario-level agreement, which is not measured. Please add a quantitative comparison (e.g., precision/recall/F1 or Cohen's kappa) with an explicit scenario-matching rule, and report results for each model and run.
- [§3.2] The manual taxonomy is used as the reference against which LLM output is judged, but the manuscript gives no information about who built it, how many people were involved, whether there was any inter-rater agreement, or how scenarios were selected and validated. It also says versions 0.46.0 and 1.0.0 were considered 'exhaustively' while later stating that 13 version updates were analyzed, which is internally inconsistent. A weak or unvalidated baseline makes 'extra scenarios' from the LLM uninterpretable, since they may reflect hallucinations or duplicates rather than genuine omissions. Please document the construction and validation of the baseline and clarify which versions are actually covered.
- [Abstract and §3.1] The abstract and introduction promise a 'unified taxonomy' produced by integrating the manual and LLM taxonomies, but the full text never presents this artifact: there is no table, figure, or appendix showing the unified taxonomy, nor a list of the migration scenarios for the analyzed versions. Since this taxonomy is the paper's main contribution and the foundation for future work, the manuscript is incomplete without it. Please include the unified taxonomy (or clearly mark it as supplementary material with an accessible link).
- [§3.3 and §4] LLM generation is stochastic, yet the paper does not report the number of independent runs, the temperature/sampling settings, or the aggregation/selection rule used for the reported outputs. A single run per model-prompt pair could easily produce favorable or unfavorable outcomes by chance, and the reported phases ('initial,' 'intermediate') appear to be based on informal observation rather than logged results. Please specify the sampling configuration and report variance across repeated runs.
- [§5] The claim that LLM-assisted taxonomy generation 'can save time of the SE expert' is plausible but unsupported: no manual effort (person-hours), LLM runtime, cost, or comparative workflow time is measured anywhere in the manuscript. Since the abstract and conclusion repeat the time-saving claim, it should either be measured (even roughly) or explicitly reframed as a conjecture.
minor comments (5)
- [§3.2 and §3.3] There are several typographical errors that should be corrected: 'Leatest' for 'Latest,' 'chanel' for 'channel,' 'Qween' for 'Qwen,' and 'asses' for 'assess' in Sections 5 and 6.
- [§3.3] The GitHub URL is rendered with spaces ('qiskit llm experiment') and should be a proper clickable link.
- [§4, Figure 2] Figure 2 is described as examples of scenarios but is used to support a general claim about correspondence; the caption should clarify that it is illustrative and not a quantitative sample.
- [§4 and §5, Figure 4] Figure 4 is referenced in Section 4 ('See Fig. 4') before it is introduced; consider moving or renumbering figures so that references follow the figures.
- [§1] The phrase 'hybrid methodology' is used without a precise definition; it seems to mean combining manual analysis with LLM assistance, but this could be stated explicitly.
Circularity Check
No significant circularity: the paper is an empirical comparison, not a derivation, and no claim reduces to its inputs by construction.
full rationale
The paper's central claim is that LLMs are 'plausible and suitable for automatic taxonomy generation of QSE migration scenarios' based on a comparison between a manually built taxonomy and LLM-generated taxonomies. This is an empirical evaluation, not a mathematical derivation, so the circularity patterns involving fitted parameters, self-referential definitions, or imported uniqueness theorems do not apply. I checked the possible sources of circularity. There is no self-citation chain: the authors do not rely on their own prior work to justify the premises. There is no uniqueness theorem invoked to force a choice. There is no ansatz smuggled via citation. The taxonomy is not a renaming of a known result; it is a new categorization of Qiskit migration scenarios. The closest potential issue is that the LLM prompt includes 'a one-shot example of a row and the information retrieved by the document extraction script' and defines 'the column names and expected content' from the manual taxonomy, so some structural agreement between the LLM output and the manual taxonomy is prompt-induced. However, the scenario content, labels, and examples are still generated by the model from the Qiskit documentation, and the paper does not define LLM suitability as equivalent to reproducing the manual taxonomy. The 'substantially reducing the required time' statement is unmeasured, but an unsupported time-saving claim is a weakness in evidence, not a circularity. The manual baseline's lack of inter-rater validation is also an external-validity concern, not a circularity. Therefore, under the rules requiring a specific reduction to inputs, no circular step can be established, and the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (2)
- domain assumption Qiskit official documentation and release notes are authoritative and sufficiently complete to enumerate migration scenarios.
- domain assumption The manual taxonomy is a reliable ground truth for comparison.
Cite this review
Pith. "Pith review of Taxonomy of migration scenarios for Qiskit refactoring using LLMs." pith.science (2026). https://pith.science/paper/63J3WVMP
@misc{pith2026250607135,
author = {Pith},
title = {Pith review of: Taxonomy of migration scenarios for Qiskit refactoring using LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/63J3WVMP}},
note = {Machine review of arXiv:2506.07135}
}
read the original abstract
As quantum computing advances, quantum programming libraries' heterogeneity and steady evolution create new challenges for software developers. Frequent updates in software libraries break working code that needs to be refactored, thus adding complexity to an already complex landscape. These refactoring challenges are, in many cases, fundamentally different from those known in classical software engineering due to the nature of quantum computing software. This study addresses these challenges by developing a taxonomy of quantum circuit's refactoring problems, providing a structured framework to analyze and compare different refactoring approaches. Large Language Models (LLMs) have proven valuable tools for classic software development, yet their value in quantum software engineering remains unexplored. This study uses LLMs to categorize refactoring needs in migration scenarios between different Qiskit versions. Qiskit documentation and release notes were scrutinized to create an initial taxonomy of refactoring required for migrating between Qiskit releases. Two taxonomies were produced: one by expert developers and one by an LLM. These taxonomies were compared, analyzing differences and similarities, and were integrated into a unified taxonomy that reflects the findings of both methods. By systematically categorizing refactoring challenges in Qiskit, the unified taxonomy is a foundation for future research on AI-assisted migration while enabling a more rigorous evaluation of automated refactoring techniques. Additionally, this work contributes to quantum software engineering (QSE) by enhancing software development workflows, improving language compatibility, and promoting best practices in quantum programming.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Ali, S., Arcaini, P., Wang, X., and Yue, T. (2021). Assessing the Effectiveness of Input and Output Coverage Criteria for Testing Quantum Programs . In 2021 14th IEEE Conference on Software Testing , Verification and Validation ( ICST ) , pages 13--23. ISSN: 2159-4848
work page 2021
-
[2]
Almeida, A., Xavier, L., and Valente, M. T. (2024). Automatic Library Migration Using Large Language Models : First Results . In Proceedings of the 18th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement , pages 427--433. arXiv:2408.16151 [cs]
arXiv 2024
-
[3]
staq -- A full-stack quantum processing toolkit
Amy, M. and Gheorghiu, V. (2020). staq -- A full-stack quantum processing toolkit. Quantum Sci. Technol. , 5(3):034016. arXiv:1912.06070 [quant-ph]
work page Pith review arXiv 2020
-
[4]
Asif, H., Basit, A., Innan, N., Kashif, M., Marchisio, A., and Shafique, M. (2025). PennyLang : Pioneering LLM - Based Quantum Code Generation with a Novel PennyLane - Centric Dataset . arXiv:2503.02497 [cs]
arXiv 2025
-
[5]
K., Houkpati, Y., Irungu, J., and Oladunni, T
Bengesi, S., El-Sayed, H., Sarker, M. K., Houkpati, Y., Irungu, J., and Oladunni, T. (2023). Advancements in Generative AI : A Comprehensive Review of GANs , GPT , Autoencoders , Diffusion Model , and Transformers . arXiv:2311.10242 [cs]
arXiv 2023
-
[6]
Biamonte, J., Wittek, P., Pancotti, N., Rebentrost, P., Wiebe, N., and Lloyd, S. (2017). Quantum machine learning. Nature , 549(7671):195–202
work page 2017
-
[7]
Cordeiro, J., Noei, S., and Zou, Y. (2024). An Empirical Study on the Code Refactoring Capability of Large Language Models . arXiv:2411.02320 [cs]
arXiv 2024
-
[8]
Deng, Y., Zhang, W., Chen, Z., and Gu, Q. (2024). Rephrase and Respond : Let Large Language Models Ask Better Questions for Themselves . arXiv:2311.04205 [cs]
arXiv 2024
Show all 35 references
-
[9]
V., Kremer, D., Faro, I., Puri, R., and Cruz-Benito, J
Dupuis, N., Buratti, L., Vishwakarma, S., Forrat, A. V., Kremer, D., Faro, I., Puri, R., and Cruz-Benito, J. (2024). Qiskit Code Assistant : Training LLMs for generating Quantum Computing Code . arXiv:2405.19495 [quant-ph]
2024 arXiv
-
[10]
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05530 [cs]
2024 arXiv
-
[11]
S., Lumsdaine, P
Green, A. S., Lumsdaine, P. L., Ross, N. J., Selinger, P., and Valiron, B. (2013). Quipper: A Scalable Quantum Programming Language . SIGPLAN Not. , 48(6):333--342. arXiv:1304.3390 [cs]
2013 arXiv
-
[12]
Guo, J., Lou, H., Li, R., Fang, W., Liu, J., Long, P., Ying, S., and Ying, M. (2023). isQ : Towards a Practical Software Stack for Quantum Programming . arXiv:2205.03866 [quant-ph]
2023 arXiv
-
[13]
V., Kafri, D., Martin, O., Heidweiller, C
Isakov, S. V., Kafri, D., Martin, O., Heidweiller, C. V., Mruczkiewicz, W., Harrigan, M. P., Rubin, N. C., Thomson, R., Broughton, M., Kissell, K., Peters, E., Gustafson, E., Li, A. C. Y., Lamm, H., Perdue, G., Ho, A. K., Strain, D., and Boixo, S. (2021). Simulations of Quantu...
2021 arXiv
-
[14]
J., Lishman, J., Gacon, J., Martiel, S., Nation, P
Javadi-Abhari, A., Treinish, M., Krsulich, K., Wood, C. J., Lishman, J., Gacon, J., Martiel, S., Nation, P. D., Bishop, L. S., Cross, A. W., Johnson, B. R., and Gambetta, J. M. (2024). Quantum computing with Qiskit . arXiv:2405.08810 [quant-ph]
2024 arXiv
-
[15]
Jiménez-Navajas, L., Pérez-Castillo, R., and Piattini, M. (2025). Code generation for classical-quantum software systems modeled in UML . Softw Syst Model
2025
-
[16]
Jin, H., Chen, H., Lu, Q., and Zhu, L. (2025). Towards Advancing Code Generation with Large Language Models : A Research Roadmap . arXiv:2501.11354 [cs]
2025 arXiv
-
[17]
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., tau Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. (2021). Retrieval-augmented generation for knowledge-intensive nlp tasks
2021
-
[18]
Li, G., Zhou, L., Yu, N., Ding, Y., Ying, M., and Xie, Y. (2020). Proq: Projection -based Runtime Assertions for Debugging on a Quantum Computer . arXiv:1911.12855 [cs]
2020 arXiv
-
[19]
M., Garcia-Alonso, J., Moguel, E., Barzen, J., Leymann, F., Ali, S., Yue, T., Arcaini, P., Castillo, R
Murillo, J. M., Garcia-Alonso, J., Moguel, E., Barzen, J., Leymann, F., Ali, S., Yue, T., Arcaini, P., Castillo, R. P., Guzmán, I. G. R. d., Piattini, M., Ruiz-Cortés, A., Brogi, A., Zhao, J., Miranskyy, A., and Wimmer, M. (2025). Quantum Software Engineering : Roadmap and Cha...
2025 arXiv
-
[20]
D., Saki, A
Nation, P. D., Saki, A. A., Brandhofer, S., Bello, L., Garion, S., Treinish, M., and Javadi-Abhari, A. (2025). Benchmarking the performance of quantum computing software. arXiv:2409.08844 [quant-ph]
2025 arXiv
- [21]
-
[22]
Preskill, J. (2018). Quantum Computing in the NISQ era and beyond. Quantum , 2:79. arXiv:1801.00862 [quant-ph]
2018 arXiv
-
[23]
Quetschlich, N., Burgholzer, L., and Wille, R. (2023). MQT Bench : Benchmarking Software and Design Automation Tools for Quantum Computing . Quantum , 7:1062. arXiv:2204.13719 [quant-ph]
2023 arXiv
-
[24]
Ramalho, N. C. L., Souza, H. A. d., and Chaim, M. L. (2024). Testing and Debugging Quantum Programs : The Road to 2030. arXiv:2405.09178 [cs]
2024 arXiv
-
[25]
E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C. C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F., Touvron, H., Mart...
2024 arXiv
-
[26]
K., Saha, S., Jain, V., Mondal, S., and Chadha, A
Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., and Chadha, A. (2024). A Systematic Survey of Prompt Engineering in Large Language Models : Techniques and Applications . arXiv:2402.07927 [cs]
2024 arXiv
-
[27]
Sivarajah, S., Dilkes, S., Cowtan, A., Simmons, W., Edgington, A., and Duncan, R. (2021). t\ \ ket\ rangle\ : A Retargetable Compiler for NISQ Devices . Quantum Sci. Technol. , 6(1):014003. arXiv:2003.10611 [quant-ph]
2021 arXiv
-
[28]
M., Geller, A., Troyer, M., Azariah, J., Granade, C., Heim, B., Kliuchnikov, V., Mykhailova, M., Paz, A., and Roetteler, M
Svore, K. M., Geller, A., Troyer, M., Azariah, J., Granade, C., Heim, B., Kliuchnikov, V., Mykhailova, M., Paz, A., and Roetteler, M. (2018). Q\#: Enabling scalable quantum computing and development with a high-level domain-specific language. In Proceedings of the Real World D...
2018 arXiv
-
[29]
Tsantalis, N., Ketkar, A., and Dig, D. (2022). RefactoringMiner 2.0. IEEE Transactions on Software Engineering , 48(3):930--950. Conference Name: IEEE Transactions on Software Engineering
2022
-
[30]
M., Harman, M., Syme, D., Noppen, J., and Nauck, D
Twist, L., Zhang, J. M., Harman, M., Syme, D., Noppen, J., and Nauck, D. (2025). Llms love python: A study of llms' bias for programming languages and libraries
2025
-
[31]
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. (2023). Chain-of-thought prompting elicits reasoning in large language models
2023
-
[32]
D., He, J., Muller, M., Hoefer, G., Miles, R., and Geyer, W
Weisz, J. D., He, J., Muller, M., Hoefer, G., Miles, R., and Geyer, W. (2024). Design Principles for Generative AI Applications . In Proceedings of the CHI Conference on Human Factors in Computing Systems , pages 1--22. arXiv:2401.14484 [cs]
2024 arXiv
-
[33]
Xu, Z., Ying, M., and Ying, S. (2018). A Logic for Recursive Quantum Programs . arXiv:1812.00349 [cs]
2018 arXiv
-
[34]
Zhao, J. (2021). Some Size and Structure Metrics for Quantum Software . arXiv:2103.08815 [cs]
2021 arXiv
-
[35]
Zhao, J. (2023). On Refactoring Quantum Programs . arXiv:2306.10517 [cs]
2023 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.