pith. sign in

arxiv: 2602.22388 · v2 · submitted 2026-02-25 · ⚛️ physics.ed-ph

Assessing student learning in quantum computing: The challenging case of phase kickback

Josephine C. Meyer , Steven J. Pollock , Bethany R. Wilcox , Gina Passante This is my paper

Pith reviewed 2026-05-15 19:01 UTC · model grok-4.3

classification ⚛️ physics.ed-ph
keywords quantum computingstudent assessmentphase kickbackconceptual surveyeducation researchquantum information scienceassessment development
0
0 comments X

The pith

Item 15 on the quantum computing conceptual survey required more revisions than the other nineteen combined

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper recounts the long development process for a single question on a survey meant to measure student understanding of quantum computing. This item tests grasp of phase kickback and turned out to be far harder to write well than any of the others. Multiple rounds of changes drew on both student answer patterns and interviews to uncover where learners were going wrong. A reader would conclude that assessing quantum ideas demands repeated checks from different angles to match how students actually reason. This matters because clearer assessment tools can help instructors spot gaps and strengthen training programs in the field.

Core claim

The central claim is that Item 15 of the Quantum Computing Conceptual Survey, which asks about phase kickback, underwent more revision and team discussion than the rest of the survey combined. Student responses and interviews revealed persistent conceptual difficulties that earlier versions of the question missed or mismeasured. The authors present this case as concrete evidence that valid assessment items in quantum computing require triangulating survey statistics with qualitative data on student thinking.

What carries the argument

Item 15 of the QCCS, a question on phase kickback whose repeated revisions were driven by patterns in student answers and interviews to better capture conceptual understanding.

If this is right

  • Quantum education researchers should collect both quantitative survey data and qualitative interview data when refining assessment items.
  • Student reasoning about phase kickback contains layers that simple questions often fail to expose on the first try.
  • Curriculum designers can use the refined item to identify which students still hold specific misconceptions after instruction.
  • Triangulating data sources during assessment development leads to questions that more reliably track learning in quantum information topics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar extended validation may be needed for survey items on other quantum operations such as superposition or measurement.
  • The final item could be paired with open-ended follow-ups in future studies to map exact student explanations.
  • Programs training future quantum workers could adopt the survey to diagnose targeted gaps before placing trainees in technical roles.

Load-bearing premise

That the many rounds of revision based on mixed data sources produced a measurably more valid probe of student understanding without separate checks on its statistical performance.

What would settle it

Giving the final version of Item 15 to a large group of experts and novices and finding no better separation in correct answers than an earlier draft would show the revisions did not improve the item.

read the original abstract

A major challenge for quantum workforce development is the need to both understand and reliably assess student learning of quantum information science (QIS) fundamentals. Yet student thinking is notoriously difficult to probe, even for seasoned education researchers. This article presents the story of Item 15 on the Quantum Computing Conceptual Survey (QCCS). This assessment item underwent more revision and discussion within the team than the remaining 19 assessment questions combined. This paper provides a behind-the-scenes look at the development of this assessment question: a story that both reveals interesting findings about student reasoning in quantum computing and illustrates why quantum education researchers insist on triangulating diverse quantitative and qualitative data sources when developing and refining assessment items, with implications for any researcher looking to understand and measure student conceptual reasoning in quantum computing, as well as for QIS curriculum and workforce development more broadly.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 3 minor

Summary. The paper narrates the iterative development of Item 15 on the Quantum Computing Conceptual Survey (QCCS), an item targeting phase kickback. It details successive revisions informed by student responses, expert review, and team discussions, using this case to illustrate common student reasoning patterns in quantum computing and to advocate for triangulating multiple quantitative and qualitative data sources when refining assessment items.

Significance. If the narrative is accurate, the work supplies a concrete example of the difficulties in probing student conceptual understanding in quantum information science and demonstrates a practical workflow for assessment refinement. This is timely for QIS education research and curriculum development, as it models how to surface and address persistent misconceptions that affect workforce preparation.

minor comments (3)
  1. [Abstract and §1] The abstract states that Item 15 required more revision than the other 19 items combined; the main text should supply at least a brief comparative summary (e.g., number of major revisions or data sources per item) so readers can gauge the claim without relying solely on the authors' qualitative judgment.
  2. [§3–4] The final version of Item 15 (or its key distractors) is never reproduced verbatim; including the exact wording of the item at each major revision stage, perhaps in a table or appendix, would make the reported student reasoning patterns directly verifiable.
  3. [§5] The paper cites the value of triangulation but does not reference specific prior QIS assessment instruments (e.g., other conceptual surveys or concept inventories) that have used similar multi-method validation; adding one or two such citations would strengthen the methodological framing.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive and accurate summary of the manuscript, which correctly captures our focus on the iterative development of Item 15 on the QCCS targeting phase kickback. We appreciate the recognition of the work's timeliness for QIS education research and the recommendation for minor revision.

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper is a purely descriptive case study recounting the iterative revision process for one assessment item. It contains no derivations, equations, fitted parameters, predictions, or theoretical claims that could reduce to self-referential inputs by construction. The central illustration—that triangulating multiple data sources aids item refinement—is carried directly by the narrative of revisions and findings described, without any load-bearing step that relies on self-citation chains, ansatzes, or renaming of prior results. This is consistent with the genre of education-research reflection pieces and requires no external validation metrics for the stated purpose to hold.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that mixed-methods triangulation yields more trustworthy measures of conceptual understanding than any single data source; no free parameters or invented entities are introduced.

axioms (1)
  • domain assumption Student responses to conceptual survey items can be interpreted through combined qualitative and quantitative analysis to reveal genuine misconceptions.
    Invoked throughout the description of Item 15 development and the call for triangulation.

pith-pipeline@v0.9.0 · 5445 in / 1169 out tokens · 25673 ms · 2026-05-15T19:01:40.351035+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 30 canonical work pages

  1. [1]

    Meyer, S

    J. Meyer, S. Goorney, T. Kushimo, Z. Seskir. Resource letter: Quantum information education (2026). (American Journal of Physics, in press)

    work page 2026

  2. [2]

    L. Hahn, P. Klein, Eye tracking in physics education research: A systematic literature review. Phys. Rev. Phys. Educ. Res.18(1), 013102 (2022). https://doi.org/10.1103/PhysRevPhysEducRes.18.013102

    work page doi:10.1103/physrevphyseducres.18.013102 2022

  3. [3]

    Schoenfeld, R

    A.H. Schoenfeld, R. Floden, F. El Chidiac, D. Gillingham, H. Fink, S. Hu, A. Sayavedra, A. Weltman, A. Zarkh, On Classroom Observations. Journal for STEM Educ Res1(1), 34–59 (2018). https://doi. org/10.1007/s41979-018-0001-7

    work page doi:10.1007/s41979-018-0001-7 2018

  4. [4]

    Leighton,Using Think-Aloud Interviews and Cognitive Labs in Educational Research(Oxford University Press, 2017)

    J.P. Leighton,Using Think-Aloud Interviews and Cognitive Labs in Educational Research(Oxford University Press, 2017)

    work page 2017

  5. [5]

    Sherin, V

    B. Sherin, V. Lee, and M. Krakowski, Conceptual dynamics in clinical interviews. AIP Conf. Proc.951, 23–26 (2007). https://doi.org/10.1063/1.2820937 14

    work page doi:10.1063/1.2820937 2007

  6. [6]

    Russ, V.R

    R.S. Russ, V.R. Lee, B.L. Sherin, Framing in cognitive clinical interviews about intuitive science knowledge: Dynamic student understandings of the discourse interaction. Sci. Educ.96(4), 573–599 (2012). https://doi.org/10.1002/sce.21014

    work page doi:10.1002/sce.21014 2012

  7. [7]

    Madsen, S

    A. Madsen, S. McKagan, E. Sayre, Resource letter RBAI-1: Research-based assessment instruments in physics and astronomy. Am. J. Phys.85, 245–264 (2017). https://doi.org/10.1119/1.4977416

    work page doi:10.1119/1.4977416 2017

  8. [8]

    Aiello, D.D

    C.D. Aiello, D.D. Awschalom, H. Bernien, T. Brower, K.R. Brown, T.A. Brun, J.R. Caram, E. Chitambar, R. Di Felice, K.M. Edmonds, M.F.J. Fox, S. Haas, A.W. Holleitner, E.R. Hudson, J.H. Hunt, R. Joynt, S. Koziol, M. Larsen, H.J. Lewandowski, D.T. McClure, J. Palsberg, G. Passante, K.L. Pudenz, C.J.K. Richardson, J.L. Rosenberg, R.S. Ross, M. Saffman, M. Si...

    work page doi:10.1088/2058-9565/abfa64 2021

  9. [9]

    Meyer, M

    J.C. Meyer, M. Griston, G. Passante, S.J. Pollock, B.R. Wilcox. Measuring student understanding in quantum computing: Development and validation of the quantum computing conceptual survey (2026). (Draft, release expected on arXiv in Q2 2026.)

    work page 2026

  10. [10]

    Plueger, BR

    JR. Plueger, BR. Wilcox, SJ. Pollock, G. Passante. Can a CNOT gate affect the control qubit? Student resources for understanding CNOT and entanglement (2026)

    work page 2026

  11. [11]

    Meyer, New Paradigms in Quantum Education for the Second Quantum Revolution

    J.C. Meyer, New Paradigms in Quantum Education for the Second Quantum Revolution. Ph.D. thesis, University of Colorado at Boulder, United States – Colorado (2025). URL https://www.proquest.com/ docview/3205639873/abstract/3BE31EA73A2D4169PQ/1

    work page arXiv 2025

  12. [12]

    Meyer, G

    J. Meyer, G. Passante, S. Pollock, and B. Wilcox, Today’s interdisciplinary quantum information classroom: Themes from a survey of quantum information science instructors. Phys. Rev. Phys. Educ. Res.18(010150) (2022). https://doi.org/10.1103/PhysRevPhysEducRes.18.010150

    work page doi:10.1103/physrevphyseducres.18.010150 2022

  13. [13]

    Passante, P

    G. Passante, P. Emigh, P. Shaffer, Student ability to differentiate between superposition states and mixed states in quantum mechanics. Phys. Rev. ST Phys. Educ. Res.11(020135) (2015). https://doi. org/10.1103/PhysRevSTPER.11.020135

    work page doi:10.1103/physrevstper.11.020135 2015

  14. [14]

    T. Wan, P. Emigh, P. Shaffer, Probing student reasoning in relating relative phase and quantum phenomena. Phys. Rev. Phys. Educ. Res.15(020139) (2019). https://doi.org/10.1103/ PhysRevPhysEducRes.15.020139

    work page 2019

  15. [15]

    Meyer, G

    J. Meyer, G. Passante, S. Pollock, M. Vignal, and B. Wilcox,Investigating Students’ Strategies for Interpreting Quantum States in an Upper-Division Quantum Computing Course, inProc. 2021 Phys. 15 Educ. Res. Conf.(2021), pp. 289–294. https://doi.org/10.1119/perc.2021.pr.Meyer

    work page doi:10.1119/perc.2021.pr.meyer 2021

  16. [16]

    Meyer, G

    J. Meyer, G. Passante, S. Pollock, and B. Wilcox,Investigating Student Interpretations of the Difference between Classical and Quantum Computers: Are Quantum Computers Just Analog Classical Computers?, inProc. 2022 Phys. Educ. Res. Conf.(2022), pp. 317–322. https://doi.org/10.1119/perc.2022.pr.Meyer

    work page doi:10.1119/perc.2022.pr.meyer 2022

  17. [17]

    Almudever, and Sebastian Feld

    T. Kushimo, B. Thacker,Investigating Students’ Strengths and Difficulties in Quantum Computing, inProc IEEE Int. Conf. Quantum Comput. Eng., vol. 3 (2023), pp. 33–39. https://doi.org/10.1109/ QCE57702.2023.20322

    work page arXiv 2023

  18. [18]

    Kaltenbaek, M

    J. Meyer, G. Passante, S. Pollock, B. Wilcox, Introductory quantum information science courses at US institutions: Content coverage. EPJ Quantum Technol.11(16) (2024). https://doi.org/10.1140/epjqt/ s40507-024-00226-0

    work page doi:10.1140/epjqt/ 2024

  19. [19]

    P. Kaye, R. Laflamme, M. Mosca,An Introduction to Quantum Computing. Oxford Scholarship Online (Oxford University Press, Oxford, 2007). https://doi.org/10.1093/oso/9780198570004.001.0001

    work page doi:10.1093/oso/9780198570004.001.0001 2007

  20. [20]

    & Jozsa, R

    D. Deutsch, R. Jozsa, Rapid solution of problems by quantum computation. Proc. R. Soc. Lond. Ser. Math. Phys. Sci.439(1907), 553–558 (1992). https://doi.org/10.1098/rspa.1992.0167

    work page doi:10.1098/rspa.1992.0167 1907

  21. [21]

    Quantum complexity theory,

    E. Bernstein, U. Vazirani,Quantum Complexity Theory, inProc. Twenty-Fifth Annu. ACM Symp. Theory Comput.(Association for Computing Machinery, New York, NY, USA, 1993), STOC ’93, pp. 11–20. https://doi.org/10.1145/167088.167097

    work page doi:10.1145/167088.167097 1993

  22. [22]

    In: Pro- ceedings 35th Annual Symposium on Foundations of Computer Science, pp

    P. Shor,Algorithms for Quantum Computation: Discrete Logarithms and Factoring, inProc 35th Annu. Symp. Found Comput Sci(1994), pp. 124–134. https://doi.org/10.1109/SFCS.1994.365700

    work page doi:10.1109/sfcs.1994.365700 1994

  23. [23]

    Vignal, K

    M. Vignal, K. Rainey, B. Wilcox, M. Caballero, H. Lewandowski,Affordances of Articulating Assessment Objectives in Research-Based Assessment Development, inProc. 2022 Phys. Educ. Res. Conf.(2022), pp. 475–480. https://doi.org/10.1119/perc.2022.pr.Vignal

    work page doi:10.1119/perc.2022.pr.vignal 2022

  24. [24]

    Ding, inAdvances in Applications of Rasch Measurement in Science Education, ed

    L. Ding, inAdvances in Applications of Rasch Measurement in Science Education, ed. by X. Liu, W. Boone (Springer Cham, 2023), pp. 19–46. https://doi.org/10.1007/978-3-031-28776-3_2

    work page doi:10.1007/978-3-031-28776-3_2 2023

  25. [25]

    Engelhardt, inGetting Started in Physics Education Research, vol

    P. Engelhardt, inGetting Started in Physics Education Research, vol. 2 (AAPT, 2009), pp. 1–40

    work page 2009

  26. [26]

    Efron, R.J

    B. Efron, R.J. Tibshirani,An Introduction to the Bootstrap(CRC Press, 1994)

    work page 1994

  27. [27]

    Forthmann, N

    B. Forthmann, N. Förster, B. Schütze, K. Hebbecker, J. Flessner, M.T. Peters, E. Souvignier, How Much g Is in the Distractor? Re-Thinking Item-Analysis of Multiple-Choice Items. J. Intell.8(1), 11 (2020). https://doi.org/10.3390/jintelligence8010011 16

    work page doi:10.3390/jintelligence8010011 2020

  28. [28]

    Jackson, J

    B. Jackson, J. Scargle, D. Barnes, S. Arabhi, A. Alt, P. Gioumousis, E. Gwin, P. Sangtrakulcharoen, L. Tan, T.T. Tsai, An algorithm for optimal partitioning of data on an interval. IEEE Signal Process. Lett.12(2), 105–108 (2005). https://doi.org/10.1109/LSP.2001.838216

    work page doi:10.1109/lsp.2001.838216 2005

  29. [29]

    Weisstein

    E.W. Weisstein. Spherical Coordinates (2025). URL https://mathworld.wolfram.com/ SphericalCoordinates.html

    work page 2025

  30. [30]

    None of the Above

    D. DiBattista, J.A. Sinnige-Egger, G. Fortuna, The “None of the Above” Option in Multiple-Choice Testing: An Experimental Study. J. Exp. Educ.82(2), 168–183 (2014). https://doi.org/10.1080/ 00220973.2013.795127 17

    work page arXiv 2014