Pith. sign in

REVIEW 4 major objections 6 minor 41 references

Using LLMs to Detect Growth in Computational Thinking in Introductory Physics

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A multimodal LLM prompted with the same rubric as human coders can scale pre–post measurement of computational-thinking growth in written physics responses.

desk verdict The LLM-scoring idea is plausible and the study is real, but the reported 85% MSP agreement is mathematically incompatible with Table II means, so the validation evidence needs correction before the main claim can be trusted. read the letter →

arxiv 2608.06200 v1 pith:JVD3KPUU submitted 2026-08-06 physics.ed-ph

Sean Savage , Anand Shanker , Grace Michlitsch , N. Sanjay Rebello This is my paper · ORCID
classification physics.ed-ph
keywords computationalthinkingphysicseducationlargelanguagemodelsautomatedscoringinter-raterreliabilitypre-postassessmentsystemsdatapractices
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a multimodal LLM, given the same rubric and survey materials humans used, can reproduce human judgment well enough to scale measurement of computational-thinking growth in large physics courses. On a 50-student validation set, the LLM reached substantial agreement with human consensus for the well-defined practices—$\kappa=0.90$ for Data Practices, $\kappa=0.78$ for Modeling and Simulation, and $\kappa=0.69$ for Computational Problem-Solving—while agreement was lower for Physics Correctness ($\kappa=0.53$) and Systems Thinking ($\kappa=0.48$). Deployed across all 936 students, the model detected the same significant pre–post growth trends in Data Practices and Computational Problem-Solving that humans found, and the same ceiling effect on a simulation-design prompt. The authors argue that the model's difficulty with Systems Thinking mirrors human raters' own difficulty, so the construct, not the model, is the limiting factor.

What carries the argument

The load-bearing mechanism is the transfer of the exact human scoring rubric to the LLM through structured prompting, combined with a three-run majority vote that mirrors the human majority-vote baseline. The rubric scores each response on a 0–2 scale for each computational-thinking practice—Data Practices, Computational Problem-Solving, Modeling and Simulation, and Systems Thinking—plus Physics Correctness, with definitions of superficial and deep engagement refined during human calibration. The LLM is fed the survey question's image, the prompt text, and the student's response in a single context window, asked to justify its reasoning, and then required to assign the same 0–2 score. Agreement between the model and the human consensus is measured with inter-rater kappa statistics before the pipeline is trusted on the full dataset.

What would settle it

Draw a fresh random sample from the 886 students whose responses were never used to validate the LLM, have at least three human raters score them with the same rubric, and compare that human consensus to the LLM's scores. If agreement on Data Practices and Computational Problem-Solving falls below moderate ($\kappa < 0.4$) or the pre–post growth directions change sign, the claim that the LLM scales human-quality evaluation fails.

Watch

Extended reading notes

Core claim

The paper claims that a large language model can serve as a reliable stand-in for human qualitative coding of computational thinking in open-ended physics explanations, provided it is given the same rubric, the same survey images and prompts, and a majority-vote procedure. On the 50-student validation set, the model's agreement with human consensus was substantial for well-specified practices—$\kappa=0.90$ for Data Practices, $0.78$ for Modeling and Simulation, $0.69$ for Computational Problem-Solving—and moderate for the more integrative Physics Correctness ($\kappa=0.53$) and Systems Thinking ($\kappa=0.48$). When deployed across all 936 students, the LLM reproduced the human-findable trends: significant pre–post growth on graph interpretation and code tracing, and no growth on the simulation-design prompt, which the authors attribute to a ceiling effect. The authors interpret the weak Systems Thinking agreement as evidence that the construct, not the model, is the limiting factor, since human coders reached similar low agreement on it.

Load-bearing premise

The 50-student human-consensus sample is representative of the full 936-student population, so agreement with that consensus on those 50 students justifies trusting the LLM's scores for the other 886 students, none of whom were independently human-coded.

Editorial extensions

If this is right

  • Instructors in large-enrollment physics courses can use a validated LLM pipeline to score open-ended computational-thinking responses, recovering rich measures instead of relying only on multiple-choice instruments.
  • The validation workflow—small human-coded sample, rubric refinement, LLM agreement check, then scaled deployment—can be reused by other courses with their own prompts.
  • Assessment items that show ceiling effects, such as the simulation-design prompt, can be revised to ask students to explain parameter interactions, and the LLM pipeline can then detect growth on the revised item.
  • LLM scores for integrative constructs such as Systems Thinking should be treated as approximate until rubrics are made more explicit, because the model's uncertainty there mirrors human raters' uncertainty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the authors do not run is an ablation of the three-run majority vote: comparing single-pass LLM scores against human consensus would show how much of the reliability comes from voting rather than from the rubric alone.
  • Because the model already generates structured justifications before scoring, the pipeline could be extended to produce per-student feedback on specific physics and computational practices, not just aggregate growth statistics.
  • Feeding the model only the text of each response, without the survey image, would isolate how much of the agreement depends on multimodal context; if text-only accuracy is similar, the visual input is not the decisive component.
  • The ceiling argument for the simulation-design prompt suggests a conservative check: look at the distribution of pre-test scores; if most students already score 2, the item cannot detect growth, and the paper's no-growth result for Modeling and Simulation is expected rather than evidence of no learning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper investigates whether a multimodal large language model (GPT-5.4-mini) can reliably score open-ended student responses for computational thinking (CT) practices in an introductory physics course. The authors first establish a human-coded baseline on N=50 students using a rubric based on the Weintrop et al. taxonomy, then compare the LLM's scores to that baseline, and finally deploy the LLM on the full dataset of N=936 students. Human coding shows significant growth in Data Practices, Computational Problem-Solving, and Systems Thinking, but not in Modeling and Simulation Practices. The LLM is reported to agree substantially with human raters on well-defined constructs, and its full-dataset results are claimed to reproduce the main growth trends. The paper concludes that LLMs offer a viable method for scaling CT assessment in large-enrollment physics courses.

Significance. If the findings hold, this would be a practical and timely contribution to physics education research, addressing a real bottleneck in assessing open-ended written work at scale. The study is grounded in an established CT framework, uses a structured rubric for both human and LLM scoring, employs majority voting over three LLM runs, and reports effect sizes for the human baseline. However, the central validation evidence is weakened by an internal inconsistency in the reported agreement statistics, by the fact that the LLM is validated only on the same sample used to develop the rubric, and by the lack of auditable inferential statistics for the full-dataset claims. The method's promise is clear, but the evidence presented in this version is not yet sufficient to support the headline claim.

major comments (4)
  1. [III.C and Table II] The reported 85% raw agreement for MSP is inconsistent with the means in Table II. For Q27 MSP, the human N=50 pre/post means are 1.62 and 1.62, while the LLM N=50 means are 1.12 and 1.18. Pooling the 100 responses, the human mean is 1.62 and the LLM mean is 1.15, a difference of 0.47. With 85% exact agreement, at most 15 of 100 scores can differ, and each disagreement changes the pooled mean by at most 2/100 = 0.02, so the maximum possible absolute pooled mean difference is 0.30. The observed difference of 0.47 is therefore impossible. One or more of the reported agreement percentage, the kappa, or the Table II means must be incorrect. Because this agreement statistic is the direct evidence for the claim that the LLM 'mirrored' human evaluations, the inconsistency must be resolved before the validation claim can be accepted.
  2. [II.C.2 and IV] The LLM is validated exclusively on the same N=50 sample that was used to develop and refine the rubric, and the full dataset of N=936 students is scored by the LLM without any independent human scoring. The generalization of the N=50 agreement to the full dataset is assumed rather than demonstrated; the distribution of response quality or wording in the remaining 886 students could differ, and the LLM's error rate could be higher on those responses. The authors should either report LLM-human agreement on a holdout sample drawn from the full dataset and not used in rubric development, or explicitly reframe the full-dataset results as exploratory. As written, this is a major gap in the evidence for RQ2.
  3. [III.C and Table II] The full-dataset LLM findings are reported only as means and a single 'p<0.001' statement, with no test statistics, standard deviations, degrees of freedom, or effect sizes. In Section III.C, the claim that the model detected 'highly statistically significant growth (p<0.001) across Q25 and Q26' is not accompanied by any inferential details, and no correction is described for the multiple question-level comparisons presented in Table II. The human baseline in Table I provides t statistics and effect sizes, but the scaled-up analysis does not. Since the paper's central contribution is the scaled-up detection of growth trends, these statistics must be reported so that the claims are auditable.
  4. [II.C.1 and Table I] The 0-2 scores are ordinal categories, but they are analyzed with paired t-tests and Cohen's d, which assume interval-level data. This is a substantive modeling choice because the reported p-values and effect sizes for the human growth claims depend on it. The authors should justify treating the scale as interval or include a robustness check using an ordinal method (e.g., the Wilcoxon signed-rank test) to confirm that the growth findings are not an artifact of the interval assumption.
minor comments (6)
  1. [II.B and Fig. 1] The full text of Q26 is omitted from the manuscript, even though Q26 is one of only three survey questions analyzed in the study. The complete instrument should be included, either in the main text or as a supplement, so readers can evaluate the question and replicate the analysis.
  2. [III.A] The per-construct Fleiss kappa values in Section III.A should specify whether they are computed on the initial N=10 calibration sample or on the final N=50 sample. The current text could be read either way, and the interpretation of the human baseline depends on this distinction.
  3. [III.C] The term 'raw accuracy' is used for the LLM-human agreement, but it is not formally defined. If it means exact match on the 0-2 scale, say so explicitly; if it means adjacent agreement, report exact-match agreement separately, because the kappa values and the means-based checks depend on that definition.
  4. [II.C.2] The full prompt given to the LLM is not included. For reproducibility, the authors should provide the complete structured prompt, including the rubric text, role instructions, and justification requirements, in an appendix or supplementary material.
  5. [II.C.2] The three independent LLM runs are combined by majority vote, but no measure of agreement across the three runs is reported. Reporting the self-agreement (e.g., pairwise kappa among runs) would show how stable the LLM scoring is and whether the majority vote is concealing substantial run-to-run variability.
  6. [References] Reference [34] is listed as a generic ChatGPT citation, but the study uses the GPT-5.4-mini API. The authors should cite the specific model and API version they used, not a generic product page.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the LLM is validated against an independent human-consensus baseline without fitted parameters, so the central claim is not equivalent to its inputs.

full rationale

The paper's derivation chain is self-contained. The human-coded N=50 consensus is an external criterion constructed before the LLM is deployed; the LLM is a fixed pretrained model prompted with the same rubric and then compared with that consensus. No parameter is fitted from the human scores, so the reported agreement metrics (kappa, percent agreement) are empirical outcomes rather than identities forced by construction. The rubric was refined during an initial N=10 calibration round, which is standard instrument development and does not make the later N=50 validation tautological. The full-dataset deployment extrapolates from the validated sample, which is an inductive claim, not a circular one. Self-citations in the reference list support background claims only and are not load-bearing. The internal numerical inconsistency between the reported 85% MSP agreement and the Table II means (human pre/post 1.62/1.62 vs LLM 1.12/1.18) is a serious auditability and correctness concern, but it is not a circularity: it does not show that the LLM scores equal the human scores by definition. Accordingly, no circular step meets the evidentiary bar of Eq. X = Eq. Y or fitted parameter renamed as prediction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or fitted parameters. Its central claim rests on the validity of the rubric and the transferability of LLM agreement from a small calibration set to the full dataset. No free parameters are fitted to the response data, though the rubric thresholds are human-defined.

assumptions (4)
  • domain assumption The Weintrop et al. taxonomy is an appropriate and complete framework for computational thinking in physics
    The paper adopts this taxonomy without discussion as the basis for defining the constructs measured. Section II.A.
  • domain assumption The 0 to 2 ordinal scale reflects meaningful differences in depth of computational thinking
    The scale is treated as an interval measure for paired t-tests and effect sizes without psychometric validation. Section II.C.1.
  • domain assumption Human majority vote on the N=50 calibration set is an error-free ground truth for validating the LLM
    The paper treats the three-rater consensus as definitive without measuring whether consensus codes are correct. Section II.C.1.
  • ad hoc to paper The LLM's agreement with humans on the N=50 subset generalizes to the full N=936 dataset without further validation
    No independent human scoring of the larger sample is performed; the paper assumes the validation result transfers. Section II.C.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Using LLMs to Detect Growth in Computational Thinking in Introductory Physics." pith.science (2026). https://pith.science/paper/JVD3KPUU

@misc{pith2026260806200,
  author       = {Pith},
  title        = {Pith review of: Using LLMs to Detect Growth in Computational Thinking in Introductory Physics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JVD3KPUU}},
  note         = {Machine review of arXiv:2608.06200}
}
read the original abstract

As computation becomes more central to physics education, creating scalable methods to assess authentic computational thinking (CT) in students remains a critical challenge. While student-written responses capture nuanced reasoning, they are difficult to evaluate at scale. In this study, we investigated the use of Large Language Models (LLMs) to analyze students' written explanations of computational physics problems on a pre- and post- semester survey. By first establishing a human-coded baseline, grounded in CT literature, we identified significant growth in Data Practices and Computational Problem-Solving Practices. When given the same responses, an LLM successfully mirrored the human evaluations and scaled up the detection of these key trends across a large dataset. Notably, both human raters and the LLM struggled to reliably evaluate more complex constructs such as Systems Thinking. Overall, this study demonstrates that LLMs offer a viable method to scale the evaluation of students' CT in large-enrollment physics courses

Figures

Figures reproduced from arXiv: 2608.06200 by the authors.

Figure 1
Figure 1. FIG. 1: Two open-ended survey CT questions. Q26 is [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 33 canonical work pages

  1. [1]

    superficial

    Human baseline & rubric development To establish a baseline for evaluating student responses, we developed a specialized rubric designed to measure both physical and computational reasoning, based on Wilensky and Weintrop’s framework [2]. The rubric was iteratively refined through discussion among the researchers to ensure consistent interpretation. Respo...

    work page

  2. [2]

    Be- cause the survey questions rely heavily on visual context (e.g., Python code snippets and system diagrams), GPT-5.4-mini was selected for its advanced multimodal capabilities

    LLM validation & analysis To determine whether generative AI could scale this qual- itative evaluation, we used the GPT-5.4-mini API [34]. Be- cause the survey questions rely heavily on visual context (e.g., Python code snippets and system diagrams), GPT-5.4-mini was selected for its advanced multimodal capabilities. The model’s context window was fed the...

    work page

  3. [3]

    V . J. Shute and J. Sun, C.and Asbell-Clarke, Demystifying computational thinking, Educational Research Review22, 142 (2017)

    work page 2017

  4. [4]

    L. A. Shepard, The role of assessment in a learning culture, Educational Researcher29, 4 (2000)

    work page 2000

  5. [5]

    AAPT Undergraduate Curriculum Task Force,AAPT Recom- mendations for Computational Physics in the Undergradu- ate Physics Curriculum, Tech. Rep. (American Association of Physics Teachers, 2016)

    work page 2016

  6. [6]

    Weintrop, E

    D. Weintrop, E. Beheshti, M. Horn, K. Orton, K. Jona, L. Trouille, and U. Wilensky, Defining computational think- ing for mathematics and science classrooms, Journal of Science Education and Technology25, 127 (2016)

    work page 2016

  7. [7]

    Singh and D

    C. Singh and D. Rosengrant, Multiple-choice test of energy and momentum concepts, American Journal of Physics71, 607 (2003)

    work page 2003

  8. [8]

    W. L. Kuechler and M. G. Simkin, How well do multiple choice tests evaluate student understanding in computer programming classes? (2003)

    2003

Show all 41 references
  1. [9]

    Hestenes, M

    D. Hestenes, M. Wells, and G. Swackhamer, Force concept in- ventory, The Physics Teacher30, 141 (1992)

    1992

  2. [10]

    R. K. Thornton and D. R. Sokoloff, Assessing student learning of newton’s laws: The force and motion conceptual evaluation and the evaluation of active learning laboratory and lecture cur- ricula, American Journal of Physics66, 338 (1998)

    1998

  3. [11]

    Nieswandt and K

    M. Nieswandt and K. Bellomo, Written extended-response questions and the assessment of science learning: What do stu- dents’ answers tell us?, International Journal of Science Edu- cation31, 2117 (2009)

    2009

  4. [12]

    K. L. McNeill and J. S. Krajcik,Supporting Grade 5-8 Stu- dents in Constructing Explanations in Science: The Claim, Ev- idence, and Reasoning Framework for Talk and Writing(Pear- son, 2011)

    2011

  5. [13]

    H. L. Roediger and E. J. Marsh, The positive and negative con- sequences of multiple-choice testing., Journal of Experimental Psychology: Learning, Memory, and Cognition31, 1155–1159 (2005)

    2005

  6. [14]

    N. S. Rebello and D. Zollman, The effect of distracters on stu- dent performance on the force concept inventory, American Journal of Physics72, 116 (2004)

    2004

  7. [15]

    P. G. Butcher and S. E. Jordan, A comparison of human and computer marking of short free-text student responses, Com- puters and Education55, 489 (2010)

    2010

  8. [16]

    Y . Zion, R. E. Zarzecki, J. Glazer, and N. D. Finkelstein, Lever- aging ai for rapid generation of physics simulations in educa- tion: Building your own virtual lab (2024), arXiv:2412.07482 [physics.ed-ph]

    2024 arXiv

  9. [17]

    Kortemeyer, Toward ai grading of student problem solutions in introductory physics: A feasibility study, Physical Review Physics Education Research19, 020163 (2023)

    G. Kortemeyer, Toward ai grading of student problem solutions in introductory physics: A feasibility study, Physical Review Physics Education Research19, 020163 (2023)

    2023

  10. [18]

    Casalino, B

    G. Casalino, B. Cafarelli, E. del Gobbo, L. Fontanella, L. Grilli, A. Guarino, P. Limone, D. Schicchi, and D. Taibi, Framing au- tomatic grading techniques for open-ended questionnaires re- sponses. a short survey (2021)

    2021

  11. [19]

    Kortemeyer and J

    G. Kortemeyer and J. Nohl, Assessing confidence in ai-assisted grading of physics exams through psychometrics: An ex- ploratory study, Phys. Rev. Phys. Educ. Res.21, 010136 (2025)

    2025

  12. [20]

    Savage and N

    S. Savage and N. S. Rebello, Using an LLM to investigate students’ explanations on conceptual physics questions, in Physics Education Research Conference 2025, PER Confer- ence (Washington, DC, 2025) pp. 399–404

    2025

  13. [21]

    B. A. Becker, P. Denny, J. Finnie-Ansley, A. Luxton-Reilly, J. Prather, and E. A. Santos, Programming is hard - or at least it used to be: Educational opportunities and challenges of ai code generation, inProceedings of the 54th ACM Technical Sym- posium on Computer Science Ed...

    2023

  14. [22]

    K. D. Wang, E. Burkholder, C. Wieman, S. Salehi, and N. Haber, Examining the potential and pitfalls of chatgpt in science and engineering problem-solving, inFrontiers in Edu- cation, V ol. 8 (Frontiers Media SA, 2024) p. 1330486

    2024

  15. [23]

    Y . Lee, A. Bralin, N. S. Rebello, and D. Goldwasser, Towards just-in-time adaptive feedback: Enhancing student learning via knowledge-grounded llm, arXiv preprint arXiv:2605.26405 (2026)

    2026 arXiv

  16. [24]

    Allen, A

    W. Allen, A. Shanker, and N. S. Rebello, Students’ perceptions to a large language model’s generated feedback and scores of argumentation essays, inPhysics Education Research Confer- ence 2025, PER Conference (Washington, DC, 2025) pp. 28– 34

    2025

  17. [26]

    N. S. Borse, R. C. Subramaniam, and N. S. Rebello, Investiga- tion of the inter-rater reliability between large language mod- els and human raters in qualitative analysis, inPhysics Educa- tion Research Conference 2025, PER Conference (Washington, DC, 2025) pp. 92–98

    2025

  18. [27]

    /1You are asked to design a physics simulation to model a sled sliding down a grassy hill. What factors would you include when making the simulation in python, and why? 5/6/26, 4:06 PM Print Quiz - Spring 2026 PHYS 17200 - Merge https://purdue.brightspace.com/d2l/le/quizzing/1...

    2026

  19. [28]

    Zhou, S.-M

    L. Zhou, S.-M. Kim, and N. Ahmed, Artificial intelligence ap- plications in education: Natural language processing in detect- ing misconceptions, Education and Information Technologies 10.1007/s10639-024-12919-1 (2024)

    2024 doi

  20. [29]

    S. F. A. Hashmi and N. S. Rebello, Analyzing undergradu- ate problem-solving in physics through interaction with an ai chatbot, inPhysics Education Research Conference 2025, PER Conference (Washington, DC, 2025) pp. 184–189

    2025

  21. [30]

    Wan and Z

    T. Wan and Z. Chen, Exploring generative ai assisted feedback writing for students’ written responses to a physics conceptual question with prompt engineering and few-shot learning, Phys- ical Review Physics Education Research20, 10.1103/phys- revphyseducres.20.010152 (2024)

    2024 doi

  22. [31]

    Latif and X

    E. Latif and X. Zhai, Fine-tuning chatgpt for automatic scoring, arXiv preprint arXiv:2310.10072 (2023)

    2023 arXiv

  23. [32]

    B. Chen, Z. Zhang, N. Langrené, and S. Zhu, Unleashing the potential of prompt engineering in large language models: a comprehensive review (2023), arXiv:2310.14735

    2023 arXiv

  24. [33]

    T. O. B. Odden, E. Lockwood, and M. D. Caballero, Physics computational literacy: An exploratory case study using com- putational essays, Physical Review Physics Education Re- search15, 020152 (2019)

    2019

  25. [34]

    Project Jupyter Community, Project jupyter, an open-source project (2016), https://jupyter.org

    2016

  26. [35]

    A. Rule, A. Birmingham, C. Zuniga, I. Altintas, S.-C. Huang, R. Knight, N. Moshiri, M. H. Nguyen, S. B. Rosenthal, F. Pérez, and P. W. Rose, Ten simple rules for writing and shar- ing computational analyses in jupyter notebooks, PLOS Com- putational Biology15, 1 (2019)

    2019

  27. [36]

    D2L Inc., Brightspace learning management system (2025), accessed May 18, 2025. 6

    2025

  28. [37]

    Fleiss, Measuring nominal scale agreement among many raters, Psychological bulletin76, 378—382 (1971)

    J. Fleiss, Measuring nominal scale agreement among many raters, Psychological bulletin76, 378—382 (1971)

    1971

  29. [38]

    OpenAI, Chatgpt (mar 14 version), https://chat.openai.com/ chat (2025), large language model

    2025

  30. [39]

    J. R. Landis and G. G. Koch, The measurement of observer agreement for categorical data, Biometrics33, 159 (1977)

    1977

  31. [40]

    J. M. Aiken, S.-Y . Lin, S. S. Douglas, E. F. Greco, B. D. Thoms, M. F. Schatz, and M. D. Caballero, The initial state of students taking an introductory physics mooc (2013), arXiv:1307.2533 [physics.ed-ph]

    2013 arXiv

  32. [41]

    M. L. McHugh, Interrater reliability: the kappa statistic, Bio- chemia Medica22, 276 (2012)

    2012

  33. [42]

    Hammer, Student resources for learning introductory physics, American Journal of Physics68, S52 (2000)

    D. Hammer, Student resources for learning introductory physics, American Journal of Physics68, S52 (2000). 7

    2000

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.