Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

With prompt and hyperparameter tuning, GPT-4o and GPT-4.5 can reproduce human qualitative coding for three of four STEM ways-of-thinking themes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

GPT-4o and GPT-4.5-preview reached moderate to substantial Cohen's kappa with human coders for three of four student thinking themes after per-theme hyperparameter and prompt tuning.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A transparent, useful pilot of LLM coding for a STEM Ways of Thinking rubric, but the reported kappa values are selected on the same data they summarize, so the 'substantial agreement' claim is optimistic. the 4 major comments →

arxiv 2508.14764 v2 pith:HP3OGVSN submitted 2025-08-20 physics.ed-ph

Investigation of the Inter-Rater Reliability between Large Language Models and Human Raters in Qualitative Analysis

Nikhil Sanjay Borse , Ravishankar Chatta Subramaniam , N. Sanjay Rebello This is my paper
classification physics.ed-ph
keywords inter-rater reliabilityCohen's kappalarge language modelsqualitative codingprompt engineeringphysics education researchengineering designways of thinking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that modern LLMs, GPT-4o and GPT-4.5-preview, can be tuned to act as reliable second raters in qualitative analysis of student discussion transcripts. On 204 text segments from 14 student groups, the authors compared LLM ratings with two human raters who coded to consensus across four ways-of-thinking themes. After polishing few-shot prompts and selecting temperature and top-p settings per theme, Cohen's kappa exceeded 0.6 for engineering design, physics concepts, and math constructs, and reached 0.55 for metacognitive thinking. The authors conclude that LLMs can scale up qualitative coding in physics education, with human oversight, but that domain-general constructs (themes requiring broad inference, like metacognitive thinking) remain harder to automate.

Core claim

The central claim is that optimizing prompts and model hyperparameters makes GPT-4o and GPT-4.5-preview reliable enough to reproduce expert human consensus coding for three of four themes in an engineering-design physics lab context. Per theme, the authors chose the better of two models and set temperature and top-p by grid search; they used polished few-shot prompts and decomposed the coding task into one binary classification per theme per text segment. The resulting Cohen's kappa values were 0.60 for Engineering Design, 0.70 for Physics Concepts, above 0.6 for Math Constructs, and 0.55 for Metacognitive Thinking. The paper presents these values as evidence that LLMs can serve as scalable

What carries the argument

The mechanism is a tuned LLM-as-rater pipeline: a polished few-shot prompt containing the theme rubric with prototypical example quotes, decomposed into one binary rating per theme per text segment, with per-theme model choice and hyperparameters (temperature and top-p) optimized through repeated API runs. Cohen's kappa, a chance-corrected agreement statistic between two raters, converts the LLM outputs and human consensus labels into a reliability estimate.

Load-bearing premise

The reported kappa values assume that selecting the best model and settings on the same 204 text segments used for evaluation gives a fair, unbiased estimate of how well the LLM would agree with human raters on new transcripts.

What would settle it

Run the optimized prompts and per-theme settings from Table II on a fresh set of engineering-design discussion transcripts, have the same human raters code them to consensus, and compute Cohen's kappa; if the new kappa for engineering design, physics concepts, or math constructs falls below 0.6, the tuned results are specific to the original 204 segments rather than general.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Human coders could use LLMs to pre-code large corpora, focusing their own effort on adjudicating and refining machine labels rather than reading every segment.
  • Domain-specific coding rubrics (physics concepts, math constructs) are likely to transfer best to LLM assistance, while rubrics that rely on broad inference (metacognitive thinking) may need more human involvement.
  • Prompt polishing with short, prototypical examples and per-theme hyperparameter search can raise LLM-human agreement by roughly 0.14 in kappa, so tuning is a meaningful step for future qualitative-coding studies.
  • Model choice matters: GPT-4o handled physics and math constructs best, while GPT-4.5 performed better on engineering design, so a multi-model pipeline may be appropriate.
  • With human oversight, LLM-assisted coding could make qualitative studies of student thinking feasible at scales previously impractical, potentially surfacing new themes in larger datasets.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct validation step left implicit in the paper is to fix the optimized settings on a training subset and then measure kappa on held-out transcripts; this would test how much of the reported agreement depends on tuning to these 204 segments.
  • A natural next test is a hybrid workflow in which the LLM flags low-confidence or borderline segments for human review; the paper's per-theme kappa values suggest the human review load would concentrate on metacognitive-thinking statements.
  • The same prompt-polishing and decomposed-coding recipe could in principle be applied to other qualitative coding rubrics, but the paper's own results imply that transfer will be strongest for rubrics with clear, domain-specific criteria.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper investigates whether LLMs (GPT-4o and GPT-4.5-preview) can reliably reproduce human qualitative coding of student audio transcripts in an engineering-design physics course. Human raters coded 204 text segments by consensus into four themes (ED, PC, MC, MT). The authors engineered few-shot prompts and tuned temperature and top-p per theme through the OpenAI API, then computed Cohen's kappa between each LLM and the human consensus. They report that after optimization, three themes reach substantial agreement (kappa > 0.6) and one (MT) moderate agreement (kappa = 0.55), and conclude that LLMs can scale qualitative analysis with human oversight.

Significance. If the reported agreement estimates are valid, this is a useful proof-of-concept for LLM-assisted qualitative coding in physics education research, with practical implications for scaling thematic analysis. The study is one of few to explicitly explore hyperparameter optimization and prompt polishing for IRR with LLMs, and it provides detailed methodological transparency (decomposed coding, 5-run averaging, per-theme optimization, and a table of optimal settings). The authors also appropriately acknowledge in the Limitations section that hyperparameter optimization may overfit and that the sample is small. However, the central quantitative claim currently rests on an in-sample selection procedure, so the magnitude of the reported kappa values is not yet established as a generalizable estimate.

major comments (4)
  1. [Sec. III, Table II, Fig. 3] The central claim of 'substantial agreement after optimization' is based on per-theme selection of model, temperature, and top-p on the same 204 segments used to compute the final Cohen's kappa. As the authors state in Limitations, this risks overfitting hyperparameters to this dataset. The reported kappa is effectively a maximum over the searched configurations, which is an upward-biased estimate of agreement on new data, especially with per-theme class counts in the tens. To support the abstract's claim, the authors should either use a held-out test set, cross-validation, or nested model selection, and report the resulting kappa values. Without such correction, the three 'substantial' results may not generalize.
  2. [Sec. II, Table I] The few-shot prompt examples appear to be drawn from the same transcripts used for evaluation. The Methods state that after testing on text segments from three of the 14 transcripts, a few-shot prompt with 'human-labeled example quotes' was adopted, and the evaluation uses '204 text segments excluding the example segments in the prompt.' If any example quotes come from the same transcripts or conversation context, the model may be exposed to near-exact language from the evaluation set, inflating kappa. The authors should clarify the provenance of example quotes and, if they come from the evaluation corpus, quantify how many of the 204 segments overlap or are closely paraphrased. This is load-bearing for the reported reliability.
  3. [Sec. II, Sec. III] No confidence intervals are reported for Cohen's kappa, and no human-human kappa is reported. With 204 segments and likely imbalanced positive/negative counts per theme, the standard error of kappa is nontrivial; a point estimate of 0.60 or 0.70 may not significantly exceed 0.6. Moreover, since the two human raters coded to consensus, the reader cannot tell whether the LLM approaches human-level agreement. Reporting bootstrapped or analytic confidence intervals, and ideally a pre-consensus human-human kappa on a subset, would substantially strengthen the conclusions.
  4. [Sec. IV, 'Conclusion & Implications'] The claim that optimization improved kappa by 0.14 on average and that this is 'statistically significant (p < 0.02) using a Mann-Whitney test' is not fully specified. It is unclear what units are compared in the Mann-Whitney test (per-theme mean kappas? 5 runs per theme?), how many observations enter the test, and whether the comparison accounts for the fact that the same segments are used for both default and optimized conditions. This significance statement should be clarified or removed.
minor comments (4)
  1. [General] A few formatting and citation issues: reference [36] has a large author list that could be compressed; reference [42] is used for the methodological flow figure but may not be the most specific source; and the abstract/body use 'Kappa' and 'κ' inconsistently. Please standardize.
  2. [Fig. 4] The four panels of Fig. 4 are informative but the axes and annotations are small; consider enlarging or adding a caption that explicitly states the numbers of runs and the error bars (if any).
  3. [Sec. II, last paragraph] The sentence 'There were 204 text segments excluding the example segments in the prompt' is ambiguous. It should state whether the example segments were part of the original corpus and were removed, or whether 204 is the total after removing them. This matters for the leakage concern.
  4. [Sec. V, Limitations] The limitation about 'small subset of the data' is useful, but the authors should also mention the absence of a human-human reliability baseline as a limitation, rather than leaving it implicit.

Circularity Check

1 steps flagged

Reported κ values are the optimized objective on the same 204 segments, so 'substantial agreement' is partly a fitted quantity.

specific steps
  1. fitted input called prediction [Sec. II (Methods), paragraph beginning 'To find agreement...'; Sec. III (Findings), Fig. 3 and Table II]
    "We optimized the LLMs for performance. Model hyperparameters like temperature and top-p, [45] were fine-tuned."

    The same 204 text segments are used both to select each theme's model, temperature, and top-p (Table II) and to compute the final Cohen's κ (Fig. 3). The selection criterion is Cohen's κ itself, so the reported κ for each theme is the maximum over the searched grid of (model × temperature × top-p) on the exact data used for selection. Thus the headline 'substantial agreement (κ > 0.6) for three themes' is the optimized objective, not an independent out-of-sample estimate; the few-shot prompt examples are also human-labeled quotes from the same coding framework/transcripts, adding further leakage. The paper's own limitation states 'the optimization of models might be overfitting the hyperparameters to our dataset,' confirming that the final κ is partly a fitted quantity rather than a genera

full rationale

The paper's derivation chain is empirical, not mathematical: 204 transcript segments are coded by humans, then LLMs rate the same segments under various prompts and hyperparameters. The reported agreement is Cohen's κ after per-theme selection of model, temperature, and top-p on the full dataset. Because the final κ is the maximum over the searched configurations on the same 204 segments, it is by construction an optimistic in-sample value; the 'substantial agreement' conclusion is not a prediction of performance on new data. The default-vs-optimized comparison and the per-theme differences are still informative, and the paper honestly flags the overfitting risk in its Limitations. However, the central claim that LLMs 'can be a reliable tool' rests on these optimized in-sample κ values, so the quantitative headline reduces, in part, to the selection procedure. This matches the fitted-input-called-prediction pattern. There is no meaningful self-citation circularity or definitional equivalence: the human coding framework is external, and the κ computation is standard. Score 6 reflects partial circularity: the reported agreement is the optimized objective, not an independent measure.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new physical or conceptual entities are introduced. The free parameters are the per-theme hyperparameters and model choices that were fitted to the evaluation data. The assumptions are standard domain assumptions for this kind of qualitative reliability study.

free parameters (3)
  • temperature = ED: 1.1, MT: 1.1, MC: 0.9, PC: 0.9
    Tuned per theme on the 204 evaluation segments to maximize Cohen's kappa (Table II). This is a fitted value, not an input from theory.
  • top-p = ED: 0.8, MT: 0.8, MC: 0.9, PC: 0.9
    Tuned per theme on the 204 evaluation segments to maximize Cohen's kappa (Table II).
  • model choice = ED: GPT-4.5; MT, MC, PC: GPT-4o
    Selected per theme based on which model gave higher observed kappa (Fig 4 and Table II). This categorical choice is another free parameter fitted to the data.
axioms (4)
  • domain assumption Two human raters coding to consensus provide a valid and error-free gold standard for each segment.
    Section II states coders coded to consensus; no human-human inter-rater reliability is reported, so the gold standard is assumed.
  • domain assumption The WoT rubric from [39] applies cleanly to the segmented audio transcripts in this physics course.
    Section II says transcripts were coded according to the framework from [39]; rubric applicability is taken from prior work.
  • domain assumption 204 text segments from 14 groups in one lab section are sufficient for stable per-theme Cohen's kappa estimates.
    The paper reports only 5-run means without confidence intervals, so stability is assumed.
  • domain assumption Manually cleaned transcripts represent the group discussion despite low audio quality.
    Section II notes the low audio quality and that transcripts are representative of the group, an assumption about data quality.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigation of the Inter-Rater Reliability between Large Language Models and Human Raters in Qualitative Analysis." pith.science (2026). https://pith.science/paper/HP3OGVSN

@misc{pith2026250814764,
  author       = {Pith},
  title        = {Pith review of: Investigation of the Inter-Rater Reliability between Large Language Models and Human Raters in Qualitative Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HP3OGVSN}},
  note         = {Machine review of arXiv:2508.14764}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Qualitative analysis is typically limited to small datasets because it is time-intensive. Moreover, a second human rater is required to ensure reliable findings. Artificial intelligence tools may replace human raters if we demonstrate high reliability compared to human ratings. We investigated the inter-rater reliability of state-of-the-art Large Language Models (LLMs), ChatGPT-4o and ChatGPT-4.5-preview, in rating audio transcripts coded manually. We explored prompts and hyperparameters to optimize model performance. The participants were 14 undergraduate student groups from a university in the midwestern United States who discussed problem-solving strategies for a project. We prompted an LLM to replicate manual coding, and calculated Cohen's Kappa for inter-rater reliability. After optimizing model hyperparameters and prompts, the results showed substantial agreement (${\kappa}>0.6$) for three themes and moderate agreement on one. Our findings demonstrate the potential of GPT-4o and GPT-4.5 for efficient, scalable qualitative analysis in physics education and identify their limitations in rating domain-general constructs.

Figures

Figures reproduced from arXiv: 2508.14764 by Nikhil Sanjay Borse, N. Sanjay Rebello, Ravishankar Chatta Subramaniam.

Figure 1
Figure 1. Figure 1: FIG. 1: Methodological Flow: Inter-Rater Reliability [ [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: FIG. 3: Improvement in rater agreement for ED, PC, MC, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Me and My Bot: What Users Talk About in AI Companion Communities on Reddit

    cs.HC 2026-08 conditional novelty 5.0

    Across eight AI companion subreddits, only about 45% of retrieved posts concerned the user's own bot, own-bot posts were markedly more emotional than general-bot posts, and companionship and romance dominated relation...

Reference graph

Works this paper leans on

52 extracted references · 45 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    National Research Council, Next Generation Science Stan- dards: For States, By States (The National Academies Press, Washington, DC, 2013)

    2013

  2. [2]

    To find agreement between human raters and LLMs, we FIG

    The prompts for the other three themes followed the same structure and were polished likewise. To find agreement between human raters and LLMs, we FIG. 2: Example of a polished few-shot prompt. performed an IRR by calculating Cohen’s κ for each theme. There were 204 text segments excluding the example seg- ments in the prompt. To classify them, we used Op...

    work page

  3. [3]

    Honey, G

    M. Honey, G. Pearson, and E. Schweingruber, Heidi,STEM In- tegration in K-12 Education: Status, Prospects, and an Agenda for Research (National Academies Press, Washington, DC, 2014)

    work page 2014

  4. [4]

    Fischer, C

    F. Fischer, C. Wecker, A. Hetmanek, J. Osborne, C. A. Chinn, R. G. Duncan, R. W. Rinhart, S. A. Siler, D. Klahr, and W. A. Sandoval, The interplay of domain-specific and domain- general factors in scientific reasoning and argumentation, in Proceedings of the International Conference of the Learning Sciences (ICLS) 2014 , V ol. 3, edited by J. L. Polman, E...

    work page 2014

  5. [5]

    N. R. Council, D. of Behavioral, S. Sciences, B. on Science Ed- ucation, and C. on a Conceptual Framework for New K-12 Sci- ence Education Standards, A framework for K-12 science edu- cation: Practices, crosscutting concepts, and core ideas (Na- tional Academies Press, 2012)

    work page 2012

  6. [6]

    Dalal, A

    M. Dalal, A. Carberry, and L. Archambault, Developing a ways of thinking framework for engineering education research, Studies in Engineering Education 1, 108 (2021)

    work page 2021

  7. [7]

    Slavit, E

    D. Slavit, E. Grace, and K. Lesseig, Stem ways of think- ing, North American Chapter of the International Group for the Psychology of Mathematics Education (2019), conference proceedings

    work page 2019

  8. [8]

    Slavit, E

    D. Slavit, E. Grace, and K. Lesseig, Student ways of thinking in stem contexts: A focus on claim making and reasoning, School Science and Mathematics 121, 466 (2021)

    work page 2021

  9. [9]

    Y .-C. N. Lien, W.-J. Wu, and Y .-L. Lu, How well do teach- ers predict students’ actions in solving an ill-defined problem in stem education: a solution using sequential pattern mining, IEEE Access 8, 134976 (2020)

    work page 2020

  10. [10]

    L. D. English, Ways of thinking in stem-based problem solv- ing, ZDM–Mathematics Education 55, 1219 (2023)

    work page 2023

  11. [11]

    Talanquer and J

    V . Talanquer and J. Pollard, Let’s teach how we think instead of what we know, Chemistry Education Research and Practice 11, 74 (2010)

    work page 2010

  12. [12]

    M. J. Luna, S. J. Selmer, and J. A. Rye, Teachers’ noticing of students’ thinking in science through classroom artifacts: In what ways are science and engineering practices evident?, Journal of Science Teacher Education 29, 148 (2018)

    work page 2018

  13. [13]

    Wilkinson, A

    I. Wilkinson, A. Soter, and P. Murphy, Developing a model of quality talk about literary text, in Bringing Reading Research to Life: Essays in Honor of Isabel L. Beck, edited by M. McK- eown and L. Kucan (The Guilford Press, New York, NY , 2010) pp. 142–169

    work page 2010

  14. [14]

    Etkina and G

    E. Etkina and G. Planinši ˇc, Thinking like a scientist, Physics world 27, 48 (2014)

    work page 2014

  15. [15]

    C. M. Firetto, E. Starrett, and M. E. Jordan, Embracing a cul- ture of talk: Stem teachers’ engagement in small-group dis- cussions about photovoltaics, International Journal of STEM Education 10, 50 (2023)

    work page 2023

  16. [16]

    N. K. Denzin and Y . S. Lincoln, Handbook of qualitative re- search, Journal of Leisure Research 28, 132 (1996)

    work page 1996

  17. [17]

    Erickson et al., Qualitative methods in research on teaching (Institute for Research on Teaching East Lansing, MI, 1985)

    F. Erickson et al., Qualitative methods in research on teaching (Institute for Research on Teaching East Lansing, MI, 1985)

    work page 1985

  18. [18]

    Braun and V

    V . Braun and V . Clarke, Using thematic analysis in psychology, Qualitative Research in Psychology 3, 77 (2006)

    work page 2006

  19. [19]

    J. W. Creswell and C. N. Poth,Qualitative inquiry and research design: Choosing among five approaches (Sage publications, 2016)

    work page 2016

  20. [20]

    J. M. Morse, Critical analysis of strategies for determining rigor in qualitative inquiry, Qualitative health research25, 1212 (2015)

    work page 2015

  21. [21]

    Saldana, The coding manual for qualitative researchers , V ol

    J. Saldana, The coding manual for qualitative researchers , V ol. 3 (Sage Publications, Thousand Oaks, CA, 2009)

    work page 2009

  22. [22]

    Houghton, K

    C. Houghton, K. Murphy, D. Shaw, and D. Casey, Qualitative case study data analysis: An example from practice, Nurse re- searcher 22 (2015)

    work page 2015

  23. [23]

    Jackson and P

    K. Jackson and P. Bazeley, Qualitative data analysis with NVivo (SAGE Publications Ltd, 2019)

    work page 2019

  24. [24]

    Uysal and N

    ˙I. Uysal and N. Do˘gan, How reliable is it to automatically score open-ended items? an application in the turkish language, Jour- nal of Measurement and Evaluation in Education and Psychol- ogy 12, 28 (2021)

    work page 2021

  25. [25]

    M. F. Turgut and Y . Baykul,E˘ gitimde ölçme ve de˘ gerlendirme, V ol. 2 (Pegem Akademi, 2010)

    work page 2010

  26. [26]

    L. R. Aiken, Psychological testing and assessment (Pearson Education India, 2009)

    work page 2009

  27. [27]

    E. A. Siverling, T. J. Moore, E. Suazo-Flores, C. A. Mathis, and S. S. Guzey, What initiates evidence-based reasoning?: Situ- ations that prompt students to support their design ideas and decisions, Journal of Engineering Education 110, 294 (2021)

    2021

  28. [28]

    H. E. Tinsley and D. J. Weiss, Interrater reliability and agree- ment of subjective judgments., Journal of Counseling Psychol- ogy 22, 358 (1975)

    work page 1975

  29. [29]

    L. A. Siiman, M. Rannastu-Avalos, J. Pöysä-Tarhonen, P. Häkkinen, and M. Pedaste, Opportunities and challenges for ai-assisted qualitative data analysis: An example from collab- orative problem-solving discourse data, in International Con- ference on Innovative Technologies and Learning (Springer,

    work page

  30. [30]

    A. Katz, S. Wei, G. Nanda, C. Brinton, and M. Ohland, Ex- ploring the efficacy of chatgpt in analyzing student team- work feedback with an existing taxonomy, arXiv preprint arXiv:2305.11882 (2023)

    Pith/arXiv arXiv 2023

  31. [31]

    Hitch, Artificial intelligence augmented qualitative analysis: the way of the future?, Qualitative Health Research 34, 595 (2024)

    D. Hitch, Artificial intelligence augmented qualitative analysis: the way of the future?, Qualitative Health Research 34, 595 (2024)

    work page 2024

  32. [32]

    Tabone and J

    W. Tabone and J. De Winter, Using chatgpt for human– computer interaction research: a primer, Royal Society Open Science 10, 231053 (2023)

    work page 2023

  33. [33]

    Zhang, C

    H. Zhang, C. Wu, J. Xie, Y . Lyu, J. Cai, and J. M. Carroll, Re- defining qualitative analysis in the ai era: Utilizing chatgpt for efficient thematic analysis, arXiv preprint arXiv:2309.10771 (2023)

    Pith/arXiv arXiv 2023

  34. [34]

    Z. O. Dunivin, Scaling hermeneutics: a guide to qualitative coding with llms for reflexive content analysis, EPJ Data Sci- ence 14, 28 (2025)

    work page 2025

  35. [35]

    X. Liu, A. F. Zambrano, R. S. Baker, A. Barany, J. Ocumpaugh, J. Zhang, M. Pankiewicz, N. Nasiar, and Z. Wei, Qualitative coding with gpt-4: Where it works better., Journal of Learning Analytics 12, 169 (2025)

    work page 2025

  36. [36]

    Z. Xiao, X. Yuan, Q. V . Liao, R. Abdelghani, and P.-Y . Oudeyer, Supporting qualitative analysis with large language models: Combining codebook with gpt-3 for deductive coding, in Companion proceedings of the 28th international conference on intelligent user interfaces (2023) pp. 75–78

    work page 2023

  37. [37]

    Achiam, S

    OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Alt- man, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V . Bal- com, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Br...

    Pith/arXiv arXiv 2024

  38. [38]

    OpenAI, Gpt-4o technical report, https://openai.com/index/ gpt-4o (2024), accessed 2025-05-25

    work page 2024

  39. [39]

    T. B. Brown et al. , Language models are few-shot learners, NeurIPS (2020)

    work page 2020

  40. [40]

    R. C. Subramaniam, C. LaFontaine, A. Bralin, J. W. Morphew, C. M. Rebello, and N. S. Rebello, Characterising stem ways of thinking in engineering design (ed)-based tasks, in2024 PERC Proceedings [Boston, MA, July 10-11, 2024] , edited by Q. X. Ryan, A. Pawl, and J. P. Zwolak (2024)

    work page 2024

  41. [41]

    R. C. Subramaniam, N. Borse, A. Bralin, J. W. Morphew, C. M. Rebello, and N. S. Rebello, Investigating the design-science connection in a multiweek engineering design-based introduc- tory physics laboratory task, Phys. Rev. Phys. Educ. Res. 21, 010118 (2025)

    work page 2025

  42. [42]

    R. C. Subramaniam, N. Borse, W. Allen, A. Sirnoorkar, J. W. Morphew, C. M. Rebello, and N. S. Rebello, Applying a stem ways of thinking framework for student-generated engineer- ing design-based physics problems (2025), arXiv:2503.05957 [physics.ed-ph]

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  43. [43]

    Bijker, S

    R. Bijker, S. S. Merkouris, N. A. Dowling, and S. N. Rodda, Chatgpt for automated qualitative research: content analysis, Journal of medical Internet research 26, e59050 (2024)

    work page 2024

  44. [44]

    E. G. Guba, Y . S. Lincoln, et al. , Competing paradigms in qualitative research, Handbook of qualitative research 2, 105 (1994)

    work page 1994

  45. [45]

    Mizumoto and M

    A. Mizumoto and M. F. Teng, Large language models fall short in classifying learners’ open-ended responses, Research Meth- ods in Applied Linguistics 4, 100210 (2025)

    work page 2025

  46. [46]

    B. Chen, Z. Zhang, N. Langrené, and S. Zhu, Unleashing the potential of prompt engineering in large language mod- els: a comprehensive review, arXiv preprint arXiv:2310.14735 (2023)

    Pith/arXiv arXiv 2023

  47. [47]

    J. R. Landis and G. G. Koch, An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers, Biometrics , 363 (1977)

    work page 1977

  48. [48]

    S. Xu, X. Huang, C. K. Lo, G. Chen, and M. S.-y. Jong, Eval- uating the performance of chatgpt and gpt-4o in coding class- room discourse data: A study of synchronous online mathe- matics instruction, Computers and Education: Artificial Intel- ligence 7, 100325 (2024)

    work page 2024

  49. [49]

    P. E. McKnight and J. Najab, Mann-whitney u test, The Corsini encyclopedia of psychology , 1 (2010)

    work page 2010

  50. [50]

    D. H. Jonassen, Learning to solve problems : A handbook for designing problem-solving learning environments (2010)

    work page 2010

  51. [51]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. , Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501.12948 (2025)

    Pith/arXiv arXiv 2025

  52. [52]

    Tschisgale, P

    P. Tschisgale, P. Wulff, and M. Kubsch, Integrating artificial intelligence-based methods into qualitative research in physics education research: A case for computational grounded theory, Phys. Rev. Phys. Educ. Res. 19, 020123 (2023)

    work page 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.