The reviewed record of science sign in
Pith

arxiv: 2606.28872 · v1 · pith:6OOCGJMR · submitted 2026-06-27 · physics.ed-ph

Testing the Validity of Embedding-Based Similarity and Clustering for Handwritten Physics Solutions

Maike Tauschhuber , Gerd Kortemeyer This is my paper

Reviewed by Pith2026-06-30 08:38 UTCgrok-4.3pith:6OOCGJMRopen to challenge →

classification physics.ed-ph
keywords text embeddingsphysics educationstudent solutionsclusteringsimilaritygradinghandwritten exams
0
0 comments X

The pith

Text embeddings of physics exam solutions show only modest alignment with human grades and prioritize surface features.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether the geometry of embedding spaces for student physics solutions preserves the distinctions that matter for grading. It does so by transcribing nearly 1000 handwritten thermodynamics solutions into multiple text forms, embedding them with nine different models, and directly comparing embedding similarity and hierarchical clusters against human-assigned scores. A sympathetic reader would care because embeddings offer a cheap way to organize large collections of written work, yet if their similarity relations do not track conceptual or scoring differences, they cannot support unsupervised grading. The experiments indicate that embeddings behave like novices, with geometry strongly shaped by surface features rather than semantic structure.

Core claim

Across models, representations, and clustering choices, embedding similarity showed a consistent but modest relationship to score similarity, and the resulting clusters were score-enriched but not score-equivalent. Experiments with a synthetic data set suggest that this may be due to embeddings behaving like novices when categorizing physics-problem solutions, that is, their similarity geometry is strongly influenced by surface features rather than conceptual, semantic structure.

What carries the argument

Direct comparison of embedding-space similarity and hierarchical clusters against human-assigned scores on transcribed handwritten solutions.

If this is right

  • Embeddings can support exploratory organization of collections of physics solutions.
  • They enable human-in-the-loop review processes.
  • They do not provide an unsupervised basis for grading without external validation against the assessment construct of interest.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The novice-like categorization pattern could be checked by comparing embedding clusters directly to clusters produced by novice human raters on the same solutions.
  • Similar surface-feature dominance may appear when embeddings are applied to student work in other STEM subjects.
  • Fine-tuning embeddings on labeled physics education data might shift the geometry toward conceptual rather than surface distinctions.

Load-bearing premise

Human-assigned scores mark the grading-relevant distinctions that embeddings should be expected to preserve.

What would settle it

An embedding model that produces clusters whose score distributions match the human score groups exactly on a held-out set of similar physics solutions without any additional validation against the assessment construct.

Figures

Figures reproduced from arXiv: 2606.28872 by Gerd Kortemeyer, Maike Tauschhuber.

Figure 1
Figure 1. Figure 1: FIG. 1. Schematic illustration of the clustering-validity crite [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: FIG. 2. Example of student work on the exam. The black [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: FIG. 3. Prompt for mode [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: FIG. 4. Transcription in mode [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: FIG. 5. Prompt for mode [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: FIG. 6. Transcription in mode [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: FIG. 7. Prompt for mode [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: FIG. 8. Transcription in mode [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: FIG. 9. Prompt for mode [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: FIG. 10. Transcription in mode [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: FIG. 11. Prompt for mode [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: FIG. 12. Transcription in mode [PITH_FULL_IMAGE:figures/full_fig_p015_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: FIG. 13. Relationship between embedding similarity and human-score difference. For each transcription condition, unordered [PITH_FULL_IMAGE:figures/full_fig_p018_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: FIG. 14. Force-directed Fruchterman-Reingold plot of the cosine similarity of the embeddings of the statements in Table V. [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: FIG. 15. Force-directed Fruchterman-Reingold plot of the centered cosine similarity of the embeddings of the statements in [PITH_FULL_IMAGE:figures/full_fig_p024_15.png] view at source ↗
read the original abstract

Text embeddings are increasingly used in physics education research to organize, compare, and cluster large collections of written text. Their appeal is clear: once student responses have been mapped into a vector space, similarity comparisons and clustering become computationally inexpensive. However, in assessment contexts, the relevant question is not merely whether clusters can be produced, but whether the geometry of the embedding space preserves grading-relevant distinctions. We tested this premise using 992 handwritten student-problem solutions from a high-stakes engineering thermodynamics exam, transcribed into five textual representations and embedded using nine embedding mechanisms. We compared embedding similarity and embedding-based hierarchical clusters against human-assigned scores. Across models, representations, and clustering choices, embedding similarity showed a consistent but modest relationship to score similarity, and the resulting clusters were score-enriched but not score-equivalent. Experiments with a synthetic data set suggest that this may be due to embeddings behaving like novices when categorizing physics-problem solutions, that is, their similarity geometry is strongly influenced by surface features rather than conceptual, semantic structure. These findings suggest that state-of-the-art embeddings can support exploratory organization and human-in-the-loop review of physics solutions, but they do not provide an unsupervised basis for grading without external validation against the assessment construct of interest.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 3 minor

Summary. The paper empirically tests whether text embeddings preserve grading-relevant distinctions in handwritten physics solutions by comparing embedding similarities and hierarchical clusters to human-assigned scores across nine embedding models, five textual representations, and 992 solutions from an engineering thermodynamics exam. It reports modest correlations and score-enriched but not equivalent clusters, uses synthetic data to suggest surface-feature sensitivity, and concludes that embeddings are suitable for exploratory organization and human-in-the-loop review but not for unsupervised grading without external validation against the assessment construct.

Significance. If the findings are robust, this work offers valuable evidence on the practical limitations of current embedding technologies in physics education assessment contexts. The large-scale real-world dataset combined with synthetic controls provides a concrete basis for cautioning against over-reliance on unsupervised embedding methods for grading, while supporting their use in exploratory analysis. This contributes to the growing literature on AI tools in education by grounding claims in specific empirical comparisons.

major comments (2)
  1. [Abstract and Results] Abstract and Results: The description of a 'consistent but modest relationship' between embedding similarity and score similarity, and 'score-enriched but not score-equivalent' clusters, lacks specific quantitative metrics such as correlation coefficients (e.g., Pearson r values), confidence intervals, p-values, or exact clustering parameters (linkage method, distance metric, dendrogram cut height). Without these, the strength and consistency of the central empirical patterns cannot be fully assessed.
  2. [Discussion] Discussion: The claim that embeddings behave like novices by prioritizing surface features over conceptual structure, leading to mismatch with grading-relevant distinctions, rests on human scores as the ground truth proxy. No independent check (such as expert annotation of conceptual vs. surface distinctions on the real data or inter-rater reliability of scores) is reported to confirm that the scores capture the intended assessment construct rather than partial-credit or other artifacts. This makes the interpretation of the mismatch as embedding limitation rather than proxy limitation vulnerable.
minor comments (3)
  1. [Methods] Clarify the exact transcription process for the five textual representations and any preprocessing steps applied before embedding.
  2. [Figures] Ensure all figures include error bars or variability measures if multiple runs or models are compared, and label axes clearly with the specific similarity metric used.
  3. [References] Add citations to prior work on embedding use in education research for context on novelty.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which highlight opportunities to strengthen the quantitative reporting and clarify the evidential basis for our interpretations. We address each major comment below.

read point-by-point responses

  1. Referee: [Abstract and Results] Abstract and Results: The description of a 'consistent but modest relationship' between embedding similarity and score similarity, and 'score-enriched but not score-equivalent' clusters, lacks specific quantitative metrics such as correlation coefficients (e.g., Pearson r values), confidence intervals, p-values, or exact clustering parameters (linkage method, distance metric, dendrogram cut height). Without these, the strength and consistency of the central empirical patterns cannot be fully assessed.

    Authors: We agree that the absence of explicit numerical metrics limits the reader's ability to evaluate the reported patterns. In the revised manuscript we will add Pearson r values (with 95% confidence intervals and p-values) for the correlation between embedding cosine similarities and score differences, computed separately for each of the nine models and five representations. We will also document the hierarchical clustering details: Ward linkage, cosine distance, and the specific height thresholds or inconsistency coefficients used to obtain the reported cluster partitions. revision: yes

  2. Referee: [Discussion] Discussion: The claim that embeddings behave like novices by prioritizing surface features over conceptual structure, leading to mismatch with grading-relevant distinctions, rests on human scores as the ground truth proxy. No independent check (such as expert annotation of conceptual vs. surface distinctions on the real data or inter-rater reliability of scores) is reported to confirm that the scores capture the intended assessment construct rather than partial-credit or other artifacts. This makes the interpretation of the mismatch as embedding limitation rather than proxy limitation vulnerable.

    Authors: The scores were produced by the course instructors using a rubric that explicitly targets conceptual understanding rather than surface features. Nevertheless, we did not collect inter-rater reliability statistics or perform independent expert coding of conceptual versus surface distinctions on the 992 solutions. In revision we will add an explicit limitations paragraph acknowledging this reliance on a single set of human scores and will emphasize that the synthetic-data experiments supply an independent line of evidence for surface-feature sensitivity. We will also note that future validation could include multi-rater reliability or targeted conceptual annotation. revision: partial

Circularity Check

0 steps flagged

No circularity: direct empirical comparison to external human scores

full rationale

The paper conducts an empirical study comparing embedding similarities and hierarchical clusters directly against human-assigned scores on 992 transcribed solutions, using nine embedding models and multiple representations. No derivations, fitted parameters, self-referential predictions, or load-bearing self-citations are present. The modest correlation findings and cluster analyses are measured against an independent external benchmark (human scores), making the central claim self-contained rather than reducing to its inputs by construction. This is the expected outcome for a validation study without mathematical modeling or ansatz smuggling.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on human scores serving as the authoritative measure of grading-relevant distinctions and on transcriptions faithfully capturing solution content; no free parameters or invented entities are introduced.

axioms (2)
  • domain assumption Human-assigned scores represent the assessment construct of interest that embeddings should preserve.
    Invoked when defining the relevant question for assessment contexts (abstract).
  • domain assumption Transcribed textual representations preserve the information needed for semantic comparison.
    Implicit in the choice to test five textual representations of handwritten solutions.

pith-pipeline@v0.9.1-grok · 5753 in / 1266 out tokens · 25172 ms · 2026-06-30T08:38:37.541752+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

102 extracted references · 3 canonical work pages · 2 internal anchors

  1. [1]

    (1)". Two horizontal segments are labeled

    Similarity Let xi ∈ Rd denote the embedding vector for student response i within a fixed problem, transcription condi- tion, and embedding mechanism. We computed two sim- ilarity matrices. The first was ordinary cosine similarity, scos ij = xi · xj ∥xi∥∥xj∥ . (1) 11 PROBLEM_LABELS: - 4 a) - b) - c) - d) - e) FORMULA_NARRATIVE: For part a), the student dra...

  2. [2]

    explain the 21 work done by a Carnot engine,

    Clustering For the clustering analysis, each similarity matrix S was converted to a distance matrix Dij = 1 − Sij. (5) We then performed agglomerative hierarchical clustering using average linkage and complete linkage. These two linkage methods represent different assumptions about what should count as a coherent group: average linkage merges clusters bas...

  3. [3]

    J. H. Larkin and F. Reif, Understanding and teaching problem-solving in physics, European journal of science education 1, 191 (1979)

    1979

  4. [4]

    Gok, The general assessment of problem solving pro- cesses and metacognition in physics education, Interna- tional Journal of Physics and Chemistry Education 2, 110 (2010)

    T. Gok, The general assessment of problem solving pro- cesses and metacognition in physics education, Interna- tional Journal of Physics and Chemistry Education 2, 110 (2010)

    2010

  5. [5]

    J. L. Docktor and J. P. Mestre, Synthesis of discipline- based education research in physics, Phys. Rev. ST Phys. Educ. Res. 10, 020119 (2014)

    2014

  6. [6]

    Ince, An overview of problem solving studies in physics education., Journal of Education and Learning 7, 191 (2018)

    E. Ince, An overview of problem solving studies in physics education., Journal of Education and Learning 7, 191 (2018)

    2018

  7. [7]

    R. J. Dufresne and W. J. Gerace, Assessing-to- learn: Formative assessment in physics instruction, The Physics Teacher 42, 428 (2004)

    2004

  8. [8]

    J. T. Laverty, W. Bauer, G. Kortemeyer, and G. West- fall, Want to reduce guessing and cheating while making students happier? give more exams!, Phys. Teach. 50, 540 (2012)

    2012

  9. [9]

    Fakcharoenphol and T

    W. Fakcharoenphol and T. Stelzer, Physics exam prepa- ration: A comparison of three methods, Physical Re- view Special Topics – Physics Education Research 10, 010108 (2014)

    2014

  10. [10]

    Wieman and K

    C. Wieman and K. Perkins, Transforming physics edu- cation, Physics today 58, 36 (2005)

    2005

  11. [11]

    Uhden, R

    O. Uhden, R. Karam, M. Pietrocola, and G. Pospiech, Modelling mathematical reasoning in physics education, Science & Education 21, 485 (2012)

    2012

  12. [12]

    Tuminaro and E

    J. Tuminaro and E. F. Redish, Elements of a cogni- tive model of physics problem solving: Epistemic games, Phys. Rev. ST Phys. Educ. Res. 3, 020101 (2007)

    2007

  13. [13]

    R. E. Teodorescu, C. Bennhold, G. Feldman, and L. Medsker, New approach to analyzing physics prob- lems: A taxonomy of introductory physics problems, Phys. Rev. ST Phys. Educ. Res. 9, 010103 (2013)

    2013

  14. [14]

    Singh, Assessing student expertise in introductory physics with isomorphic problems

    C. Singh, Assessing student expertise in introductory physics with isomorphic problems. ii. effect of some po- tential factors on problem solving and transfer, Phys. Rev. ST Phys. Educ. Res. 4, 010105 (2008). 25

    2008

  15. [15]

    Larkin, J

    J. Larkin, J. McDermott, D. P. Simon, and H. A. Si- mon, Expert and novice performance in solving physics problems, Science 208, 1335 (1980)

    1980

  16. [16]

    M. M. Hull, E. Kuo, A. Gupta, and A. Elby, Problem- solving rubrics revisited: Attending to the blending of informal conceptual and formal mathematical reason- ing, Phys. Rev. ST Phys. Educ. Res. 9, 010105 (2013)

    2013

  17. [17]

    J. L. Docktor, J. Dornfeld, E. Frodermann, K. Heller, L. Hsu, K. A. Jackson, A. Mason, Q. X. Ryan, and J. Yang, Assessing student written problem solutions: A problem-solving rubric with application to introductory physics, Phys. Rev. Phys. Educ. Res.12, 010130 (2016)

    2016

  18. [18]

    D. E. Meltzer, Relation between students’ problem- solving performance and representational format, Amer- ican journal of physics 73, 463 (2005)

    2005

  19. [19]

    P. B. Kohl and N. D. Finkelstein, Student represen- tational competence and self-assessment when solving physics problems, Phys. Rev. ST Phys. Educ. Res. 1, 010104 (2005)

    2005

  20. [20]

    P. B. Kohl, D. Rosengrant, and N. D. Finkelstein, Strongly and weakly directed approaches to teaching multiple representation use in physics, Phys. Rev. ST Phys. Educ. Res. 3, 010108 (2007)

    2007

  21. [21]

    Nguyen and N

    D.-H. Nguyen and N. S. Rebello, Students’ difficulties in transfer of problem solving across representations, in AIP conference Proceedings, Vol. 1179 (American Insti- tute of Physics, 2009) pp. 221–224

    2009

  22. [22]

    B. R. Wilcox and S. J. Pollock, Coupled multiple- response versus free-response conceptual assessment: An example from upper-division physics, Physical Re- view Special Topics-Physics Education Research 10, 020124 (2014)

    2014

  23. [23]

    F. Reif, J. H. Larkin, and G. C. Brackett, Teaching gen- eral learning and problem-solving skills, American Jour- nal of Physics 44, 212 (1976)

    1976

  24. [24]

    Reif, Millikan lecture 1994: Understanding and teaching important scientific thought processes, Ameri- can Journal of Physics 63, 17 (1995)

    F. Reif, Millikan lecture 1994: Understanding and teaching important scientific thought processes, Ameri- can Journal of Physics 63, 17 (1995)

    1994

  25. [25]

    L. Hsu, E. Brewe, T. M. Foster, and K. A. Harper, Re- source letter rps-1: Research in problem solving, Amer- ican journal of physics 72, 1147 (2004)

    2004

  26. [26]

    Hattie, Visible learning: A synthesis of over 800 meta-analyses relating to achievement (routledge, 2008)

    J. Hattie, Visible learning: A synthesis of over 800 meta-analyses relating to achievement (routledge, 2008)

    2008

  27. [27]

    Al-Salmani, J

    F. Al-Salmani, J. Johnson, and B. Thacker, Assessing thinking skills in free-response exam problems: Pan- demic online and in-person, Phys. Rev. Phys. Educ. Res. 19, 010131 (2023)

    2023

  28. [28]

    D. A. Kashy, G. Albertelli, G. Ashkenazi, E. Kashy, H.- K. Ng, and M. Thoennessen, Individualized interactive exercises: a promising role for network technology, in Proc. Frontiers in Education, Vol. 31 (2001) pp. 1073– 1078

    2001

  29. [29]

    Kortemeyer, E

    G. Kortemeyer, E. Kashy, W. Benenson, and W. Bauer, Experiences using the open-source learning content management and assessment system LON-CAPA in in- troductory physics courses, Am. J. Phys 76, 438 (2008)

    2008

  30. [30]

    Risley, Motivating students to learn physics using an online homework system, Newsletter of the APS Forum on Education F all, 3 (2001)

    J. Risley, Motivating students to learn physics using an online homework system, Newsletter of the APS Forum on Education F all, 3 (2001)

    2001

  31. [31]

    Stelzer and G

    T. Stelzer and G. Gladding, The evolution of web-based activities in physics at illinois, Newsletter of the APS Forum on Education F all, 7 (2001)

    2001

  32. [32]

    R. J. Dufresne, D. Hart, J. P. Mestre, and K. Rath, The effect of web-based homework on test performance in large enrollment introductory physics courses, Journal of Computers in Mathematics and Science Teaching 21, 229 (2002)

    2002

  33. [33]

    Fredericks, Patterns of Behavior in Online Home- work for Introductory Physics , Ph.D

    C. Fredericks, Patterns of Behavior in Online Home- work for Introductory Physics , Ph.D. thesis, University of Massachusetts (2007)

    2007

  34. [34]

    Richards-Babb, J

    M. Richards-Babb, J. Drelick, Z. Henry, and J. Robertson-Honecker, Online homework, help or hin- drance? what students think and how they perform, Journal of College Science Teaching 40, 81 (2011)

    2011

  35. [35]

    D. C. Perdian, Early identification of student perfor- mance and effort using an online homework system: A pilot study, Journal of Science Education and Technol- ogy 22, 697 (2013)

    2013

  36. [36]

    Burkholder, J

    E. Burkholder, J. Miles, T. Layden, K. Wang, A. Fritz, and C. Wieman, Template for teaching and assessment of problem solving in introductory physics, Physical Re- view Physics Education Research 16, 010123 (2020)

    2020

  37. [37]

    E. G. Offerdahl and J. B. Arneson, Formative assess- ment to improve student learning in biochemistry, in Biochemistry education: From theory to practice (ACS Publications, Washington, DC, 2019) pp. 197–218

    2019

  38. [38]

    T. M. Clark, C. S. Callam, N. M. Paul, M. W. Stoltzfus, and D. Turner, Testing in the time of COVID-19: A sudden transition to unproctored online exams, Journal of chemical education 97, 3413 (2020)

    2020

  39. [39]

    T. M. Gomez, C. Luciano, T. Nguyen, S. M. Villafa˜ ne, and M. N. Groves, Student success and experience in a flipped, senior physical chemistry course spanning be- fore and after the covid-19 pandemic, Chemistry Edu- cation Research and Practice 26, 210 (2025)

    2025

  40. [40]

    Kashy, B

    E. Kashy, B. M. Sherrill, D. T. Y. Tsai, D. Weinshank, M. Engelmann, and D. J. Morrissey, Capa, an integrated computer assisted personalized assignment system, Am. J. Phys 61, 1124 (1993)

    1993

  41. [41]

    Kashy, S

    E. Kashy, S. J. Gaff, N. Pawley, W. L. Stretch, S. Wolfe, D. J. Morrissey, and Y. Tsai, Conceptual questions in computer-assisted assignments, Am. J. Phys 63, 1000 (1995)

    1995

  42. [42]

    Warnakulasooriya and D

    R. Warnakulasooriya and D. Pritchard, Learning and problem-solving transfer between physics problems us- ing web-based homework tutor, in EdMedia+ Innovate Learning (Association for the Advancement of Comput- ing in Education (AACE), 2005) pp. 2976–2983

    2005

  43. [43]

    Gladding, B

    G. Gladding, B. Gutmann, N. Schroeder, and T. Stelzer, Clinical study of student learning using mastery style versus immediate feedback online activities, Phys. Rev. ST Phys. Educ. Res. 11, 010114 (2015)

    2015

  44. [44]

    Gutmann, G

    B. Gutmann, G. Gladding, M. Lundsgaard, and T. Stelzer, Mastery-style homework exercises in intro- ductory physics courses: Implementation matters, Phys. Rev. Phys. Educ. Res. 14, 010128 (2018)

    2018

  45. [45]

    Scott, T

    M. Scott, T. Stelzer, and G. Gladding, Evaluating multiple-choice exams in large introductory physics courses, Phys. Rev. ST Phys. Educ. Res. 2, 020102 (2006)

    2006

  46. [46]

    A. Pawl, R. Teodorescu, and J. Peterson, Assessing class-wide consistency and randomness in responses to true or false questions administered online, Phys. Rev. ST Phys. Educ. Res. 9, 020102 (2013)

    2013

  47. [47]

    Kortemeyer, The psychometric properties of class- room response system data: a case study, Journal of Science Education and Technology 25, 561 (2016)

    G. Kortemeyer, The psychometric properties of class- room response system data: a case study, Journal of Science Education and Technology 25, 561 (2016). 26

    2016

  48. [48]

    Stewart and S

    J. Stewart and S. Ballard, Effect of written presentation on performance in introductory physics, Phys. Rev. ST Phys. Educ. Res. 6, 020120 (2010)

    2010

  49. [49]

    Davis and T

    J. Davis and T. McDonald, Online, handwritten or hy- brid homework: What’s best for our students in the long run?, Journal of Online Engineering Education 7 (2016)

    2016

  50. [50]

    D. J. Palazzo, Y.-J. Lee, R. Warnakulasooriya, and D. E. Pritchard, Patterns, correlates, and reduction of homework copying, Physical Review Special Topics – Physics Education Research 6, 010104 (2010)

    2010

  51. [51]

    G¨ on¨ ulate¸ s and G

    E. G¨ on¨ ulate¸ s and G. Kortemeyer, Modeling unproduc- tive behavior in online homework in terms of latent stu- dent traits: An approach based on item response theory, Journal of Science Education and Technology 26, 139 (2017)

    2017

  52. [52]

    Kortemeyer, An empirical study of the effect of granting multiple tries for online homework, Am

    G. Kortemeyer, An empirical study of the effect of granting multiple tries for online homework, Am. J. Phys. 83, 646 (2015)

    2015

  53. [53]

    VanLehn, B

    K. VanLehn, B. Van De Sande, R. Shelby, and S. Ger- shman, The andes physics tutoring system: An exper- iment in freedom, Advances in intelligent tutoring sys- tems , 421 (2010)

    2010

  54. [54]

    C. M. Nakamura, S. K. Murphy, M. G. Christel, S. M. Stevens, and D. A. Zollman, Automated analysis of short responses in an interactive synthetic tutoring sys- tem for introductory physics, Phys. Rev. Phys. Educ. Res. 12, 010122 (2016)

    2016

  55. [55]

    J. A. Shapiro, An algebra subsystem for diagnosing stu- dents’ input in a physics tutoring system, International Journal of Artificial Intelligence in Education 15, 205 (2005)

    2005

  56. [56]

    OpenAI, ChatGPT, https://chat.openai.com/ (ac- cessed April 2024)

    2024

  57. [57]

    OpenAI, ChatGPT, https://openai.com/research/ gpt-4 (accessed April 2024)

    2024

  58. [58]

    OpenAI, Hello GPT-4o, https://openai.com/index/ hello-gpt-4o/ (accessed June 2024)

    2024

  59. [59]

    J. G. Meyer, R. J. Urbanowicz, P. C. Martin, K. O’Connor, R. Li, P.-C. Peng, T. J. Bright, N. Tatonetti, K. J. Won, G. Gonzalez-Hernandez,et al., ChatGPT and large language models in academia: op- portunities and challenges, BioData Mining 16, 20 (2023)

    2023

  60. [60]

    Kasneci, K

    E. Kasneci, K. Seßler, S. K¨ uchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. G¨ unnemann, E. H¨ ullermeier,et al. , ChatGPT for good? on opportunities and challenges of large language models for education, Learning and individual differ- ences 103, 102274 (2023)

    2023

  61. [61]

    Yeadon and T

    W. Yeadon and T. Hardy, The impact of AI in physics education: a comprehensive review from GCSE to uni- versity levels, Physics Education 59, 025010 (2024)

    2024

  62. [62]

    Sperling and J

    A. Sperling and J. Lincoln, Artificial intelligence and high school physics, The Physics Teacher62, 314 (2024)

    2024

  63. [63]

    Polverini and B

    G. Polverini and B. Gregorcic, How understanding large language models can inform the use of ChatGPT in physics education, Eur. J. Phys. 45, 025701 (2024)

    2024

  64. [64]

    Tschisgale, P

    P. Tschisgale, P. Wulff, and M. Kubsch, Integrating ar- tificial intelligence-based methods into qualitative re- search in physics education research: A case for com- putational grounded theory, Physical Review Physics Education Research 19, 020123 (2023)

    2023

  65. [65]

    Kieser, P

    F. Kieser, P. Wulff, J. Kuhn, and S. K¨ uchemann, Ed- ucational data augmentation in physics education re- search using ChatGPT, Phys. Rev. Phys. Educ. Res. 19, 020150 (2023)

    2023

  66. [66]

    P. Wulff, Physics language and language use in physics—what do we know and how ai might enhance language-related research and instruction, European Journal of Physics 45, 023001 (2024)

    2024

  67. [67]

    T. H. Kung, M. Cheatham, A. Medinilla, ChatGPT, C. Sillos, L. De Leon, C. Elepano, M. Madriaga, R. Ag- gabao, G. Diaz-Candido, et al., Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models, medRxiv , 2022 (2022)

    2022

  68. [68]

    GPT-4 Technical Report

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Alt- man, S. Anadkat, et al., GPT-4 technical report, arXiv preprint arXiv:2303.08774 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  69. [69]

    Kortemeyer, Could an artificial-intelligence agent pass an introductory physics course?, Phys

    G. Kortemeyer, Could an artificial-intelligence agent pass an introductory physics course?, Phys. Rev. Phys. Educ. Res. 19, 010132 (2023)

    2023

  70. [70]

    L´ opez-Sim´ o and M

    V. L´ opez-Sim´ o and M. F. Rezende, Challenging chat- gpt with different types of physics education questions, Phys. Teach. 62, 290 (2024)

    2024

  71. [71]

    K. D. Wang, E. Burkholder, C. Wieman, S. Salehi, and N. Haber, Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving, in Frontiers in Education, Vol. 8 (Frontiers Media SA,

  72. [72]

    Kortemeyer, M

    G. Kortemeyer, M. Babayeva, G. Polverini, R. Widen- horn, and B. Gregorcic, Multilingual performance of a multimodal artificial intelligence system on multisubject physics concept inventories, Physical Review Physics Education Research 21, 020101 (2025)

    2025

  73. [73]

    K¨ uchemann, S

    S. K¨ uchemann, S. Steinert, N. Revenga, M. Schwein- berger, Y. Dinc, K. E. Avila, and J. Kuhn, Can Chat- GPT support prospective teachers in physics task de- velopment?, Phys. Rev. Phys. Educ. Res. 19, 020128 (2023)

    2023

  74. [74]

    Wilson, B

    J. Wilson, B. Pollard, J. M. Aiken, M. D. Caballero, and H. J. Lewandowski, Classification of open-ended responses to a research-based assessment using natural language processing, Phys. Rev. Phys. Educ. Res. 18, 010141 (2022)

    2022

  75. [75]

    Wan and Z

    T. Wan and Z. Chen, Exploring generative AI assisted feedback writing for students’ written responses to a physics conceptual question with prompt engineering and few-shot learning, Phys. Rev. Phys. Educ. Res. 20, 010152 (2024)

    2024

  76. [76]

    Kortemeyer, Toward AI grading of student problem solutions in introductory physics: A feasibility study, Phys

    G. Kortemeyer, Toward AI grading of student problem solutions in introductory physics: A feasibility study, Phys. Rev. Phys. Educ. Res. 19, 020163 (2023)

    2023

  77. [77]

    Kortemeyer, J

    G. Kortemeyer, J. N¨ ohl, and D. Onishchuk, Grading as- sistance for a handwritten thermodynamics exam using artificial intelligence: An exploratory study, Physical Review Physics Education Research 20, 020144 (2024)

    2024

  78. [78]

    T. Liu, J. Chatain, L. Kobel-Keller, G. Kortemeyer, T. Willwacher, and M. Sachan, Ai-assisted automated short answer grading of handwritten university level mathematics exams, arXiv preprint arXiv:2408.11728 (2024)

    work page arXiv 2024

  79. [79]

    Kortemeyer and J

    G. Kortemeyer and J. N¨ ohl, Assessing confidence in ai- assisted grading of physics exams through psychomet- rics: An exploratory study, Physical Review Physics Ed- ucation Research 21, 010136 (2025). 27

    2025

  80. [80]

    Frakn´ oi, A

    ´A. Frakn´ oi, A. Kornai, and Z. Zombori, Embedding mathematical formulas into vector space, in 8th Con- ference on Artificial Intelligence and Theorem Proving (2023)

    2023

Showing first 80 references.