A bottom-up taxonomy of student discourse with a Socratic AI physics tutor

N. Sanjay Rebello; Syed Furqan Abbas Hashmi

REVIEW 3 major objections 6 minor 20 references

A bottom-up taxonomy of student discourse with a Socratic AI physics tutor

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper builds a scalable taxonomy of what students say to a Socratic AI physics tutor and finds that a few dozen moves dominate, with 'Next Step Guidance' second only to writing the energy equation.

desk verdict A transparent descriptive taxonomy of student–AI-tutor discourse with a credible concentration finding, but the central interpretation about students ceding control rests on unverified tutor behavior; worth refereeing. read the letter →

arxiv 2608.07373 v1 pith:NN66ODFR submitted 2026-08-07 physics.ed-ph

Syed Furqan Abbas Hashmi , N. Sanjay Rebello This is my paper · ORCID

classification physics.ed-ph

keywords studentdiscourseSocraticAItutorLLM-basedcodingtaxonomymeta-proceduralrequestsequation-handlingphysicseducationresearchembedding-basedclustering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks a descriptive question that precedes any effectiveness claim: when students work with a Socratic AI physics tutor, what do they actually say? The authors build a taxonomy from 2,874 student turns, letting a large language model assign free-text labels to each turn and then merging near-synonyms into 357 categories. The main finding is concentration: the top 25 categories cover roughly half of all turns, and the most frequent categories split into equation-handling moves and meta-procedural requests. In particular, 'Next Step Guidance' is the second-most common category, which the authors interpret as students treating the tutor as the director of the procedural flow. The contribution is a descriptive baseline for physics education research at a scale hand coding cannot reach.

What carries the argument

The carrying mechanism is a two-stage emergent coding pipeline. In the first stage, a large language model reads each student turn with the previous four conversational turns as context and writes a short free-text label for the discourse move, the machine analogue of open coding. In the second stage, all raw labels are converted to text embeddings, and agglomerative clustering (a bottom-up merging routine) on cosine distance merges near-paraphrases into consolidated categories, which are then named by the model and spot-checked by a human rater. This pipeline converts thousands of messages into a manageable taxonomy, and its validity rests on the agreement of the resulting labels with human expert coding.

What would settle it

Code a stratified sample of tutor turns and count how often the tutor's response directly supplies the next procedural step; if those direct tutor turns largely coincide with, or immediately precede, the student's 'Next Step Guidance' turns, the claim that students are ceding control on their own initiative would be undermined.

Watch

Extended reading notes

Core claim

The paper establishes that student discourse with this Socratic AI tutor is sharply concentrated and thematically lopsided. Across 221 sessions and 2,874 student turns, the LLM coder produced 833 raw labels that were consolidated into 357 categories; the top 25 categories cover roughly half of all turns, while categories with at least five messages cover 86% of the corpus. Two bands dominate the most frequent categories: equation-handling and symbolic execution (Writing Energy Equation, Velocity Solving, Solving for Height, Centripetal Relation Setup, among others), and meta-procedural requests (Next Step Guidance, Requesting Solution Help, Asking for Principles, Assumptions About Problem). 'Next Step Guidance' alone is the second-largest category at 4.4% of turns, and the authors read this as students framing the activity as one in which the tutor supplies the next strategic move. The automatic labels agree with a human expert on a stratified 10% sample at $\kappa = 0.78$, which the paper offers as evidence that the pipeline can scale to much larger transcript corpora.

Load-bearing premise

The central interpretation assumes the Socratic tutor actually behaved Socratically, withholding direct answers as its system prompt instructed, so the frequent 'Next Step Guidance' turns reflect students' own framing rather than the tutor's directiveness.

Editorial extensions

If this is right

A short codebook of roughly 25 categories can describe over half of the student turns in this corpus, making large-scale transcript analysis practical.
For designers, 'Next Step Guidance' at 4.4% of turns gives a concrete behavioral target: the effectiveness of a Socratic tutor may hinge on how it redirects explicit requests for the next step.
The agreement level of about 0.78 between LLM and human labels supports using automated coding as a scalable first pass for similar transcript corpora.
The top of the distribution contains essentially no conceptual-reasoning or prediction moves, so the intended Socratic scaffolding is not mirrored by high-frequency student behaviors in this dataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Going beyond the paper, running the same coding on a non-Socratic direct-instruction tutor would test whether the 'Next Step Guidance' pattern is a reaction to the Socratic design or a general student habit.
A randomized version that blocks 'Next Step Guidance' requests and forces students to propose their own next step would test whether the procedural framing is sticky or easily shifted.
The long tail of low-count categories, which the paper leaves unexamined, may hold most of the conceptual-reasoning moves; looking there could change how the absence of conceptual talk at the head is read.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: this paper gives the field a usable, transparent pipeline for turning LLM tutor transcripts into a bottom-up taxonomy, and the headline empirical result—student discourse concentrates into a few dozen moves—is credible. The softer part is the interpretation of RQ2: the claim that students cede strategic control depends on the tutor actually having withheld direct steps, and that premise is never checked.

What is genuinely new: the method is spelled out in enough detail to reproduce—emergent LLM labeling, embedding-based consolidation, threshold selection, and a human reliability sample. The authors cite prior computational grounded theory work (Nelson; Tschisgale et al.) and adapt it honestly. They report kappa, raw agreement, and limitations. The 357-category taxonomy and the heavy-tailed distribution are the real contributions; the top 25 covering roughly half of turns across 221 independent sessions is a useful baseline for PER.

Soft spots, in proportion: first, the validation maps both LLM and expert labels onto the consolidated category set that the LLM clustering produced. That makes the reported kappa partly a function of the clustering, so the reliability evidence is weaker than the number suggests. Second, the clustering threshold was chosen by one rater's coherence judgment, and there are no confidence intervals on the proportions. These are fixable. Third, and most important: the RQ2 interpretation. Sections IV.B and V read the prevalence of Next Step Guidance as evidence that students frame the tutor as the director. That inference only works if the tutor behaved Socratically in practice. The paper only reports the system prompt; it never analyzes tutor turns. If the tutor often asked "what next?" or supplied step-by-step instructions, the category would reflect tutor moves rather than student framing. The paper does concede in IV.B that the taxonomy alone cannot resolve this, and the limitations section repeats it, but the conclusion still leans on the unverified premise. That tension should be fixed by analyzing tutor turns or clearly softening the RQ2 claim.

Who this is for: PER researchers working with LLM tutors, and anyone building discourse-analysis pipelines. The descriptive core is solid and the interpretive flaw is addressable. I'd send it out for peer review, with a request to add a tutor-turn analysis or downgrade the RQ2 conclusion accordingly.

Referee Report

3 major / 6 minor

Summary. The paper proposes a bottom-up, LLM-assisted pipeline for building a taxonomy of student discourse from transcripts of student turns with a Socratic AI physics tutor. Using 2,874 student turns from 221 sessions in an introductory mechanics course, the authors assign emergent free-text labels to each turn, consolidate near-paraphrase labels via embedding-based agglomerative clustering into 357 categories, and validate the Step 1 labels against a stratified expert-coded sample (Cohen's kappa = 0.78). The central descriptive finding is that the discourse is sharply concentrated: the top 25 categories cover roughly half of all student turns, and the head of the distribution is dominated by equation-handling and meta-procedural moves. The paper interprets the prevalence of 'Next Step Guidance' (4.4%, the second-largest category) as evidence that students cede strategic control to the tutor despite its Socratic design, and discusses implications for PER and AI tutor design.

Significance. If the descriptive findings hold, the paper makes a useful empirical contribution to physics education research by providing a scalable, transparent method for mapping large AI-tutor corpora into discourse categories and by documenting what students actually say to a Socratic AI tutor in a real course. The paper is appropriately cautious in several places: it explicitly notes the observational nature of the study, the single-site/single-problem scope, the 10% validation sample, and the single-rater coherence audit. The open-coding approach and the use of a human-coded reliability sample are methodological strengths, as is the clear reporting of the clustering procedure and the heavy-tailed distribution. The paper also connects its findings to existing PER constructs (symbolic forms, epistemic games, epistemological framing), giving the taxonomy interpretive traction. The main weakness is that the RQ2 interpretation depends on unverified assumptions about actual tutor behavior, and the validation design partly inherits the LLM-generated category structure; these issues are load-bearing for the paper's strongest interpretive claims but are addressable within the manuscript's scope.

major comments (3)

[Section IV.B and Section V] The claim that 'Next Step Guidance' shows students cede strategic control to the tutor rests on the premise that the tutor actually withheld direct next-step guidance and did not itself prompt meta-procedural responses such as 'What should you do next?' The paper reports only the system prompt instruction (Section III.A) and never analyzes tutor turns to verify the enacted tutoring behavior. Because the corpus contains the tutor responses, the authors can directly test this premise; without such a check, the RQ2 conclusion in Section V ('a tutor explicitly designed not to direct students nevertheless elicits a discourse in which directing is the second-most-requested service') overstates what the taxonomy alone can establish. Section IV.B itself concedes that 'whether this represents a failure... is not something the taxonomy alone can resolve,' so the conclusion should be correspondingly tempered or supported with a tutor-turn analysis.
[Section III.D] The reliability validation maps both expert and LLM free-text labels onto the consolidated category set produced by the LLM-based clustering before computing Cohen's kappa. This means the reported agreement partly reflects the mapping step, which inherits the LLM clustering boundaries, rather than an independent check of the taxonomy. The authors should report agreement at the level of the raw free-text labels (or provide a detailed account of how expert labels were mapped and how many required manual adjudication), and ideally have experts code directly into an independently derived category set for at least a subsample. Without this, the kappa of 0.78 may overstate the validity of the consolidated categories.
[Section III.C.2] The agglomerative clustering distance threshold (0.40) was selected by a single rater's coherence judgment on a stratified sample of 50 groups, with no second rater or quantitative stability check. Because this threshold directly determines the number and boundaries of the 357 consolidated categories and therefore every frequency claim in Section IV, the taxonomy's central quantitative findings rest on a parameter that has only face validity. The acknowledged single-rater limitation is a step toward transparency, but the authors should add a sensitivity analysis showing how the head-of-distribution claims (top 25 covering roughly half of all turns, the rank of 'Next Step Guidance') change across the 0.20–0.60 threshold sweep, or at least across a few neighboring thresholds.

minor comments (6)

[Abstract] The abstract states 'top 25 categories cover roughly half of all student turns' while Section IV reports the top 20 cover 47.2%; since the top-25 table is not shown, the reader cannot verify the exact percentage, and a precise value or a reference to a supplementary table would be helpful.
[Section III.A] There are typos in the deployment description ('students were saked to use the tutor to sove the REC'), which should be corrected for a journal submission.
[Section III.B] The context-window pilot (0, 2, 4, 8 turns) is described only briefly; reporting the inter-rater agreement or label-consistency measure used to select four turns would strengthen the reproducibility of the coding protocol.
[Table I] The table would be easier to interpret if it included the cumulative percentage and clearly indicated that the top-20 total (1357) corresponds to 47.2% of 2,874 turns; adding a top-25 row or noting the 52% figure would align the table with the abstract.
[Section IV.B] The phrase 'the second-largest category overall' is supported by Table I, but the text could clarify that the ranking is among consolidated categories and that 'Next Step Guidance' is second only to 'Writing Energy Equation.'
[Section VI] The limitations section is candid and helpful; however, it does not mention the absence of any reported inter-rater reliability for the Step 2 nomenclature spot-checks, which would be a useful addition given the paper's emphasis on trustworthiness.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the taxonomy is a direct empirical description of student turns, with independent human validation and an explicitly qualified RQ2 interpretation.

full rationale

The paper's central output is a descriptive taxonomy built from 2,874 student turns through an emergent LLM labeling step, an embedding-based agglomerative clustering step with a transparently selected distance threshold, and a human validation on a 10% stratified sample. The headline frequency claims (e.g., 'top 25 categories cover roughly half of all turns') are direct counts over the resulting categories; they are not predictions from a fitted model, nor are any categories defined in terms of the frequencies they are later used to explain. The validation step maps expert labels onto the consolidated category set derived from the LLM labels, which is a methodological limitation that could affect the reported Cohen's kappa, but the expert labels are generated independently and the agreement is not forced by construction; the paper also discloses this as a reliability limitation. The RQ2 interpretation that students 'cede strategic control' is explicitly qualified by the authors in Section IV.B ('Whether this represents a failure... is not something the taxonomy alone can resolve'), so it is not presented as a necessary derivation from the taxonomy. The only self-citation ([16]) describes the tutor's provenance and does not carry the load of any finding. No circular step can be exhibited by quoting a specific reduction, so the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The taxonomy is an analytical construct, not a physical entity. The central claim rests on assumptions about LLM coding validity, embedding semantics, tutor behavior, validation mapping, and corpus representativeness. The clustering threshold and context window are human-chosen parameters that shape the results.

free parameters (2)

Agglomerative clustering distance threshold = 0.40
Selected by sweeping 0.20 to 0.60 and first-author coherence judgment on a sample of 50 groups; this choice determines category granularity and directly affects the reported concentration of the taxonomy.
LLM context window length = 4 turns
Piloted windows of 0, 2, 4, and 8 turns; chosen as the smallest window that recovered consistent labels for context-dependent moves, affecting label quality and distinctiveness.

assumptions (5)

domain assumption LLM-generated free-text labels correspond to meaningful student discourse moves.
Step 1 uses GPT-5.4-mini as the coder; if LLM labels are unreliable, the entire taxonomy is undermined. The paper validates against a single human coder.
domain assumption Text embedding cosine similarity captures semantic equivalence of labels.
Step 2 consolidates raw labels by embedding distance; the validity of the 357 categories depends on embeddings aligning with human notions of paraphrases.
domain assumption The tutor actually behaved Socratically as instructed.
The system prompt instructed Socratic prompting and conceptual scaffolding, but the paper does not analyze tutor turns to confirm; the interpretation of meta-procedural requests relies on the tutor withholding direct answers.
domain assumption Mapping expert labels onto the LLM-generated category set provides an unbiased reliability estimate.
Section III D validation maps human free-text labels to the consolidated categories; if the mapping is biased toward LLM labels, kappa=0.78 overstates reliability.
domain assumption The Week 8 corpus represents the discourse this tutor design elicits.
Single site, single course, single tutor, single problem; the paper acknowledges limited generalizability.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A bottom-up taxonomy of student discourse with a Socratic AI physics tutor." pith.science (2026). https://pith.science/paper/NN66ODFR

@misc{pith2026260807373,
  author       = {Pith},
  title        = {Pith review of: A bottom-up taxonomy of student discourse with a Socratic AI physics tutor},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NN66ODFR}},
  note         = {Machine review of arXiv:2608.07373}
}

read the original abstract

Large language model (LLM) tutors are being deployed in introductory physics courses at a scale that produces transcript corpora far larger than traditional qualitative coding can absorb. A central question for physics education research (PER) is empirical and prior to any claim about effectiveness: what do students actually say to these tutors? We address this question for one Socratic AI tutor deployed in an introductory calculus-based mechanics course by building a bottom-up taxonomy of student discourse. Each student turn is assigned an emergent free-text label by an LLM coder using the surrounding conversational context; near-paraphrase labels are then consolidated into a smaller set of discourse categories using a similarity-based grouping procedure. The procedure is validated against a stratified human-coded sample. The resulting taxonomy of 357 categories is strikingly concentrated: the top 25 categories cover roughly half of all student turns, and two thematic bands: equation-handling and meta-procedural requests together dominate the head of the distribution. The substantive contribution is the taxonomy itself: a description of the discourse PER researchers can expect to encounter when students work with an AI tutor of this design, including a striking prevalence of meta-procedural turns in which students cede strategic control to the tutor

Figures

Figures reproduced from arXiv: 2608.07373 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages

[1]

The intuition is a map on which related phrases sit near each other and unre- lated phrases sit far apart

Similarity-based Grouping Each raw label was converted into atext embedding, which is a vector produced by a neural network (OpenAI’s text-embedding-3-small) such that labels with sim- ilar meaning end up as similar lists of numbers, and unre- lated labels end up as dissimilar vectors. The intuition is a map on which related phrases sit near each other an...

work page
[2]

We used average-linkage merging, two groups are considered “close” if their members are close on av- erage

Agglomerative Clustering We then appliedagglomerative clustering, a bottom- up grouping procedure that begins with every label as its own group and repeatedly merges the two closest groups until no remaining pair is closer than a chosen distance threshold. We used average-linkage merging, two groups are considered “close” if their members are close on av-...

work page
[3]

This produced 357 consolidated discourse categories

LLM-generated Nomenclature Each resulting group was then passed to an LLM call with the constituent raw labels and their counts, and the model was asked to produce a short canonical category name; cluster names were spot-checked by a human rater. This produced 357 consolidated discourse categories. The top 25 categories cover∼52% of all messages; categori...

work page
[4]

Kestin, K

G. Kestin, K. Miller, A. Klales, T. Milbourne, and G. Ponti, Scientific Reports15, 17458 (2025)

work page 2025
[5]

Létourneau, M

A. Létourneau, M. Deslandes Martineau, P. Charland, J. A. Karran, J. Boasen, and P. M. Léger, npj Science of Learning 10, 29 (2025)

work page 2025
[6]

Kregear, M

T. Kregear, M. Babayeva, and R. Widenhorn, International Journal of Artificial Intelligence in Education35, 2993 (2025)

work page 2025
[7]

Larkin, J

J. Larkin, J. McDermott, D. P. Simon, and H. A. Simon, Sci- ence208, 1335 (1980)

work page 1980
[8]

M. T. H. Chi, P. J. Feltovich, and R. Glaser, Cognitive Science 5, 121 (1981)

work page 1981

Show all 20 references

[9]

J. I. Heller and F. Reif, Cognition and Instruction1, 177 (1984)

1984
[10]

B. L. Sherin, Cognition and Instruction19, 479 (2001)

2001
[11]

Tuminaro and E

J. Tuminaro and E. F. Redish, Physical Review Special Topics — Physics Education Research3, 020101 (2007)

2007
[12]

R. E. Scherr and D. Hammer, Cognition and Instruction27, 147 (2009)

2009
[13]

L. K. Nelson, Sociological Methods & Research49, 3 (2020)

2020
[14]

Tschisgale, P

P. Tschisgale, P. Wulff, and M. Kubsch, Physical Review Physics Education Research19, 020123 (2023)

2023
[15]

T. O. B. Odden, A. Marin, and M. D. Caballero, Physical Re- view Physics Education Research16, 010142 (2020)

2020
[16]

T. O. B. Odden, H. Tyseng, J. T. Mjaaland, M. F. Kreutzer, and A. Malthe-Sørenssen, Physical Review Physics Education Research20, 020151 (2024)

2024
[17]

R. K. Fussell, E. M. Stump, and N. G. Holmes, Physical Re- view Physics Education Research20, 010113 (2024)

2024
[18]

Kortemeyer, Physical Review Physics Education Research 19, 010132 (2023)

G. Kortemeyer, Physical Review Physics Education Research 19, 010132 (2023)

2023
[19]

S. F. A. Hashmi and N. S. Rebello, in2025 Physics Education Research Conference Proceedings(American Association of Physics Teachers, Washington, DC, 2025)

2025
[20]

Bangs, International handbook on teaching and learning eco- nomics , 48 (2012)

J. Bangs, International handbook on teaching and learning eco- nomics , 48 (2012)

2012

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

The intuition is a map on which related phrases sit near each other and unre- lated phrases sit far apart

Similarity-based Grouping Each raw label was converted into atext embedding, which is a vector produced by a neural network (OpenAI’s text-embedding-3-small) such that labels with sim- ilar meaning end up as similar lists of numbers, and unre- lated labels end up as dissimilar vectors. The intuition is a map on which related phrases sit near each other an...

work page

[2] [2]

We used average-linkage merging, two groups are considered “close” if their members are close on av- erage

Agglomerative Clustering We then appliedagglomerative clustering, a bottom- up grouping procedure that begins with every label as its own group and repeatedly merges the two closest groups until no remaining pair is closer than a chosen distance threshold. We used average-linkage merging, two groups are considered “close” if their members are close on av-...

work page

[3] [3]

This produced 357 consolidated discourse categories

LLM-generated Nomenclature Each resulting group was then passed to an LLM call with the constituent raw labels and their counts, and the model was asked to produce a short canonical category name; cluster names were spot-checked by a human rater. This produced 357 consolidated discourse categories. The top 25 categories cover∼52% of all messages; categori...

work page

[4] [4]

Kestin, K

G. Kestin, K. Miller, A. Klales, T. Milbourne, and G. Ponti, Scientific Reports15, 17458 (2025)

work page 2025

[5] [5]

Létourneau, M

A. Létourneau, M. Deslandes Martineau, P. Charland, J. A. Karran, J. Boasen, and P. M. Léger, npj Science of Learning 10, 29 (2025)

work page 2025

[6] [6]

Kregear, M

T. Kregear, M. Babayeva, and R. Widenhorn, International Journal of Artificial Intelligence in Education35, 2993 (2025)

work page 2025

[7] [7]

Larkin, J

J. Larkin, J. McDermott, D. P. Simon, and H. A. Simon, Sci- ence208, 1335 (1980)

work page 1980

[8] [8]

M. T. H. Chi, P. J. Feltovich, and R. Glaser, Cognitive Science 5, 121 (1981)

work page 1981