Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles

Ramatu Oiza Abdulsalam; Segun Aroyehun

REVIEW 3 major objections 6 minor 2 references

Large language models can match expert human tutors in perceived math-tutoring quality—if their responses restate the student's reasoning and dial back politeness and agentic language.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

On identical math tutoring turns, the strongest LLMs reach near-expert perceived quality while systematically differing from experts in instructional and linguistic profile.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection A competently designed comparative study of human and LLM tutoring quality whose central regression rests on unvalidated classifier scores; worth refereeing with a request for validation. the 3 major comments →

arxiv 2512.20780 v3 pith:OQUO73G7 submitted 2025-12-23 cs.CL cs.CY

Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles

Ramatu Oiza Abdulsalam , Segun Aroyehun This is my paper

classification cs.CL cs.CY

keywords large language modelsmath tutoringpedagogical qualityrestating/revoicinglexical diversitypolitenessagencywithin-conversation regression

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that recent large language models produce math-tutoring responses rated as pedagogically close to expert human tutors, even though they get there with different moves: longer, more lexically diverse, more polite, less readable, and less likely to restate the student's reasoning. Using the same student-error prompts as a controlled setting, the authors show that expert tutors beat novices, and the strongest LLMs approach expert-level perceived quality. The key statistical finding is that pedagogical quality tracks specific response features rather than tutor identity: restating/revoicing the student's reasoning and lexical diversity are strong positive correlates, while politeness and agentic language are negative correlates. A sympathetic reader would care because it redirects LLM tutor design from imitating humans or maximizing fluency toward engineering specific instructional-linguistic combinations.

Core claim

On the paper's own terms, the discovery is that perceived pedagogical quality is not a property of who or what generates a tutoring response, but of the response's instructional moves and linguistic shape. On identical student-error turns, the strongest LLMs receive ratings that approach expert human tutors on average, while novices fall far below. Yet the surface style LLMs naturally produce—longer, more lexically diverse, more polite, less readable—does not match the style that predicts quality. Within-conversation regressions show that restating/revoicing the student's reasoning and pressing for accuracy, along with lexical diversity, are positively associated with perceived pedagogical q

What carries the argument

The analytical engine is a within-conversation comparison. For each student-error turn, every tutor's pedagogical quality score—the average of four annotations (mistake identification, mistake location, guidance, actionability)—is centered on the turn mean, and a linear regression is estimated after demeaning predictors inside each conversation. This removes task difficulty and error type as confounds, so the coefficients compare tutors against one another on identical prompts. The features entering that regression are probabilistic outputs of pre-trained classifiers for pressing, restating/revoicing, politeness, and agency, plus MTLD lexical diversity, Flesch-Kincaid readability, and log re

Load-bearing premise

The load-bearing premise is that the three pre-trained classifiers used to measure restating/revoicing, politeness, and agency produce trustworthy scores on these brief mixed human-and-AI tutoring responses; the regression relies on those scores without calibration or validation on this text type.

What would settle it

Hand-label a random subset of the same responses for restating/revoicing, pressing, politeness, and agency, then re-estimate the within-conversation regression using the human labels instead of classifier probabilities. If the positive coefficients for restating and lexical diversity, or the negative coefficients for politeness and agency, shrink toward zero or flip, the paper's central associations do not survive measurement. Alternatively, a prompt-ablation experiment: keep everything about a response identical except insert an explicit restatement of the student's reasoning and check whethe

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Better LLM tutors should be optimized to restate/revoice the student's reasoning and press for accuracy; the regression estimates these as the strongest positive correlates of perceived quality.
Because agentic and polite language are negatively associated with perceived quality, reducing encouraging filler and assertive phrasing may matter more than making responses sound friendlier.
Pedagogical quality ratings are not explained by who (or what) produced the response; the same features predict quality for human and LLM responses, so evaluation should focus on the response itself.
Mere scale helps: larger LLMs approach expert performance, but scale alone does not eliminate the profile gap—the underuse of restating/revoicing persists across models.
Readability and response length show no detectable association with perceived quality in this specification, so interventions that simply make feedback shorter or easier to read would not be supported by these data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

An experiment that randomly inserts or removes a restatement of the student's reasoning, holding the rest of the response fixed, would turn the paper's correlational claim into a causal one; the authors do not run such an intervention.
Re-estimating the regression on a subsample where the discourse features (restating, pressing, politeness, agency) are labeled by human annotators would test whether the pre-trained classifier probabilities are the load-bearing part of the measurement; the paper includes no such validation.
The paper's quality score captures annotated error handling, not whether the student later solves the problem correctly; linking feature profiles to downstream learning gains would show whether 'perceived quality' corresponds to actual efficacy.
Because the data are English middle-school math only, a transfer check to other subjects or languages would reveal whether restating/revoicing is a general tutoring strategy or specific to this context.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A competently designed comparative study of human and LLM tutoring quality whose central regression rests on unvalidated classifier scores; worth refereeing with a request for validation.

read the letter

Quick take: this is a genuinely useful descriptive study, and the stress-test concern about classifier validity is real and load-bearing, not a nitpick.

What's new: they compare expert and novice human tutors plus seven LLMs on the exact same student-error turns, and jointly profile instructional moves (pressing, revoicing) and linguistic features (length, MTLD, readability, politeness, agency). That joint turn-level design is a real step beyond earlier human-vs-LLM comparisons. The RQ2 result—larger LLMs approach expert-level perceived quality while novices lag—is cleanly demonstrated with within-turn demeaning and clustered SEs. Credit also for reporting the regression coefficients and confidence intervals in an appendix, which makes the claims checkable.

Soft spots, in order:

1. The stress-test note lands. The features in RQ1 and RQ3 come from three pre-trained classifiers used as continuous probability scores with no calibration, no domain-shift validation, and no comparison against human annotations on this text type. TalkMoves was trained on K-12 classroom discourse; the politeness and agency models on other genres. If these scores are systematically biased by source (LLM vs human) or by response length, the whole profile picture and the regression coefficients could shift. This is load-bearing because the prescriptive conclusion—optimize revoicing and lexical diversity, reduce politeness and agency—depends on it. The paper cites prior work using classifier probabilities in downstream analyses, but that doesn't establish validity here. I'd want at least a small human-annotation validation subsample or a robustness check with discrete labels.

2. Minor: the sample-count description is sloppy. "300 teacher-student dialogues" vs "2476 individual conversations" vs N=2444 in the regression. Probably explainable, but needs cleaning.

3. The regression has no tutor-type fixed effect, so the feature-quality associations could partly reflect source-level differences (experts restate more; novices are more agentic). The within-turn demeaning controls for turn difficulty, but not for unmeasured response-level confounds. That said, this is a descriptive regression, not a causal claim, and the authors don't overclaim.

4. The outcome is perceived pedagogical quality from annotations, not learning gains. The authors acknowledge this in the Limitations. Fine, but readers should keep it in mind.

Net: The design is sound for a descriptive profile, the data are appropriate, and the main RQ2 finding is solid. The classifier issue is the one thing that could make the RQ1/RQ3 findings artifacts. It's fixable, but it's central.

I'd send this to peer review. The question is not whether it's important enough—it is, for the AI-in-education crowd—but whether the referees will catch the validity issue. The paper deserves a serious referee, with the request that the classifier validation be addressed or the claims tempered.

Referee Report

3 major / 6 minor

Summary. The paper analyzes a dataset of math-remediation turns in which an expert tutor, a novice tutor, and seven LLMs respond to the same student errors. It extracts instructional features (pressing for accuracy, restating/revoicing) and linguistic features (log length, MTLD lexical diversity, Flesch-Kincaid readability, politeness, agency) and compares profiles against expert tutors. RQ2 finds that larger LLMs approach expert-level perceived pedagogical quality, while RQ3 regressions report that restating/revoicing, pressing, and lexical diversity are positively associated with perceived quality, and politeness and agency are negatively associated. The paper concludes that optimizing these feature combinations rather than imitating human tutors is the path to better LLM tutors.

Significance. The controlled within-turn design is a genuine strength: every tutor responds to the same student errors, and standard errors are clustered by conversation. If the feature measurements are valid, the finding that LLMs can approach expert perceived quality while using different instructional and linguistic strategies is important for both tutoring-system design and evaluation methodology. The paper is also transparent about several limitations, including single-turn annotations and the absence of downstream learning outcomes. However, the main empirical claims depend on pre-trained classifiers used without validation on the target text type, and the RQ3 regression omits tutor-type fixed effects, so the headline associations and the prescriptive conclusion are stronger than the current evidence supports.

major comments (3)

[§3.2, Fig. 1, Table 1] The probabilistic scores from the TalkMoves, Srinivasan & Choi, and Nikadon et al. classifiers are used directly as continuous predictors with no calibration, validation, or domain-shift analysis on math-tutoring responses. These classifiers were trained on classroom discourse, multilingual politeness, and general agency texts, not short, mixed human/LLM tutoring responses. Since RQ1 and RQ3 rely entirely on these scores, systematic miscalibration by source or text type could produce both the profile differences in Figure 1 and the regression coefficients in Table 1. The paper should report internal validation (e.g., human-coded subset of tutoring responses, calibration plots, by-source error analysis) or substantially temper the conclusions.
[§4.3, Table 1, Conclusion] The within-conversation regression does not include tutor-type or source fixed effects. Because sources differ systematically on both the features and perceived quality, the coefficients may reflect source composition rather than a within-source feature-quality association. For instance, expert tutors restate more and score higher, so the large positive coefficient for restating/revoicing (β=0.78) could be an expert-vs-novice contrast. The Conclusion's claim that pedagogical quality is a property of response-level features 'regardless of whether the response is produced by a human or an LLM' requires controlling for tutor type or demonstrating that the associations hold within source. Adding tutor-type fixed effects or interaction analyses would directly test this.
[§4.3, Fig. 3 caption] Figure 3's caption calls response length a 'control,' but Table 1 lists it as a predictor with its own coefficient and p-value, and the text interprets its non-significance substantively. This ambiguity matters for interpreting the specification. If length is a control, its coefficient should not be interpreted; if it is a predictor, the caption should say so. Please clarify the role of response length and, if it is a control, the exact set of variables in the model.

minor comments (6)

[§3.1] The dataset description says '300 teacher-student dialogues' and then 'The dataset includes 2476 individual conversations.' These units are inconsistent. Clarify whether the 2,476 are individual student turns within 300 dialogues, and ensure N in Table 1 (2,444) is reconciled.
[§3.3] The terms 'conversation' and 'turn' are used interchangeably in places (e.g., 'turn-level mean' vs. 'within-conversation regression'). Define whether a conversation is a single student-error turn with multiple tutor responses.
[§4.1, Fig. 1] The caption says error bars indicate 95% confidence intervals, but the main text does not say whether these are derived from clustered standard errors. State the clustering and estimation method for the mean-difference estimates.
[Appendix A] The example responses with extracted features are illustrative but not tied to the actual probability scores or to the regression. Adding the scores for each example would help readers assess the validity of the classifier outputs.
[§4.3] The sentence 'response length is not statistically associated with pedagogical quality' is followed by p = 0.059. Consider wording this as 'marginally associated' or 'not significant at the 5% level' for precision.
[References] The Zanotto and Aroyehun (2025) reference lists EMNLP 2025 with pages and city; verify the venue details are correct for the actual published version.

Circularity Check

0 steps flagged

No circular derivation: outcome and features are independently measured; the minor self-citations are background only.

full rationale

The central analysis is empirical and self-contained rather than derivational. Pedagogical quality is a separate four-dimension human annotation (mistake identification, mistake location, guidance, actionability) aggregated into a composite score, while the instructional and linguistic predictors come from external pre-trained classifiers (TalkMoves, Srinivasan & Choi politeness, Nikadon et al. agency) and standard text metrics (MTLD, Flesch–Kincaid, log length). The RQ3 regression demeans outcome and predictors within conversations to control for turn context, but this is a normalization, not a definition of the predictors in terms of the outcome; the coefficients are descriptive associations, not predictions forced by construction. The within-turn relative quality score is a re-centering of the annotation outcome, so it cannot manufacture feature–outcome relationships. The two self-citations (Zanotto & Aroyehun 2025; Lasser et al. 2025, including a co-author) appear in related-work motivation and as a methodological-consistency note; neither supplies the paper's findings, and deleting them would not alter the data analysis or conclusions. Remaining concerns about classifier calibration and domain shift on short human/LLM tutoring responses are measurement-validity risks, not circularity. Hence no circular step can be quoted with a specific reduction; score 1 reflects only the minor non-load-bearing self-citations.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No new theoretical entities or hand-set constants. The statistical model is descriptive; its coefficients are estimated, not imposed. The main assumptions concern construct validity of annotations and classifiers and the sufficiency of turn-level controls.

axioms (4)

domain assumption The four annotated dimensions (mistake identification, mistake location, guidance, actionability) capture perceived pedagogical quality.
The outcome variable is built entirely from this taxonomy (Section 3.1). No validation against expert ratings or learning outcomes is provided; the paper acknowledges this in Limitations.
domain assumption The pre-trained classifiers' probability outputs are valid interval-scale measures of the target features in this tutoring domain.
Section 3.2 uses TalkMoves, politeness, and agency classifier probabilities directly, without calibration or domain-shift checks.
domain assumption The dataset's expert and novice labels correctly identify skill levels.
The entire expert-vs-novice comparison depends on these labels being accurate, but the paper does not independently verify them.
domain assumption Within-conversation demeaning removes all relevant confounders; no response-level confounders (e.g., tutor type) need additional control.
Section 3.3 controls for conversation-level context but does not include tutor-type fixed effects, so unmeasured tutor-level confounds may bias the RQ3 coefficients.

reviewed 2026-08-03 · how reviews work

0 comments

Cite this review

Pith. "Pith review of Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles." pith.science (2026). https://pith.science/paper/OQUO73G7

@misc{pith2026251220780,
  author       = {Pith},
  title        = {Pith review of: Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OQUO73G7}},
  note         = {Machine review of arXiv:2512.20780}
}

Share X Bluesky LinkedIn Reddit HN

read the original abstract

Recent work has explored the use of large language models (LLMs) to generate tutoring responses in mathematics, yet it remains unclear how closely their instructional behavior aligns with expert human practice. We analyze a dataset of math remediation dialogues in which expert tutors, novice tutors, and seven LLMs of varying sizes, comprising both open-weight and commercial models, respond to the same student errors. We examine instructional strategies and linguistic characteristics of tutoring responses, including uptake (restating and revoicing), pressing for accuracy and reasoning, lexical diversity, readability, politeness, and agency. We find that expert tutors produce higher-quality responses than novices, and that larger LLMs generally receive higher pedagogical quality ratings than smaller models, approaching expert performance on average. However, LLMs exhibit systematic differences in their instructional profiles: they underuse discursive strategies characteristic of expert tutors while generating longer, more lexically diverse, and more polite responses. Regression analyses show that pressing for accuracy and reasoning, restating and revoicing, and lexical diversity, are positively associated with perceived pedagogical quality, whereas higher levels of agentic and polite language are negatively associated. These findings highlight the importance of analyzing instructional strategies and linguistic characteristics when evaluating tutoring responses across human tutors and intelligent tutoring systems.

Figures

Figures reproduced from arXiv: 2512.20780 by Ramatu Oiza Abdulsalam, Segun Aroyehun.

**Figure 1.** Figure 1: Instructional and linguistic profiles of tutors relative to expert tutor baseline. Each panel shows the [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

**Figure 2.** Figure 2: Relative pedagogical quality across tutors. Each datapoint shows average relative pedagogical quality for [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Figure 3: Instructional and linguistic correlates of pedagogical quality. Coefficients from an ordinary least squares [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith

[2023]

arxiv preprint.arXiv preprint arXiv:2310.06825, 100

Mistral 7b. arxiv preprint.arXiv preprint arXiv:2310.06825, 100. Sanjit Kakarla, Danielle Thomas, Jionghao Lin, Shiv- ang Gupta, and Kenneth R Koedinger. 2024. Using large language models to assess tutors’ performance in reacting to students making math errors.arXiv preprint arXiv:2401.03238. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and B...

Pith/arXiv arXiv 2024
[2025]

Can I ask what you thought about the third year for?

Comparing behavioral patterns of llm and human tutors: A population-level analysis with the cima dataset. InWorkshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics. Jana Lasser, Alina Herderich, Joshua Garland, Se- gun Taofeek Aroyehun, David Garcia, and Mirta Galesic. 2025. Collective moderation ...

Pith/arXiv arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.

[1] [2023]

arxiv preprint.arXiv preprint arXiv:2310.06825, 100

Mistral 7b. arxiv preprint.arXiv preprint arXiv:2310.06825, 100. Sanjit Kakarla, Danielle Thomas, Jionghao Lin, Shiv- ang Gupta, and Kenneth R Koedinger. 2024. Using large language models to assess tutors’ performance in reacting to students making math errors.arXiv preprint arXiv:2401.03238. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and B...

Pith/arXiv arXiv 2024

[2] [2025]

Can I ask what you thought about the third year for?

Comparing behavioral patterns of llm and human tutors: A population-level analysis with the cima dataset. InWorkshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics. Jana Lasser, Alina Herderich, Joshua Garland, Se- gun Taofeek Aroyehun, David Garcia, and Mirta Galesic. 2025. Collective moderation ...

Pith/arXiv arXiv 2025

Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles

The reading

Core claim §

What carries the argument

Load-bearing premise §

What would settle it §

If this is right

Where Pith is reading between the lines

discussion (0)

Core claim

Load-bearing premise

What would settle it