How Real Is AI Tutoring? Comparing Simulated and Human Dialogues in One-on-One Instruction

Bo Jiang; Jiatong Wang; Ruijia Li; Yuan-Hao Jiang

REVIEW 3 major objections 5 minor 6 references

How Real Is AI Tutoring? Comparing Simulated and Human Dialogues in One-on-One Instruction

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that AI-simulated one-on-one tutoring dialogues are structurally different from human ones: humans use a question-factual response-feedback loop, while AI uses an explanation-simplistic response loop.

desk verdict Plausible and useful matched comparison, but the headline ENA contrast rests on a BERT coder applied to AI text with no reported reliability—needs that evidence before the specific loop claim is taken as established. read the letter →

arxiv 2509.01914 v1 pith:UXYJVBCV submitted 2025-09-02 cs.AI cs.CLcs.MA

Ruijia Li , Yuan-Hao Jiang , Jiatong Wang , Bo Jiang This is my paper · ORCID

classification cs.AIcs.CLcs.MA

keywords AItutoringdialoguestructureInitiation-Response-Feedback(IRF)EpistemicNetworkAnalysisLLMsimulationone-on-oneinstructionSocraticquestioninggenerativeeducational

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large-language-model-generated teacher-student dialogues reproduce the structure of real one-on-one tutoring. Comparing authentic human dialogues with AI-simulated dialogues on the same math tutoring prompts, it finds they do not: human tutors initiate more, ask more questions, give more general feedback, and draw out longer factual student answers, while AI tutors explain more and receive short confirmatory replies. The signature result is a statistically separated pair of interaction loops in the network analysis: human dialogue centers on a question-factual response-feedback loop, AI dialogue on an explanation-simplistic response loop. If true, the finding gives a concrete target for improving generative educational dialogue systems—synthetic tutoring data is not just slightly different but structurally shallower—and offers a quantitative way to evaluate progress.

What carries the argument

The analysis rests on an IRF-based coding scheme (Initiation-Response-Feedback) with ten subcategories—Questioning, Hints, Modeling; Refusal, Simplistic Response, Factual Response, Open-ended Response; Feeding Back, Instructing, Explaining—combined with Epistemic Network Analysis, a method that projects how often coded behaviors co-occur into a low-dimensional space. The load-bearing edges are the I-Q↔R-FR connection in human dialogues and the F-E↔R-SR connection in AI dialogues; the separation of the two networks' centroids along the X-axis is the quantitative evidence for a fundamental divergence.

What would settle it

A concrete falsifying check: have trained human coders independently code the same AI-simulated dialogues with the IRF scheme, then compare their codes with the automatic coder's on the F-E and R-SR categories. If agreement is poor on AI text and the ENA networks built from human codes no longer separate along the X-axis (or the F-E↔R-SR connection weakens), the claimed fundamental divergence is an artifact of automated coding.

Watch

Extended reading notes

Core claim

The central claim is that the difference between AI-simulated and human tutoring is not merely fluency or utterance length but the underlying interaction structure. In the coded behavior space, human dialogues are organized around a tightly coupled Questioning–Factual Response pair (I-Q with R-FR), with feedback closing the loop; AI dialogues are organized around Explaining–Simplistic Response (F-E with R-SR). The network centroids separate significantly along one core dimension (t(84.35)=9.33, p<0.001, d=1.97), which the authors interpret as a single axis separating question-centered guided instruction from explanation-centered information transfer. The paper concludes that human dialogue i

Load-bearing premise

The argument assumes the automatic coder, trained on human-labelled dialogues, labels AI-simulated dialogues without systematic bias; if it tags AI explanations and short replies differently for tool reasons, the central contrast could be an artifact of the coding rather than of the dialogues.

Editorial extensions

If this is right

LLM-generated tutoring corpora should not be treated as drop-in substitutes for human teaching data in training or evaluation, because their interaction structure is demonstrably different.
Evaluation of generative educational dialogue systems should measure interaction-loop structure—whether the system supports question → factual response → feedback rather than explanation → simple acknowledgment—not just linguistic fluency.
The deficits are specific: AI dialogues are weakest in initiation, questioning, and general feedback, while strong in explanation, pointing to concrete behaviors to target with prompt design or fine-tuning.
The IRF+ENA signatures give a quantitative benchmark for tracking whether future AI tutoring systems become more human-like as their behavior changes.
The absence of the question-factual response-feedback loop is a measurable warning sign that an AI tutor is likely providing information transfer rather than Socratic cognitive guidance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: The same IRF+ENA contrast could serve as an automated outcome metric for attempts to improve AI tutoring—e.g., fine-tuning or preference-optimizing toward I-Q→R-FR transitions—rather than only measuring fluency or task success.
Editorial inference: Because the simulation was seeded with each human dialogue's distilled core tutoring approach, the observed divergence may understate how far an unconstrained LLM tutor would drift from human structure; a free-instructed condition would reveal whether the constraint helps or hides the gap.
Editorial inference: The coding pipeline is the most testable seam—a blind sample of AI dialogues double-coded by trained humans would separate real structural differences from classifier bias and either strengthen or qualify the central claim.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this paper offers a genuinely useful matched comparison and a plausible qualitative conclusion—human tutors scaffold by questioning, AI tutors tend to explain and wait for a simple acknowledgment. But the specific ENA structure—the 'explanation-simplistic response' loop—depends on a BERT coder applied to AI-simulated text with no reported reliability. That is the soft spot that matters.

What's new: the authors took 49 real one-on-one math tutoring dialogues, extracted the same teaching prompts, and used a three-agent LLM framework to generate matched AI dialogues. Both corpora were coded with the same fine-grained IRF scheme, then analyzed with ENA. The paired design and the focus on interactional loops rather than just frequency counts are a step up from earlier descriptive comparisons. The t-tests and centroid separation are reported with effect sizes. And the qualitative framing—human dialogue as cognitive guidance vs. AI as information transfer—matches what many in the field suspected.

Where I'd push back: Section 3.4 says human dialogues were coded by two researchers with Kappa .824, then a BERT model fine-tuned on those codes was used for the AI dialogues, with only 'manual verification.' No inter-coder reliability, confusion matrix, or error analysis for the AI text is given. Since AI-generated language is systematically different—more formulaic explanations, shorter student replies—the coder could be biased in exactly the direction of the headline finding. The paired t-tests in Table 2 inherit that risk because the proportions come from the same coder. This isn't about honesty; it's about whether the measurement instrument transfers.

A second, smaller issue: human transcripts were polished with GPT before coding. That could smooth away the very disfluencies that distinguish human talk from AI talk, though it is unlikely to create the whole pattern. The sample is small (49 dialogues) and only one LLM and one framework, so generalizability is limited. The authors acknowledge some limitations, but not the coder transfer problem.

Overall, this is worth engaging with. The framework is a solid contribution, and the central intuition is probably right. But the specific ENA loop should be treated as provisional until the authors show the BERT coder works on AI-simulated text—say, by human-coding a subsample and reporting reliability. I'd send it out for peer review with a clear request for that evidence.

Referee Report

3 major / 5 minor

Summary. The paper compares 49 authentic human one-on-one math tutoring dialogues (fifth-grade students; university volunteer tutors; transcripts polished with GPT) with 49 AI-simulated dialogues generated using GPT-4o and a three-agent SocraticLM framework, with the same tutoring questions and a 'core tutoring approach' distilled from each human dialogue as inputs. Both corpora were coded with a nine-code IRF scheme: human dialogues were double-coded by human annotators (Cohen's κ = 0.824), while AI dialogues were coded by a BERT classifier fine-tuned on the human-coded data, with an unspecified 'manual verification.' The authors then ran paired t-tests on behavioral proportions and Epistemic Network Analysis (ENA). Results show that human dialogues have longer utterances and higher proportions of I-Q, R-FR, and F-F, whereas AI dialogues have higher R-SR, R-RR, and F-E. ENA reveals a large X-axis centroid separation (t(84.35)=9.33, p<0.001, d=1.97), with human networks centered on an I-Q↔R-FR connection and AI networks on an F-E↔R-SR connection. The paper concludes that human tutoring is cognitively guided, while AI tutoring is essentially an information-transfer loop. The authors acknowledge limitations regarding the tutor sample, simulated student diversity, and the entanglement of LLM and agent design.

Significance. If the measurement pipeline is valid for both corpora, this is a valuable empirical contribution: it uses a paired design that controls for problem content, applies a well-established IRF framework, and demonstrates ENA as a visualization and inference tool for comparing human and AI educational dialogue. The large effect sizes and the clear separation of interactional patterns make the finding easy to communicate to the AIED community. The paper is also honest about several limitations (student tutors, one LLM/agent framework, simplified AI student). However, the central result rests on two unvalidated preprocessing/coding steps: the BERT coder's reliability on AI-generated text and the GPT-based polishing of human transcripts. These concerns are not peripheral; they directly affect the headline comparisons in Table 2 and Figures 3–4. The manuscript is therefore a promising but not yet fully supported contribution; the required additional validation is feasible within the scope of a revision.

major comments (3)

[§3.4 Data analysis] The reliability evidence reported for human coding does not transfer to the AI-simulated corpus. Cohen's κ = 0.824 applies to two human coders on human dialogues; the AI dialogues were then coded by a BERT model fine-tuned on human-coded examples, and the 'manual verification' is not quantified. Because AI-generated text is systematically different from human transcripts, the coder can mislabel specific codes in a consistent direction — for example, over-assigning F-E to AI teacher explanations or R-SR to AI student replies. Table 2 and the ENA networks in Figures 3–4 are computed entirely from these codes, so the central 'explanation–simplistic response' loop may in part be a coding artifact. Please report a human-coded reliability sample for AI dialogues: per-code precision/recall or a confusion matrix, plus Kappa, and state how disagreements with the 'manual verification' were resolve
[§3.1 Participants and experiment procedure] All human transcripts were 'refined sentence by sentence using GPT-based text polishing' before coding. This preprocessing can inflate measured utterance lengths and rewrite or remove short, disfluent student turns such as 'mm,' 'okay,' and hesitations — exactly the kinds of utterances coded as R-SR. The human-vs-AI differences in utterance length and in R-SR/F-F proportions could therefore be partly introduced by the polishing step rather than by authentic interaction. Please quantify this effect: e.g., compare original ASR transcripts with the polished versions on a subset, or code the raw turns most likely to be affected, and report the polishing model/version and the editing criteria.
[§3.2 Simulation Data Generation] The paper does not specify how the 'core tutoring approach' was extracted from each human dialogue, how many AI dialogues were generated (presumably 49, but this is not stated), or the generation parameters (model version, temperature, sampling, number of runs, prompt template). If each AI dialogue is paired with a human dialogue via the extracted 'core tutoring approach,' the paired t-tests in Table 2 need this pairing made explicit. More generally, the abstract and conclusion claim a 'fundamental divergence' for 'AI tutoring' as a whole, but the evidence comes from one LLM (GPT-4o), one three-agent framework, and one prompt template. The Discussion does acknowledge this entanglement, but the claims in the abstract should be scoped to the tested configuration. Please report the full generation protocol so that readers can assess reproducibility and generality.

minor comments (5)

[Table 2] Ten paired t-tests are run without a multiple-comparison correction; report adjusted p-values or an FDR procedure, and include effect sizes (e.g., Cohen's d) for each behavioral subtype. Also clarify whether the pairs are (human dialogue i, AI dialogue generated from i).
[Figure 3 / ENA settings] Report the ENA parameters used in the ENA Web Toolkit: window size, co-occurrence threshold, normalization, and rotation method. State how many 'lines' or units were used in the t-test for centroid separation (the df = 84.35 suggests a Welch test; please specify the unit of analysis).
[§3.1, GPT polishing] The phrase 'GPT-based text polishing' is underspecified. State which model version was used and whether polishing was done automatically or with human oversight; this matters for reproducibility and for the measurement concern above.
[References] The paper cites 'Hila, A. (2025)' as the source for ENA, but this reference does not appear to be the methodological origin of Epistemic Network Analysis. Please cite the canonical ENA methodology papers (e.g., Shaffer et al.) in addition to any application-specific reference.
[§3.2 / Figure 1] The 'core tutoring approach' is a key input to the simulation but is never defined. Provide an example or a formal description of what the distilled approach contains (e.g., sequence of pedagogical moves, solution steps) to support replication.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical comparison whose coding and analysis are independent of its conclusions; the BERT-coding validity concern is a measurement risk, not a circular derivation.

full rationale

The paper compares human and AI-simulated tutoring dialogues through IRF coding and ENA. Human dialogues were coded by two human researchers with Cohen's Kappa 0.824; AI dialogues were coded by a BERT model fine-tuned on those human codes, with manual verification. This is a measurement pipeline, not a derivation in which a fitted parameter determines the outcome. The AI dialogues were generated using the human dialogue's question and 'core tutoring approach' as constraints, which would bias the simulation toward similarity, not toward the observed differences; the reported differences are therefore not forced by construction. The IRF coding scheme defines categories (I-Q, R-FR, F-E, R-SR), but it does not logically determine which categories will have higher frequencies or which ENA edges will be strongest. The ENA centroids and network structures are computed from the coded data, and the 'question-factual response-feedback' versus 'explanation-simplistic response' interpretation is a post-hoc characterization of observed patterns, not an algebraic consequence of the coding definitions. Self-citations in the paper appear in related work and contextualization, and none are load-bearing for the central claim. The lack of reported inter-coder reliability or confusion matrices for the BERT coder applied to AI-simulated text is a legitimate external-validity concern about possible systematic misclassification, but it is not a circularity: no equation in the paper reduces the central result to its own inputs. Thus the derivation chain is self-contained and the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim depends on the validity of the coding framework, the transferability of the BERT coder to AI text, the neutrality of GPT-based transcript polishing, and the representativeness of the single simulation design. No free parameters are explicitly fitted to produce the main result; the BERT model is a fitted tool whose parameters are not reported.

assumptions (5)

domain assumption The IRF coding framework with the eight subcategories is a valid representation of pedagogical dialogue quality for both human and AI dialogues.
Adopted from Van de Pol et al. (2010) and Yang et al. (2023), but no independent validation that the coding captures pedagogical value across both dialogue types.
domain assumption The BERT model fine-tuned on human-coded dialogues generalizes to AI-simulated dialogues without distribution-shift bias.
Section 3.4: the fine-tuned BERT automatically codes AI dialogues, with manual verification but no kappa or error analysis reported for out-of-domain text.
domain assumption GPT-based polishing of human transcripts does not systematically alter the behavioral codes or utterance lengths measured.
Section 3.1: 'refined sentence by sentence using GPT-based text polishing', but no check on how this modifies dialogue features.
domain assumption The SocraticLM tripartite simulation framework with GPT-4o is representative of current AI tutoring dialogue generation.
Section 3.2: single framework, single LLM; authors acknowledge the AI's behavior is an entangled product of the LLM and agent design.
domain assumption Extracting the 'core tutoring approach' from each human dialogue and using it as simulation input does not bias the comparison.
Section 3.2: the extraction procedure is not specified in detail, and its effect on the generated dialogues is not evaluated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of How Real Is AI Tutoring? Comparing Simulated and Human Dialogues in One-on-One Instruction." pith.science (2026). https://pith.science/paper/UXYJVBCV

@misc{pith2026250901914,
  author       = {Pith},
  title        = {Pith review of: How Real Is AI Tutoring? Comparing Simulated and Human Dialogues in One-on-One Instruction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UXYJVBCV}},
  note         = {Machine review of arXiv:2509.01914}
}

read the original abstract

Heuristic and scaffolded teacher-student dialogues are widely regarded as critical for fostering students' higher-order thinking and deep learning. However, large language models (LLMs) currently face challenges in generating pedagogically rich interactions. This study systematically investigates the structural and behavioral differences between AI-simulated and authentic human tutoring dialogues. We conducted a quantitative comparison using an Initiation-Response-Feedback (IRF) coding scheme and Epistemic Network Analysis (ENA). The results show that human dialogues are significantly superior to their AI counterparts in utterance length, as well as in questioning (I-Q) and general feedback (F-F) behaviors. More importantly, ENA results reveal a fundamental divergence in interactional patterns: human dialogues are more cognitively guided and diverse, centered around a "question-factual response-feedback" teaching loop that clearly reflects pedagogical guidance and student-driven thinking; in contrast, simulated dialogues exhibit a pattern of structural simplification and behavioral convergence, revolving around an "explanation-simplistic response" loop that is essentially a simple information transfer between the teacher and student. These findings illuminate key limitations in current AI-generated tutoring and provide empirical guidance for designing and evaluating more pedagogically effective generative educational dialogue systems.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages

[1]

Learning-by-teaching with ChatGPT: The effect of teachable ChatGPT agent on programming education

Alexander,R.J.(2008). Towards dialogic teaching: Rethinking classroom talk. Alisoy, H. (2025). From Echo Chambers to Critical Dialogue: A Comparative Case Study of Social Media-Based Pedagogy For Addressing Scientific Misinformation. Global Spectrum of Research and Humanities, 2(4),35–47.https://doi.org/10.69760/gsrh.0250203002 Cavagnetto,A.,Hand,B.M.,&No...

work page Pith review arXiv 2008
[4]

Distinct interactional patterns of human and AI dialogues Jiang, B. et al. (Eds.) (2025). Proceedings of the 33rd International Conference on Computers in Education. Asia-Pacific Society for Computers in Education Cuskley, C.,Woods, R., & Flaherty, M. (2024). Thelimitations of large language models for understanding humanlanguageandcognition. Open Mind, 8...

work page 2025
[6]

Waring, H. Z. (2008). Using explicit positive assessment in the language classroom: IRF, feedback, and learningopportunities. The Modern Language Journal, 92(4),577–594. Waring,H.Z.(2009). Movingoutof IRF (Initiation-Response-Feedback):A singlecaseanalysis. Language Learning, 59(4),796–824. Wei,Y.,Qi,C.,Jiang,Y.-H.,&Dai,L.(Eds.).(2024). Enhancing Educatio...

work page doi:10.52305/ruig5131 2008
[12]

Wang, W., Zheng, J., Song, L., Tu, Y., & Gao, P. (2021). Test Assembly for Cognitive Diagnosis Using Mixed-IntegerLinearProgramming. Frontiers in Psychology,

work page 2021
[20]

AI & SOCIETY

Hila,A.(2025).Theepistemologicalconsequencesoflargelanguagemodels:Rethinkingcollectiveintelligence andinstitutionalknowledge. AI & SOCIETY. Hong, H., Dai, L., & Zheng, X. (2025). Advances in Wearable Sensors for Learning Analytics: Trends, Challenges,andProspects. Sensors, 25(9),Article9.https://doi.org/10.3390/s25092714 Jiang,Y.-H.,Chen,Z.-W.,Zhao,C.,Tan...

work page arXiv 2025
[829]

(2024).ECC+ +:An algorithmfamilybasedon ensembleof classifier chains for classifying imbalanced multi-label data

Duan,J.,Gu,Y.,Yu,H.,Yang,X.,& Gao,S. (2024).ECC+ +:An algorithmfamilybasedon ensembleof classifier chains for classifying imbalanced multi-label data. Expert Systems with Applications, 236, 121366.https://doi.org/10.1016/j.eswa.2023.121366 Duan, J., Shen, H.-J., Duan, B.-M., Wang, Q., & Jiang, Y.-H. (2024). EFDR-CI: An Ensemble Learning FrameworkforAssess...

work page arXiv 2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Learning-by-teaching with ChatGPT: The effect of teachable ChatGPT agent on programming education

Alexander,R.J.(2008). Towards dialogic teaching: Rethinking classroom talk. Alisoy, H. (2025). From Echo Chambers to Critical Dialogue: A Comparative Case Study of Social Media-Based Pedagogy For Addressing Scientific Misinformation. Global Spectrum of Research and Humanities, 2(4),35–47.https://doi.org/10.69760/gsrh.0250203002 Cavagnetto,A.,Hand,B.M.,&No...

work page Pith review arXiv 2008

[2] [4]

Distinct interactional patterns of human and AI dialogues Jiang, B. et al. (Eds.) (2025). Proceedings of the 33rd International Conference on Computers in Education. Asia-Pacific Society for Computers in Education Cuskley, C.,Woods, R., & Flaherty, M. (2024). Thelimitations of large language models for understanding humanlanguageandcognition. Open Mind, 8...

work page 2025

[3] [6]

Waring, H. Z. (2008). Using explicit positive assessment in the language classroom: IRF, feedback, and learningopportunities. The Modern Language Journal, 92(4),577–594. Waring,H.Z.(2009). Movingoutof IRF (Initiation-Response-Feedback):A singlecaseanalysis. Language Learning, 59(4),796–824. Wei,Y.,Qi,C.,Jiang,Y.-H.,&Dai,L.(Eds.).(2024). Enhancing Educatio...

work page doi:10.52305/ruig5131 2008

[4] [12]

Wang, W., Zheng, J., Song, L., Tu, Y., & Gao, P. (2021). Test Assembly for Cognitive Diagnosis Using Mixed-IntegerLinearProgramming. Frontiers in Psychology,

work page 2021

[5] [20]

AI & SOCIETY

Hila,A.(2025).Theepistemologicalconsequencesoflargelanguagemodels:Rethinkingcollectiveintelligence andinstitutionalknowledge. AI & SOCIETY. Hong, H., Dai, L., & Zheng, X. (2025). Advances in Wearable Sensors for Learning Analytics: Trends, Challenges,andProspects. Sensors, 25(9),Article9.https://doi.org/10.3390/s25092714 Jiang,Y.-H.,Chen,Z.-W.,Zhao,C.,Tan...

work page arXiv 2025

[6] [829]

(2024).ECC+ +:An algorithmfamilybasedon ensembleof classifier chains for classifying imbalanced multi-label data

Duan,J.,Gu,Y.,Yu,H.,Yang,X.,& Gao,S. (2024).ECC+ +:An algorithmfamilybasedon ensembleof classifier chains for classifying imbalanced multi-label data. Expert Systems with Applications, 236, 121366.https://doi.org/10.1016/j.eswa.2023.121366 Duan, J., Shen, H.-J., Duan, B.-M., Wang, Q., & Jiang, Y.-H. (2024). EFDR-CI: An Ensemble Learning FrameworkforAssess...

work page arXiv 2024