Automated Classification of Tutors' Dialogue Acts Using Generative AI: A Case Study Using the CIMA Corpus

Jiaqi Xu; Liqun He

REVIEW 4 major objections 5 minor 12 references

Automated Classification of Tutors' Dialogue Acts Using Generative AI: A Case Study Using the CIMA Corpus

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This case study claims that GPT-4, prompted with label definitions, step-by-step instructions, and two preceding turns, classifies tutor dialogue acts at 80% accuracy with Cohen's Kappa 0.74, without fine-tuning.

desk verdict Honest small prompt-engineering case study, but the headline 80% is the best condition picked on the same 80 test examples, so treat it as an upper bound; the Table 5 inconsistency in the internal note is a column-order misread. read the letter →

arxiv 2509.09125 v1 pith:NVX6IB6Z submitted 2025-09-11 cs.CL cs.AI

Liqun He , Jiaqi Xu This is my paper · ORCID

classification cs.CLcs.AI

keywords DialogueactclassificationGenerativeAIGPT-4PromptdesignAutomaticcodingEducationalanalysisCIMAcorpusTutoring

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that generative AI, specifically GPT-4, can classify tutors' dialogue acts (Question, Hint, Correction, Confirmation) with substantial agreement to human annotators, without fine-tuning, by using an appropriately designed prompt. On a balanced 80-sample subset of the CIMA tutoring corpus, the best prompt achieved 80% accuracy, weighted F1 0.81, and Cohen's Kappa 0.74. The authors argue this matters because it would let educators and researchers automate dialogue analysis without manual pre-annotation or model training. They also report that clear label definitions and conversational context drive the performance, while chain-of-thought steps help only marginally. The study is a small case study, and the authors explicitly flag that larger data and more categories are needed.

What carries the argument

The central object is a four-condition prompt design: the system message (basic, elaborative label definitions, chain-of-thought steps, or combined) and the user message's 'n' preceding turns (0, 1, or 2). The combined condition—step-by-step instructions plus explicit definitions of 'Question,' 'Hint,' 'Correction,' and 'Confirmation'—is what carries the argument. It is the mechanism that elicits the reported 80% accuracy from GPT-4. The other key component is the choice of evaluation metrics: accuracy, weighted F1, and Cohen's Kappa on a balanced 80-sample subset of the CIMA corpus.

What would settle it

Run the same combined prompt on the full CIMA corpus (or a fresh random sample that preserves the natural label distribution, including 'Others') and see whether GPT-4's accuracy and Cohen's Kappa stay near 0.80 and 0.74; if agreement drops substantially, the reported result is an artifact of balanced sampling rather than a general capability.

Watch

Extended reading notes

Core claim

The central claim is that a zero-shot GPT-4 model, given a specific prompt that defines each of the four dialogue acts and guides the model through steps, and given the tutor's response together with the preceding exchange, reproduces human dialogue-act labels on a balanced 80-sample subset of the CIMA corpus at 80% accuracy, weighted F1 of 0.81, and Cohen's Kappa of 0.74. The authors compare this favourably to a GloVe-based model that achieved F1 0.72 on a larger dataset, and emphasize that their result requires no manual pre-annotation and no fine-tuning. They also report that including label definitions and one preceding turn improved performance, while chain-of-thought instructions did n

Load-bearing premise

The entire performance claim rests on the assumption that the randomly selected, balanced 80-sample subset fairly represents the CIMA corpus's actual mix of tutor dialogue acts; if that sample is not representative, the 80% accuracy and kappa 0.74 do not generalize to the corpus or to practice.

Editorial extensions

If this is right

Dialogue-act coding for tutoring can be performed with natural-language prompts, eliminating the need for manually labelled training data or fine-tuning.
Including explicit definitions of each dialogue-act label in the system message improves accuracy compared to a bare label list.
Providing the previous student utterance (and sometimes an additional prior turn) as context improves performance over classifying the tutor turn alone.
The agreement level (kappa 0.74) is in the range normally described as 'substantial,' suggesting the model could serve as a second/co-coder in the traditional two-coder coding workflow, reducing human coding effort.
The open-source script lets others reproduce the experiments and test the approach on new tutoring corpora.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: Because the 80-sample test set is balanced (20 per label), the reported kappa may be optimistic for the natural CIMA label distribution; evaluating on a naturally distributed sample would likely lower the number and would clarify how much the result actually transfers.
Editorial inference: The paper shows that context helps but does not resolve how much context is optimal; a natural next test would vary n from 0 to 3 while measuring per-label errors to see whether 'Correction' needs more context or gets confused by it.
Editorial inference: The prompt template (definitions plus steps plus context) is likely portable to other educational coding schemes, but the paper only tests four categories; extending to a larger scheme would be the direct stress test of the claim.
Editorial inference: Because the models are used through an external API, the privacy concerns the paper raises suggest that real deployments with vulnerable student populations may require local or encrypted inference, not just anonymisation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a decent, honest case study, not a benchmark. The new bit is showing that prompt-based GPT-3.5/4 can tag tutor dialogue acts in the CIMA corpus without fine-tuning, and they ship the script. The ablations on label definitions and context are sensible, and the null result for chain-of-thought alone is a useful data point. The paper is transparent about its small sample and positions itself as a feasibility study.

The soft spots are real. The 80% accuracy / 0.81 F1 / 0.74 Kappa is the maximum over five GPT-4 conditions, and those five were selected after running twelve GPT-3.5 conditions on the same 80 balanced examples. So the number is selected on the test set, not a stable estimate for new tutoring dialogues. The balanced design also turns the weighted F1 into a macro average, and the GloVe baseline (F1 0.72) comes from a different sample, so the comparison is suggestive at best. No confidence intervals or error bars.

One correction to the stress-test note: the Table 5 inconsistency does not hold up. If you read the columns as per-label F1s, average F1, Cohen's k, accuracy, precision, recall, then the row for 4 combined / n=2 gives 0.80 accuracy, 0.81 F1, and 0.74 kappa, exactly as the abstract says. The table formatting is just easy to misread.

Who gets value from this: educational dialogue researchers who want a no-training coding tool and are willing to hand-check outputs, and anyone thinking about prompt selection pitfalls in small-N annotation studies. The paper is a reasonable workshop-grade contribution and deserves referee time if the journal is appropriate. I would ask the authors for a held-out evaluation, error bars, and a report of the unselected prompt performance; without that, the main claim stays conditional.

Referee Report

4 major / 5 minor

Summary. The paper investigates whether generative AI models (GPT-3.5-turbo and GPT-4) can classify tutors' dialogue acts (DAs) in the CIMA corpus into four categories (Question, Hint, Correction, Confirmation) without fine-tuning. It compares four prompt designs (basic, elaborative, chain-of-thought, combined) and three context windows (n=0, 1, 2) on an 80-sample balanced subset, reports F1 for the GPT-3.5 conditions, then tests the top five conditions with GPT-4. The headline claim is that a combined prompt with two preceding turns achieves 80% accuracy, weighted F1 of 0.81, and Cohen's Kappa of 0.74, outperforming a GloVe baseline. The paper also discusses prompt engineering insights, implications for educational research, limitations, and ethical considerations.

Significance. If the headline results are robust, the paper would make a useful contribution: it would show that LLM prompting can approximate human DA coding without a training phase, with openly shared scripts (GitHub) and a human-annotated corpus. The authors are appropriately transparent about the small sample and four-category limitation, and they engage with ethical issues. However, the central claim is currently not verifiable from the reported table, and the evaluation protocol has selection-on-test-set issues that materially affect the expected performance on new data. The balanced design and unmatched baseline further weaken the generalization claims. The paper's strengths are the direct empirical evaluation against human annotations, the systematic comparison of prompt conditions, and the availability of the script; its limitations are primarily in the inference and reporting of the headline numbers.

major comments (4)

[§5.1 and Table 5] The abstract and Section 5.1 state that the best condition ('4 combined / n=2') achieved 80% accuracy, weighted F1 0.81, and Cohen's Kappa 0.74. Table 5 does not contain this triple. The row '4 combined / n=2' is reported as 0.88 (presumably weighted F1), Cohen's kappa 0.86, accuracy 0.77, precision 0.71, recall 0.81, and per-label F1s 0.74/0.80/0.80/0.80. No row in Table 5 has A=0.80, weighted F1=0.81, and K=0.74. Please correct either the table or the text, and ensure all reported metrics are consistent.
[§3.3 and §4] All experiments were evaluated on the same 80 samples. The GPT-3.5 phase selected the top five of 12 conditions based on those samples, and the GPT-4 phase selected the best of those five on the same samples. The reported 80% accuracy is therefore the maximum over 17 evaluated conditions on one test set, not the expected performance of the chosen prompt. This selection-on-test-set inflation means the headline number is not a stable estimate. Please report a separate validation/test split, cross-validation, or at minimum confidence intervals and an explicit correction for multiple comparisons.
[§3.3 and §5.1] The 80-sample test set is balanced to 20 instances per label after excluding 'Others' and deduplicating. This does not represent the natural label distribution of the CIMA corpus. Consequently, the 'weighted F1' computed on this balanced set is effectively a macro-average and will not transfer to the corpus or to real tutoring dialogues if the true distribution is imbalanced. The paper should report the original label distribution, state whether the balanced sample is used for feasibility only, and avoid corpus-level generalization claims without a distribution-matched evaluation.
[§5.1] The claim 'increasing F1 scores from 0.72 to 0.81' compares the GPT-4 result on the 80-sample balanced subset with a GloVe F1 of 0.72 reported by Stasaski et al. (2020) on a different CIMA sample (2,296 samples per Section 2.3). This is not an apples-to-apples comparison. Either run the GloVe baseline on the same 80 items or clearly label the comparison as cross-study and not evidence of improvement on this test set.

minor comments (5)

[§3.3] Typo: 'phrases' should be 'phases' ('the study comprised two main phrases').
[Table 5] The table header is confusing: the column labeled 'avg.' and the per-label F1 columns are not clearly separated. Please re-format so that the reader can identify which number is weighted F1, accuracy, precision, recall, Cohen's kappa, and per-label F1.
[§3.1 vs Table 1] The label definition for 'Hint' in Section 3.1 ('scaffolds a student's understanding through providing hints') differs from the definition in Prompt 2/4 ('answering a student's question and/or providing additional information'). If the prompt definitions are the coding scheme, the discrepancy should be reconciled or explicitly acknowledged, since it could affect annotation alignment.
[§5.1] The phrase 'two preceding tutor-student turns (n=2)' is imprecise: Section 3.2 defines n=2 as the previous tutor-student exchange, i.e., two preceding turns plus the target response. Please use consistent terminology.
[Tables 2-5] No variance or repeat-run information is reported. Given the stochasticity of GPT models, at least a small number of repeated runs per condition would help assess stability; this is especially relevant because the best condition was selected from multiple runs.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical evaluation against independent human annotations.

full rationale

The paper is an empirical benchmark, not a derivation: GPT-4's outputs are compared with independent human annotations from the CIMA corpus. The only way the result could be circular would be if the prompt or evaluation were constructed from the gold labels in a way that forces the outcome. Here, the label definitions in the prompts are the corpus's coding scheme (the task definition), and the human labels are used only as ground truth for scoring. No parameters are fitted, no equations appear, and no load-bearing claim is justified by a self-citation. The reported accuracy/F1/Kappa may be optimistic because the same 80 samples were used to select the top prompt conditions (Tables 2 and 5), and the abstract's numbers do not match Table 5, but selection bias and reporting inconsistency are methodological/validity issues, not circularity. The paper itself acknowledges the small n (Section 5.3). Thus, no circular step can be exhibited under the required standard.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper's central claim rests on the quality of the CIMA ground-truth labels, the adequacy of the four-category scheme, the representativeness of the balanced 80-sample subset, and the stability of GPT-4 outputs. None of these are established by the paper itself; the balanced sampling issue in particular is not flagged as a limitation.

assumptions (4)

domain assumption Human annotations in the CIMA corpus are accurate ground truth for tutors' dialogue acts.
The paper uses pre-annotated labels as the reference standard for accuracy, F1, and Cohen's Kappa (Section 3.1), so if these labels are noisy, all reported agreements are upper bounds.
domain assumption The four DA categories (Question, Hint, Correction, Confirmation) sufficiently capture tutor intentions for the classification task.
The study excludes 'Others' (<1%) and does not test more complex coding schemes; Section 5.3 acknowledges this limitation.
domain assumption The balanced 80-sample subset (20 per label) is representative of the CIMA corpus for evaluating classification performance.
Section 3.3 describes random selection of 20 per label; the resulting uniform label distribution does not match the corpus's natural distribution, which affects weighted F1 and accuracy as estimates of real-world performance.
domain assumption GPT-4 API outputs are stable enough that performance on this 80-sample test set is a reliable estimate.
The paper does not report variance across runs or API temperature settings, yet generalizes from one evaluation pass (Sections 3.3-4).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automated Classification of Tutors' Dialogue Acts Using Generative AI: A Case Study Using the CIMA Corpus." pith.science (2026). https://pith.science/paper/NVX6IB6Z

@misc{pith2026250909125,
  author       = {Pith},
  title        = {Pith review of: Automated Classification of Tutors' Dialogue Acts Using Generative AI: A Case Study Using the CIMA Corpus},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NVX6IB6Z}},
  note         = {Machine review of arXiv:2509.09125}
}

read the original abstract

This study explores the use of generative AI for automating the classification of tutors' Dialogue Acts (DAs), aiming to reduce the time and effort required by traditional manual coding. This case study uses the open-source CIMA corpus, in which tutors' responses are pre-annotated into four DA categories. Both GPT-3.5-turbo and GPT-4 models were tested using tailored prompts. Results show that GPT-4 achieved 80% accuracy, a weighted F1-score of 0.81, and a Cohen's Kappa of 0.74, surpassing baseline performance and indicating substantial agreement with human annotations. These findings suggest that generative AI has strong potential to provide an efficient and accessible approach to DA classification, with meaningful implications for educational dialogue analysis. The study also highlights the importance of task-specific label definitions and contextual information in enhancing the quality of automated annotation. Finally, it underscores the ethical considerations associated with the use of generative AI and the need for responsible and transparent research practices. The script of this research is publicly available at https://github.com/liqunhe27/Generative-AI-for-educational-dialogue-act-tagging.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 2 linked inside Pith

[2]

However, when using the gpt-4 model (Table 5), there was some improvement with CoT

did not lead to a significant improvement and, at times, even resulted in a decline. However, when using the gpt-4 model (Table 5), there was some improvement with CoT. Due to the limited size of the experimental dataset, further investigation is needed to explore this aspect. 5.2 Implications The generative AI-based method proposed here offers a more eff...

2016
[3]

Correspondingly, we made changes in system messages for these three situations

n=2: inputting the tutor's response with the previous round of tutor-student exchange. Correspondingly, we made changes in system messages for these three situations. Specifically, in the 2nd and 4th system messages, we modified the expression of the first step, 'Identify student's utterance and tutor's response', as follows: • for n=0, it's now 'Identify...

2023
[4]

Choose ONLY ONE tag from 'Hint' and 'Correction' In Table 1, we illustrated four different system messages for prompting generative AI to classify tutors' DAs

Determine whether tutor is answering a student's question and/or providing additional information ('Hint') or is explicitly correcting student's mistake ('Correction'). Choose ONLY ONE tag from 'Hint' and 'Correction' In Table 1, we illustrated four different system messages for prompting generative AI to classify tutors' DAs. Specifically, Prompt 1 (Basi...

2023
[5]

In this phase, we conducted experiments to evaluate the performance of the 'gpt-4' model under the top five performing conditional combinations identified in the first phase. The results show that under the best-performing condition (utilising a combined system message with the previous tutor-student exchange), we were able to accurately assign 80% labels...

2023
[7]

Table 3 - Impact of System Messages on Model Performance Experimental Conditions f1-score Confirmation Correction Hint Question avg

did not yield a particularly significant improvement. Table 3 - Impact of System Messages on Model Performance Experimental Conditions f1-score Confirmation Correction Hint Question avg. 1 Basic 0.67 0.56 0.10 0.49 0.46 2 elaborative 0.76 0.75 0.51 0.72 0.69 3 chain-of-thought 0.65 0.68 0.00 0.60 0.48 4 combined 0.72 0.69 0.53 0.69 0.66 The Table 4 displa...

2022
[10]

To mitigate these risks, researchers should follow data minimisation principles by collecting only essential personal information

or be accessed by external parties (Stahl & Eke, 2024). To mitigate these risks, researchers should follow data minimisation principles by collecting only essential personal information. Additionally, anonymisation or pseudonymisation should be employed to remove or mask any identifiers that may reveal individual identities (Wright, 2011). Transparency an...

2024
[12]

Morgan, D. L. (2023). Exploring the Use of Artificial Intelligence for Qualitative Data Analysis: The Case of ChatGPT. International Journal of Qualitative Methods, 22, 16094069231211248. https://doi.org/10.1177/16094069231211248 OpenAI. (2023). OpenAI Platform. https://platform.openai.com Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mis...

arXiv 2023
[1969]

the form of utterances), but on why it is said (the function of the utterances)

Thus, in analysing tutoring dialogue, the emphasis lies not only on what is said (i.e. the form of utterances), but on why it is said (the function of the utterances). 'Dialogue act' (DA), also known as 'speech act', is used to represent the hidden intention of the functioning of the utterance (Traum, 1999). According to Speech Act theory (Searle, 1965, 1...

1999

Show all 12 references

[2001]

and through various forms of dialogue, tutors achieve various pedagogical goals (Roscoe & Chi, 2007). Thus, a systematic examination of the tutoring dialogue between teachers and students is necessary for understanding and enhancing the quality of online tutoring (Chen et al.,...

2007
[2011]

and makes it difficult to identify and address potential biases within the data, such as those related to race, gender, or sexuality (Morgan, 2023). Researchers must acknowledge these limitations and communicate them clearly to participants, while also advocating for stronger ...

2023 arXiv
[2014]

All of these models are designed to help machines translate human language into a format they can understand

and Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019). All of these models are designed to help machines translate human language into a format they can understand. Subsequently, through fine-tuning, machines can ultimately acquire the abilit...

2019
[2023]

The final version will be available open access via the journal. Title Automated Classification of Tutors' Dialogue Acts Using Generative AI: A Case Study Using the CIMA Corpus Authors 1/ Liqun He (Corresponding) Institution: University College London (Gower Street, London, WC...

2001

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [2]

However, when using the gpt-4 model (Table 5), there was some improvement with CoT

did not lead to a significant improvement and, at times, even resulted in a decline. However, when using the gpt-4 model (Table 5), there was some improvement with CoT. Due to the limited size of the experimental dataset, further investigation is needed to explore this aspect. 5.2 Implications The generative AI-based method proposed here offers a more eff...

2016

[2] [3]

Correspondingly, we made changes in system messages for these three situations

n=2: inputting the tutor's response with the previous round of tutor-student exchange. Correspondingly, we made changes in system messages for these three situations. Specifically, in the 2nd and 4th system messages, we modified the expression of the first step, 'Identify student's utterance and tutor's response', as follows: • for n=0, it's now 'Identify...

2023

[3] [4]

Choose ONLY ONE tag from 'Hint' and 'Correction' In Table 1, we illustrated four different system messages for prompting generative AI to classify tutors' DAs

Determine whether tutor is answering a student's question and/or providing additional information ('Hint') or is explicitly correcting student's mistake ('Correction'). Choose ONLY ONE tag from 'Hint' and 'Correction' In Table 1, we illustrated four different system messages for prompting generative AI to classify tutors' DAs. Specifically, Prompt 1 (Basi...

2023

[4] [5]

In this phase, we conducted experiments to evaluate the performance of the 'gpt-4' model under the top five performing conditional combinations identified in the first phase. The results show that under the best-performing condition (utilising a combined system message with the previous tutor-student exchange), we were able to accurately assign 80% labels...

2023

[5] [7]

Table 3 - Impact of System Messages on Model Performance Experimental Conditions f1-score Confirmation Correction Hint Question avg

did not yield a particularly significant improvement. Table 3 - Impact of System Messages on Model Performance Experimental Conditions f1-score Confirmation Correction Hint Question avg. 1 Basic 0.67 0.56 0.10 0.49 0.46 2 elaborative 0.76 0.75 0.51 0.72 0.69 3 chain-of-thought 0.65 0.68 0.00 0.60 0.48 4 combined 0.72 0.69 0.53 0.69 0.66 The Table 4 displa...

2022

[6] [10]

To mitigate these risks, researchers should follow data minimisation principles by collecting only essential personal information

or be accessed by external parties (Stahl & Eke, 2024). To mitigate these risks, researchers should follow data minimisation principles by collecting only essential personal information. Additionally, anonymisation or pseudonymisation should be employed to remove or mask any identifiers that may reveal individual identities (Wright, 2011). Transparency an...

2024

[7] [12]

Morgan, D. L. (2023). Exploring the Use of Artificial Intelligence for Qualitative Data Analysis: The Case of ChatGPT. International Journal of Qualitative Methods, 22, 16094069231211248. https://doi.org/10.1177/16094069231211248 OpenAI. (2023). OpenAI Platform. https://platform.openai.com Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mis...

arXiv 2023

[8] [1969]

the form of utterances), but on why it is said (the function of the utterances)

Thus, in analysing tutoring dialogue, the emphasis lies not only on what is said (i.e. the form of utterances), but on why it is said (the function of the utterances). 'Dialogue act' (DA), also known as 'speech act', is used to represent the hidden intention of the functioning of the utterance (Traum, 1999). According to Speech Act theory (Searle, 1965, 1...

1999