{"id":"47b13f87-436b-4392-9df4-2f8d8d9df4bb","arxiv_id":"2509.09125","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GPT-4, prompted with label definitions and conversation context, classified tutor dialogue acts in the CIMA corpus with 80% accuracy and Cohen's Kappa 0.74 without fine-tuning.","lead":"This paper tests whether OpenAI's GPT models can automatically label tutors' dialogue moves (questions, hints, corrections, confirmations) in the open CIMA tutoring corpus using carefully designed prompts instead of model training. The best GPT-4 prompt reached 80% accuracy and substantial agreement with human coders on a small 80-utterance sample, suggesting a cheaper, faster alternative to manual coding for education researchers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 80%/0.74 headline is not reproduced by Table 5 and is the maximum over prompt conditions selected on the same 80 balanced samples; a fresh held-out evaluation is needed.","rationale":"The reader correctly identifies the balanced 80-sample subset as a load-bearing limitation affecting generalization. My stress-test agrees with that but points to an even more immediate problem: the paper's own results table does not contain the headline numbers, and the reported best result was selected as the maximum over multiple prompt conditions evaluated on the same 80 samples. Both issues are addressable: correct the table if it is a transcription error, and add a pre-registered evaluation on a fresh naturally distributed sample. These are consistent with a CONDITIONAL verdict rather than ACCEPT or REJECT, so I leave the reader's verdict unchanged.","tokens_in":9539,"tokens_out":13517,"duration_ms":125831,"concrete_test":"Use the public script to run the exact 'combined / n=2' prompt on a fresh, naturally distributed held-out sample of at least 150 examples drawn from the CIMA corpus (after the same exclusion of 'Others' and deduplication), with no further prompt selection. Compute accuracy, weighted F1, Cohen's Kappa, and 95% bootstrap CIs, and also verify whether the script reproduces Table 5 or §5.1's numbers. If the held-out accuracy/Kappa fall substantially below 80%/0.74 (or if the script reproduces Table 5 rather than the text), the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §5.1 (80% accuracy, weighted F1 0.81, K=0.74 for '4 combined / n=2') cannot be verified from the paper's own Table 5: the row for that condition is printed as f1-score 0.88, Cohen's k 0.86, A 0.77, with per-label F1s 0.74/0.80/0.80/0.80, and no row matches the abstract's triple. Even setting aside this table inconsistency, the number is not a stable estimate of the combined prompt: §3.3 reduced CIMA to 80 balanced examples (20 per label), and the same 80 examples were used both to select the top five gpt-3.5 conditions and to pick the best gpt-4 condition in Table 5. Reporting the maximum over these comparisons on the same test set inflates accuracy relative to what a user would get on a new tutoring dialogue; the balanced design also overweights rare labels, so the 'weighted F1' is effectively a macro average and does not transfer to CIMA's natural distribution. No confidence intervals or error bars are provided, and the baseline comparison (GloVe F1=0.72 from a different CIMA sample) is not on the same evaluation set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether generative AI models (GPT-3.5-turbo and GPT-4) can classify tutors' dialogue acts (DAs) in the CIMA corpus into four categories (Question, Hint, Correction, Confirmation) without fine-tuning. It compares four prompt designs (basic, elaborative, chain-of-thought, combined) and three context windows (n=0, 1, 2) on an 80-sample balanced subset, reports F1 for the GPT-3.5 conditions, then tests the top five conditions with GPT-4. The headline claim is that a combined prompt with two preceding turns achieves 80% accuracy, weighted F1 of 0.81, and Cohen's Kappa of 0.74, outperforming a GloVe baseline. The paper also discusses prompt engineering insights, implications for educational research, limitations, and ethical considerations.","tokens_in":9867,"tokens_out":3940,"duration_ms":42639,"significance":"If the headline results are robust, the paper would make a useful contribution: it would show that LLM prompting can approximate human DA coding without a training phase, with openly shared scripts (GitHub) and a human-annotated corpus. The authors are appropriately transparent about the small sample and four-category limitation, and they engage with ethical issues. However, the central claim is currently not verifiable from the reported table, and the evaluation protocol has selection-on-test-set issues that materially affect the expected performance on new data. The balanced design and unmatched baseline further weaken the generalization claims. The paper's strengths are the direct empirical evaluation against human annotations, the systematic comparison of prompt conditions, and the availability of the script; its limitations are primarily in the inference and reporting of the headline numbers.","major_comments":[{"comment":"The abstract and Section 5.1 state that the best condition ('4 combined / n=2') achieved 80% accuracy, weighted F1 0.81, and Cohen's Kappa 0.74. Table 5 does not contain this triple. The row '4 combined / n=2' is reported as 0.88 (presumably weighted F1), Cohen's kappa 0.86, accuracy 0.77, precision 0.71, recall 0.81, and per-label F1s 0.74/0.80/0.80/0.80. No row in Table 5 has A=0.80, weighted F1=0.81, and K=0.74. Please correct either the table or the text, and ensure all reported metrics are consistent.","section":"§5.1 and Table 5"},{"comment":"All experiments were evaluated on the same 80 samples. The GPT-3.5 phase selected the top five of 12 conditions based on those samples, and the GPT-4 phase selected the best of those five on the same samples. The reported 80% accuracy is therefore the maximum over 17 evaluated conditions on one test set, not the expected performance of the chosen prompt. This selection-on-test-set inflation means the headline number is not a stable estimate. Please report a separate validation/test split, cross-validation, or at minimum confidence intervals and an explicit correction for multiple comparisons.","section":"§3.3 and §4"},{"comment":"The 80-sample test set is balanced to 20 instances per label after excluding 'Others' and deduplicating. This does not represent the natural label distribution of the CIMA corpus. Consequently, the 'weighted F1' computed on this balanced set is effectively a macro-average and will not transfer to the corpus or to real tutoring dialogues if the true distribution is imbalanced. The paper should report the original label distribution, state whether the balanced sample is used for feasibility only, and avoid corpus-level generalization claims without a distribution-matched evaluation.","section":"§3.3 and §5.1"},{"comment":"The claim 'increasing F1 scores from 0.72 to 0.81' compares the GPT-4 result on the 80-sample balanced subset with a GloVe F1 of 0.72 reported by Stasaski et al. (2020) on a different CIMA sample (2,296 samples per Section 2.3). This is not an apples-to-apples comparison. Either run the GloVe baseline on the same 80 items or clearly label the comparison as cross-study and not evidence of improvement on this test set.","section":"§5.1"}],"minor_comments":[{"comment":"Typo: 'phrases' should be 'phases' ('the study comprised two main phrases').","section":"§3.3"},{"comment":"The table header is confusing: the column labeled 'avg.' and the per-label F1 columns are not clearly separated. Please re-format so that the reader can identify which number is weighted F1, accuracy, precision, recall, Cohen's kappa, and per-label F1.","section":"Table 5"},{"comment":"The label definition for 'Hint' in Section 3.1 ('scaffolds a student's understanding through providing hints') differs from the definition in Prompt 2/4 ('answering a student's question and/or providing additional information'). If the prompt definitions are the coding scheme, the discrepancy should be reconciled or explicitly acknowledged, since it could affect annotation alignment.","section":"§3.1 vs Table 1"},{"comment":"The phrase 'two preceding tutor-student turns (n=2)' is imprecise: Section 3.2 defines n=2 as the previous tutor-student exchange, i.e., two preceding turns plus the target response. Please use consistent terminology.","section":"§5.1"},{"comment":"No variance or repeat-run information is reported. Given the stochasticity of GPT models, at least a small number of repeated runs per condition would help assess stability; this is especially relevant because the best condition was selected from multiple runs.","section":"Tables 2-5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript header states it is 'Accepted for publication' in a journal. If the paper is simultaneously under journal review or already accepted, the refereeing process at this venue should be clarified; this is an editorial matter. The scientific issues above are independent of that."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a decent, honest case study, not a benchmark. The new bit is showing that prompt-based GPT-3.5/4 can tag tutor dialogue acts in the CIMA corpus without fine-tuning, and they ship the script. The ablations on label definitions and context are sensible, and the null result for chain-of-thought alone is a useful data point. The paper is transparent about its small sample and positions itself as a feasibility study.\n\nThe soft spots are real. The 80% accuracy / 0.81 F1 / 0.74 Kappa is the maximum over five GPT-4 conditions, and those five were selected after running twelve GPT-3.5 conditions on the same 80 balanced examples. So the number is selected on the test set, not a stable estimate for new tutoring dialogues. The balanced design also turns the weighted F1 into a macro average, and the GloVe baseline (F1 0.72) comes from a different sample, so the comparison is suggestive at best. No confidence intervals or error bars.\n\nOne correction to the stress-test note: the Table 5 inconsistency does not hold up. If you read the columns as per-label F1s, average F1, Cohen's k, accuracy, precision, recall, then the row for 4 combined / n=2 gives 0.80 accuracy, 0.81 F1, and 0.74 kappa, exactly as the abstract says. The table formatting is just easy to misread.\n\nWho gets value from this: educational dialogue researchers who want a no-training coding tool and are willing to hand-check outputs, and anyone thinking about prompt selection pitfalls in small-N annotation studies. The paper is a reasonable workshop-grade contribution and deserves referee time if the journal is appropriate. I would ask the authors for a held-out evaluation, error bars, and a report of the unselected prompt performance; without that, the main claim stays conditional.","headline":"Honest small prompt-engineering case study, but the headline 80% is the best condition picked on the same 80 test examples, so treat it as an upper bound; the Table 5 inconsistency in the internal note is a column-order misread.","tokens_in":10308,"tokens_out":5314,"would_cite":false,"duration_ms":54299,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This case study claims that GPT-4, prompted with label definitions, step-by-step instructions, and two preceding turns, classifies tutor dialogue acts at 80% accuracy with Cohen's Kappa 0.74, without fine-tuning.","keywords":["Dialogue act classification","Generative AI","GPT-4","Prompt design","Automatic coding","Educational dialogue analysis","CIMA corpus","Tutoring"],"falsifier":"Run the same combined prompt on the full CIMA corpus (or a fresh random sample that preserves the natural label distribution, including 'Others') and see whether GPT-4's accuracy and Cohen's Kappa stay near 0.80 and 0.74; if agreement drops substantially, the reported result is an artifact of balanced sampling rather than a general capability.","tokens_in":9456,"feed_emoji":"🤖","tokens_out":5215,"duration_ms":50646,"temperature":0.7,"pith_summary":"This paper tries to establish that generative AI, specifically GPT-4, can classify tutors' dialogue acts (Question, Hint, Correction, Confirmation) with substantial agreement to human annotators, without fine-tuning, by using an appropriately designed prompt. On a balanced 80-sample subset of the CIMA tutoring corpus, the best prompt achieved 80% accuracy, weighted F1 0.81, and Cohen's Kappa 0.74. The authors argue this matters because it would let educators and researchers automate dialogue analysis without manual pre-annotation or model training. They also report that clear label definitions and conversational context drive the performance, while chain-of-thought steps help only marginally. The study is a small case study, and the authors explicitly flag that larger data and more categories are needed.","feed_headline":"GPT-4 tags tutor dialogue acts at 80% accuracy","feed_subtitle":"A carefully written prompt with two prior turns lets generative AI match human coding without model training.","key_machinery":"The central object is a four-condition prompt design: the system message (basic, elaborative label definitions, chain-of-thought steps, or combined) and the user message's 'n' preceding turns (0, 1, or 2). The combined condition—step-by-step instructions plus explicit definitions of 'Question,' 'Hint,' 'Correction,' and 'Confirmation'—is what carries the argument. It is the mechanism that elicits the reported 80% accuracy from GPT-4. The other key component is the choice of evaluation metrics: accuracy, weighted F1, and Cohen's Kappa on a balanced 80-sample subset of the CIMA corpus.","core_discovery":"The central claim is that a zero-shot GPT-4 model, given a specific prompt that defines each of the four dialogue acts and guides the model through steps, and given the tutor's response together with the preceding exchange, reproduces human dialogue-act labels on a balanced 80-sample subset of the CIMA corpus at 80% accuracy, weighted F1 of 0.81, and Cohen's Kappa of 0.74. The authors compare this favourably to a GloVe-based model that achieved F1 0.72 on a larger dataset, and emphasize that their result requires no manual pre-annotation and no fine-tuning. They also report that including label definitions and one preceding turn improved performance, while chain-of-thought instructions did n","pith_inferences":["Editorial inference: Because the 80-sample test set is balanced (20 per label), the reported kappa may be optimistic for the natural CIMA label distribution; evaluating on a naturally distributed sample would likely lower the number and would clarify how much the result actually transfers.","Editorial inference: The paper shows that context helps but does not resolve how much context is optimal; a natural next test would vary n from 0 to 3 while measuring per-label errors to see whether 'Correction' needs more context or gets confused by it.","Editorial inference: The prompt template (definitions plus steps plus context) is likely portable to other educational coding schemes, but the paper only tests four categories; extending to a larger scheme would be the direct stress test of the claim.","Editorial inference: Because the models are used through an external API, the privacy concerns the paper raises suggest that real deployments with vulnerable student populations may require local or encrypted inference, not just anonymisation."],"forward_implications":["Dialogue-act coding for tutoring can be performed with natural-language prompts, eliminating the need for manually labelled training data or fine-tuning.","Including explicit definitions of each dialogue-act label in the system message improves accuracy compared to a bare label list.","Providing the previous student utterance (and sometimes an additional prior turn) as context improves performance over classifying the tutor turn alone.","The agreement level (kappa 0.74) is in the range normally described as 'substantial,' suggesting the model could serve as a second/co-coder in the traditional two-coder coding workflow, reducing human coding effort.","The open-source script lets others reproduce the experiments and test the approach on new tutoring corpora."],"fun_headline_variants":["Zero-shot GPT-4 matches human tutor tags at 80%","No training needed: GPT-4 codes tutor dialogue at 80%","GPT-4 hits 80% on tutor dialogue acts via prompt","Prompt-only GPT-4 beats trained model on tutoring tags","80% accuracy: GPT-4 automates tutor dialogue act coding"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire performance claim rests on the assumption that the randomly selected, balanced 80-sample subset fairly represents the CIMA corpus's actual mix of tutor dialogue acts; if that sample is not representative, the 80% accuracy and kappa 0.74 do not generalize to the corpus or to practice.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot GPT-4 matches human tutor tags at 80%","No training needed: GPT-4 codes tutor dialogue at 80%","GPT-4 hits 80% on tutor dialogue acts via prompt","Prompt-only GPT-4 beats trained model on tutoring tags","80% accuracy: GPT-4 automates tutor dialogue act coding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1430,"prompt_tokens":749,"completion_tokens":681,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":591}},"tokens_in":493,"tokens_out":681,"duration_ms":8253,"temperature":1.0,"reasoning_tokens":591,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T19:38:24.145261+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same combined prompt on the full CIMA corpus (or a fresh random sample that preserves the natural label distribution, including 'Others') and see whether GPT-4's accuracy and Cohen's Kappa stay near 0.80 and 0.74; if agreement drops substantially, the reported result is an artifact of balanced sampling rather than a general capability.","supporting_citations":[],"review_version":1}