Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Comparing Few-Shot Prompting of GPT-4 LLMs with BERT Classifiers for Open-Response Assessment in Tutor Equity Training

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuned BERT outperformed few-shot GPT-4o and GPT-4 Turbo on all four open-response grading tasks in a tutor equity-training study.

desk verdict A reproducible small-scale comparison that supports 'fine-tuned BERT beats this few-shot GPT setup' but not the paper's broader claim about GPT models lacking nuance. read the letter →

arxiv 2501.06658 v1 pith:J5DLNCPK submitted 2025-01-11 cs.HC

Sanjit Kakarla , Conrad Borchers , Danielle Thomas , Shambhavi Bhushan , Kenneth R. Koedinger This is my paper · ORCID
classification cs.HC
keywords open-responseassessmentequitytrainingBERTfine-tuningfew-shotpromptingGPT-4tutoreducationautomatedgradingill-defineddomains
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

An open question in automated assessment is whether very large commercial language models can judge open-ended student responses in ill-defined domains, where rubrics require contextual judgment. This paper compares two routes on a tutor-equity training task: fine-tuning a small BERT classifier on about 100 human-annotated replies versus prompting GPT-4o and GPT-4 Turbo with 2–3 examples. Across four binary grading tasks (predicting and explaining appropriate responses in two lessons), the fine-tuned BERT classifier averaged 0.837 accuracy and 0.886 F1, beating GPT-4o (0.726 accuracy) and GPT-4 Turbo (0.549 accuracy) while being cheaper, offline, and open-source. The authors conclude that contemporary GPT models, at least under few-shot prompting, do not adequately capture the nuanced response patterns this grading requires. If replicated, the result favors task-specific fine-tuning over general-purpose prompting for rubric-based open-response assessment.

What carries the argument

The load-bearing machinery is the contrast between two prediction pipelines. The BERT pipeline is a fine-tuned bert-base-uncased classifier with a binary classification head, trained with AdamW at learning rate 2e-5, batch size 16, and five epochs per fold, evaluated by stratified 5-fold cross-validation on tokenized responses (max length 256). The GPT pipeline is a few-shot prompting protocol: 2–3 rubric-aligned example responses are placed in the prompt, the model is instructed to output 0 or 1 indicating adequacy, and the resulting binarized labels are scored with the same fold structure. The paper uses this comparison to attribute the performance gap to the capacity of few-shot prompting to generalize nuanced, context-dependent rubrics, as opposed to supervised fine-tuning on task-specific annotations.

What would settle it

Run the same four prediction tasks with GPT-4o and GPT-4 Turbo under a systematic prompt sweep—varying the number of examples, including the full human rubric, requiring an explanation before the 0/1 label, and averaging multiple temperature-0 runs—and compare average accuracy and F1 to BERT's 0.837 and 0.886; if any configuration matches or exceeds BERT, the paper's conclusion that few-shot GPT cannot match fine-tuned BERT in this domain would need revision.

Watch

Extended reading notes

Core claim

The paper's central claim is that, for this ill-defined educational assessment task, fine-tuning an early-generation BERT classifier on a small set of human annotations yields higher accuracy and F1 than prompting state-of-the-art GPT-4 models with a few examples. The evidence is a stratified 5-fold cross-validation over 243 human-annotated tutor responses across two equity-training lessons, with four prediction tasks formed by crossing lesson with response type (explain vs. predict). BERT averages 0.837 accuracy and 0.886 F1 across tasks, while GPT-4o averages 0.726 accuracy and 0.720 F1 and GPT-4 Turbo averages 0.549 accuracy and 0.550 F1. Because GPT outputs are binarized rather than probabilistic, AUC cannot be computed for them, while BERT's AUC averages 0.870. The authors infer that contemporary GPT models, when used with few-shot prompting, may not adequately capture the nuanced patterns in open responses that require explanation and rubric-based judgment, and that resource-efficient offline fine-tuning is a viable, cost-effective alternative.

Load-bearing premise

The load-bearing premise is that the few-shot prompting setup used for the GPT models—two or three examples, one prompt design, binarized outputs, and no systematic prompt optimization—is a fair and representative deployment of GPT-4's capabilities on this grading task, because if the prompts were suboptimal the observed gap might reflect prompt choices rather than the models' ceiling.

Editorial extensions

If this is right

  • For the four grading tasks studied, a fine-tuned BERT classifier outperforms both GPT-4o and GPT-4 Turbo under few-shot prompting on average accuracy and F1, so educators scoring similar open-ended equity-training responses can expect better results from fine-tuning a small open model than from prompting a commercial API.
  • Because BERT runs offline on roughly 100 labeled examples, the result implies that automated open-response grading with modest annotation budgets is feasible without API costs, version churn, or dependence on external model availability.
  • GPT models' inability to emit calibrated probabilities in this setup rules out ROC-style threshold analysis, so any deployment relying on confidence scores would need a different approach, such as LLM embeddings or fine-tuned GPT.
  • The strongest GPT shortfall appears on explanation-heavy tasks, such as GPT-4 Turbo at 0.293 accuracy for predicting responses in the inequity lesson, suggesting that rubric-based judgments are harder than generating plausible text.
  • The authors' public release of rubrics, data, prompts, and code means the BERT-versus-GPT comparison can be directly replicated and extended as newer GPT models appear.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The comparison bundles together model choice and prompting strategy, so the result is best read as evidence about few-shot prompting as deployed here rather than about GPT-4's ceiling; a fine-tuned GPT or a more systematically optimized prompt could plausibly close or reverse the gap.
  • A natural testable extension is to give GPT the full human rubric text or to use probability/logit outputs and embedding-based classifiers, which would enable AUC comparisons and reveal whether the gap shrinks when the model receives more decision-relevant information.
  • If these findings generalize, other contextual educational domains, such as microaggressions or opportunity-gap training, may also favor local fine-tuned classifiers over general-purpose prompting, especially when grading is binary and rubric-driven.
  • Because GPT model versions change over time, the reported GPT accuracy figures may not hold for later releases; a follow-up that re-runs the same prompts on updated model versions would show how much of the gap is version-dependent.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript compares fine-tuned BERT with few-shot prompted GPT-4o and GPT-4 Turbo on four binary open-response grading tasks in a tutor-equity-training context. Using 243 human-annotated responses from two lessons, the authors report stratified 5-fold cross-validation accuracy, F1, and (for BERT only) AUC. They find that fine-tuned BERT outperforms both GPT variants on average accuracy and F1 across the four tasks, and they argue that offline fine-tuning is more resource-efficient and more stable than commercial LLM prompting. The central claim is that fine-tuning an early-generation model such as BERT yields superior accuracy to few-shot LLM prompting in this ill-defined domain, and the paper concludes that contemporary GPT models may not adequately capture nuanced response patterns. The authors state that no hold-out test set was used and that GPT models did not produce probability outputs, so AUC could not be computed for them.

Significance. If the comparison were fully controlled, the finding would be practically useful: many educational-assessment deployments must choose between fine-tuning an open-source small model and paying per-token for a commercial LLM with a few examples. The paper has real strengths: it uses a domain that is genuinely ill-defined and understudied; the human annotation procedure is documented with high inter-rater reliability (Cohen's kappa 0.86-0.93); and the authors commit to releasing data, rubrics, prompts, and code, which supports reproducibility. The reported BERT numbers (average accuracy 0.837, average F1 0.886) are competitive and the resource-efficiency argument is reasonable. However, the significance of the claim as stated is limited by the lack of uncertainty quantification, the absence of a hold-out test set, and an apparently unvalidated few-shot prompting protocol. The paper is a useful empirical data point for a specific deployment configuration, but it does not yet support the broad statement that contemporary GPT models cannot handle nuanced response patterns.

major comments (4)
  1. [Section 3, GPT prediction paragraph; Table 1] The few-shot protocol appears unvalidated, and one condition is below the majority baseline. For Predict - Helping Students Manage Inequity, GPT-4 Turbo achieves accuracy 0.293 on a task whose positive class is 85%, which is far below always predicting the majority class. That pattern is consistent with output-format confusion, label inversion, or a malformed prompt rather than with the model lacking 'nuanced response patterns.' The manuscript should report the majority-class baseline for every task and should include a sanity check that the prompted model is following the binary-output instruction. Without such a check, the comparison conflates the prompt configuration with the model's capability, and the conclusion in the abstract and Section 5 is not supported.
  2. [Section 4, Table 1] The central superiority claim is reported without confidence intervals, standard deviations across cross-validation folds, or any significance test. With only 79 and 164 labeled responses, differences such as BERT 0.896 versus GPT-4o 0.868 on Explain - Helping Students Manage Inequity may easily be noise. The manuscript should report per-fold results or at least a bootstrap confidence interval for each accuracy and F1 value, and should avoid claiming 'superior performance' on the basis of point estimates alone. This is load-bearing because the abstract and conclusion depend on BERT being reliably better.
  3. [Abstract; Section 5.2; Section 6] The manuscript overgeneralizes from one specific deployment configuration. The abstract states that 'contemporary GPT models may not adequately capture nuanced response patterns' and Section 6 says the results 'underscore the limitations of few-shot prompting for GPT models' in ill-defined domains. But the experiments use exactly 2-3 examples per prompt, a single decoding run, no temperature control, and no probability outputs. The limitations section itself concedes that future work should enhance prompt engineering. The conclusion should be scoped to 'the specific few-shot configuration tested here' or the protocol should be validated with a baseline and multiple prompt variants. As written, the title claim and the discussion go beyond the evidence.
  4. [Section 3; Section 5.2] The evaluation design gives GPT models fewer opportunities to be calibrated than BERT. BERT is fine-tuned and evaluated with stratified cross-validation, while GPT is prompted once per response with binary forced choice and no probability or confidence output. This makes the metric comparison incomplete: accuracy and F1 are compared, but AUC is only reported for BERT, and there is no threshold-free comparison. It would strengthen the paper to also evaluate GPT through token log probabilities (where available), repeated sampling, or a small prompt-validation step on a development subset. As it stands, the reported GPT results may reflect the prompt and decoding settings rather than the underlying model quality.
minor comments (5)
  1. [Abstract and Section 3] The manuscript inconsistently reports the dataset size: the abstract says 243 human-annotated open responses, the introduction says 291 lesson completions from 243 students, and Section 3 reports 79 plus 164 labeled responses. Please clarify whether the 291 figure refers to an unlabeled superset and make the counts consistent.
  2. [Table 1] The table header has typographical issues: 'Model T ask' should be 'Model Task', and the model names contain stray spaces such as 'BER T' and 'T urbo'. These should be corrected.
  3. [Section 3] The F1 metric is not defined precisely. Please state whether it is macro-averaged, weighted, or computed for the positive class, and report the positive-class definition for each task.
  4. [Section 3] Please report the decoding parameters for the GPT models, including temperature, max tokens, and the exact prompt format. The repository may contain the prompts, but the paper should state enough detail for the reader to reproduce the protocol without external resources.
  5. [Introduction] The text mentions 'GPT-4.o' in one place ('GPT-4.o was used as it is substantially more cost-effective'); this should be 'GPT-4o'.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the BERT-vs-GPT comparison is an empirical evaluation on human-annotated labels, with only background self-citations.

full rationale

The central claim—that fine-tuned BERT outperforms few-shot prompted GPT-4o/GPT-4 Turbo on four open-response equity-training classification tasks—rests on an empirical evaluation, not on a definitional equivalence or a fitted-input prediction. BERT is fine-tuned on human-annotated responses and evaluated via stratified 5-fold cross-validation; GPT predictions are generated on the same observations without being trained on them and are compared on accuracy and F1. Human annotations with Cohen's kappa 0.86–0.93 supply ground truth external to both model families. No fitted parameter is renamed as a prediction, and no metric is forced by construction. The cited prior works by the same authors (Thomas et al. 2023/2024, Kakarla et al. 2024, etc.) provide lesson context and motivation, but the comparison's outcome does not reduce to those citations; the human labels and the cross-validation protocol carry the argument. The paper's weakest point—that the few-shot GPT protocol used 2–3 examples, no probability outputs, no temperature control, and no systematic prompt validation—is a concern about external validity and fair deployment of the GPT condition, not about circularity. It could support a correctness or generalization critique, but it does not make the derivation equivalent to its own inputs. Accordingly, no circular step meets the evidentiary bar; the score of 1 reflects only minor, non-load-bearing self-citation in the framing and lesson design.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on annotation quality, the cross-validation setup, the stability of GPT API outputs, and the fairness of the few-shot prompt. No new theoretical entities are introduced; the comparison itself is empirical.

free parameters (2)
  • BERT fine-tuning hyperparameters = lr=0.00002, batch size 16, epochs 5
    Chosen a priori and not tuned; these affect BERT performance and therefore the comparison.
  • Few-shot example count = 2-3 examples
    Prompt design choice; not varied systematically, and it could meaningfully change GPT performance.
assumptions (4)
  • domain assumption Human binary annotations are a valid ground truth for nuanced equity responses.
    Inter-rater reliability is high (kappa 0.86 to 0.93), but binary adequacy labels may compress genuine nuance; if labels are noisy, both models' scores inherit that noise.
  • domain assumption Cross-validation performance estimates generalize to new learners.
    No holdout test set was used; with 243 responses, 5-fold CV can overstate generalizability, especially for the fine-tuned BERT.
  • domain assumption GPT-4 API outputs are stable enough for a single evaluation pass.
    The paper does not report temperature, seeds, or repeated sampling; commercial model versions change over time, so GPT numbers may not reproduce exactly.
  • ad hoc to paper The few-shot prompts represent a fair deployment of GPT for this task.
    The prompt design (2-3 examples, binary output instruction) was chosen by the authors and not systematically optimized; this assumption is load-bearing because the paper generalizes from few-shot performance to model capability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Comparing Few-Shot Prompting of GPT-4 LLMs with BERT Classifiers for Open-Response Assessment in Tutor Equity Training." pith.science (2026). https://pith.science/paper/J5DLNCPK

@misc{pith2026250106658,
  author       = {Pith},
  title        = {Pith review of: Comparing Few-Shot Prompting of GPT-4 LLMs with BERT Classifiers for Open-Response Assessment in Tutor Equity Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J5DLNCPK}},
  note         = {Machine review of arXiv:2501.06658}
}
read the original abstract

Assessing learners in ill-defined domains, such as scenario-based human tutoring training, is an area of limited research. Equity training requires a nuanced understanding of context, but do contemporary large language models (LLMs) have a knowledge base that can navigate these nuances? Legacy transformer models like BERT, in contrast, have less real-world knowledge but can be more easily fine-tuned than commercial LLMs. Here, we study whether fine-tuning BERT on human annotations outperforms state-of-the-art LLMs (GPT-4o and GPT-4-Turbo) with few-shot prompting and instruction. We evaluate performance on four prediction tasks involving generating and explaining open-ended responses in advocacy-focused training lessons in a higher education student population learning to become middle school tutors. Leveraging a dataset of 243 human-annotated open responses from tutor training lessons, we find that BERT demonstrates superior performance using an offline fine-tuning approach, which is more resource-efficient than commercial GPT models. We conclude that contemporary GPT models may not adequately capture nuanced response patterns, especially in complex tasks requiring explanation. This work advances the understanding of AI-driven learner evaluation under the lens of fine-tuning versus few-shot prompting on the nuanced task of equity training, contributing to more effective training solutions and assisting practitioners in choosing adequate assessment methods.

Figures

Figures reproduced from arXiv: 2501.06658 by the authors.

Figure 1
Figure 1. Predict Scenario for the Helping Students Manage Inequity Lesson [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Explain Scenario for the Helping Students Manage Inequity Lesson For evaluation, human-labeled data were split into training and validation subsets ac￾cording to stratified 5-fold cross-validation. Given the small overall sample size, no hold-out test set evaluation was performed. Rather, we report model performance averages across cross-validation folds. Responses were tokenized using the BERT Wordpiece tokenizer w… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Assessment in Team Problem-Solving Exercises in Computing Education

    cs.CY 2026-07 conditional novelty 5.0 of 10

    Clustering teams by their logged actions in cybersecurity tabletop exercises aligns reasonably with instructor scores, while GPT-4o and GPT-5.2 rubric-based assessments of team communication still deviate substantiall...

  2. Exploring LLM-Generated Feedback for Economics Essays: How Teaching Assistants Evaluate and Envision Its Use

    cs.HC 2025-05 conditional novelty 5.0 of 10

    In a think-aloud study, five economics teaching assistants found AI-generated essay feedback useful as suggestions, especially when it included highlighted evidence and intermediate judgments.

Reference graph

Works this paper leans on

17 extracted references · 13 canonical work pages · cited by 2 Pith papers

  1. [1]

    Chatbot interaction with artificial intelligence: human data augmentation with t5 and language transformer ensemble for text classification

    Jordan J Bird, Anik \'o Ek \'a rt, and Diego R Faria. Chatbot interaction with artificial intelligence: human data augmentation with t5 and language transformer ensemble for text classification. Journal of Ambient Intelligence and Humanized Computing, 14 0 (4): 0 3129--3144, 2023

    work page 2023

  2. [2]

    Conrad Borchers, Kexin Yang, Jionghao Lin, Nikol Rummel, Kenneth R Koedinger, and Vincent Aleven. Combining dialog acts and skill modeling: What chat interactions enhance learning rates during ai-supported peer tutoring? In Proceedings of the 17th International Conference on Educational Data Mining, 2024

    work page 2024

  3. [3]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  4. [4]

    Ptt5: Pretraining and validating the t5 model on brazilian portuguese data

    Diedre Carmo, Marcos Piau, Israel Campiotti, Rodrigo Nogueira, and Roberto Lotufo. Ptt5: Pretraining and validating the t5 model on brazilian portuguese data. arXiv preprint arXiv:2008.09144, 2020

    arXiv 2008

  5. [5]

    Bert: Bidirectional encoder representations from transformers

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Bidirectional encoder representations from transformers. arXiv preprint arXiv:1810.04805, page 15, 2018

    arXiv 2018

  6. [6]

    A systematic review on machine learning models for online learning and examination systems

    Sanaa Kaddoura, Daniela Elena Popescu, and Jude D Hemanth. A systematic review on machine learning models for online learning and examination systems. PeerJ Computer Science, 8: 0 e986, 2022

    work page 2022

  7. [7]

    Using Large Language Models to Assess Tutors' Performance in Reacting to Students Making Math Errors

    Sanjit Kakarla, Danielle Thomas, Jionghao Lin, Shivang Gupta, and Kenneth R Koedinger. Using large language models to assess tutors' performance in reacting to students making math errors. arXiv preprint arXiv:2401.03238, 2024

    work page Pith review arXiv 2024

  8. [8]

    A survey of gpt-3 family large language models including chatgpt and gpt-4

    Katikapalli Subramanyam Kalyan. A survey of gpt-3 family large language models including chatgpt and gpt-4. Natural Language Processing Journal, page 100048, 2023

    work page 2023

Show all 17 references
  1. [9]

    An improved aspect-category sentiment analysis model for text sentiment analysis based on roberta

    Wenxiong Liao, Bi Zeng, Xiuwen Yin, and Pengfei Wei. An improved aspect-category sentiment analysis model for text sentiment analysis based on roberta. Applied Intelligence, 51: 0 3522--3533, 2021

    2021

  2. [10]

    How can i get it right? using gpt to rephrase incorrect trainee responses

    Jionghao Lin, Zifei Han, Danielle R Thomas, Ashish Gurung, Shivang Gupta, Vincent Aleven, and Kenneth R Koedinger. How can i get it right? using gpt to rephrase incorrect trainee responses. International Journal of Artificial Intelligence in Education, pages 1--27, 2024

    2024

  3. [11]

    When the tutor becomes the student: Design and evaluation of efficient scenario-based lessons for tutors

    Danielle Thomas, Xinyu Yang, Shivang Gupta, Adetunji Adeniran, Elizabeth Mclaughlin, and Kenneth Koedinger. When the tutor becomes the student: Design and evaluation of efficient scenario-based lessons for tutors. In LAK23: 13th International Learning Analytics and Knowledge C...

    2023

  4. [12]

    Do tutors learn from equity training and can generative ai assess it? arXiv preprint arXiv:2412.11255, 2024

    Danielle R Thomas, Conrad Borchers, Sanjit Kakarla, Jionghao Lin, Shambhavi Bhushan, Boyuan Guo, Erin Gatz, and Kenneth R Koedinger. Do tutors learn from equity training and can generative ai assess it? arXiv preprint arXiv:2412.11255, 2024

    2024 arXiv

  5. [13]

    Beyond the rubric: Classroom assessment tools and assessment practice

    MaryLou Vercellotti. Beyond the rubric: Classroom assessment tools and assessment practice. Tesl-ej, 25 0 (3): 0 n3, 2021

    2021

  6. [14]

    Five ways to look at cohen's kappa

    Matthijs J Warrens. Five ways to look at cohen's kappa. Journal of Psychology & Psychotherapy, 5, 2015

    2015

  7. [15]

    Ai and machine learning for next generation sci-ence assessments

    Xiaoming Zhai. Ai and machine learning for next generation sci-ence assessments. Machine Learning, Natural Language Processing, and Psychometrics, page 201, 2024

    2024

  8. [16]

    Applying machine learning in science assessment: a systematic review

    Xiaoming Zhai, Yue Yin, James W Pellegrino, Kevin C Haudek, and Lehong Shi. Applying machine learning in science assessment: a systematic review. Studies in Science Education, 56 0 (1): 0 111--151, 2020

    2020

  9. [17]

    Using large language models to detect self-regulated learning in think-aloud protocols

    Jiayi Zhang, Conrad Borchers, Vincent Aleven, and Ryan S Baker. Using large language models to detect self-regulated learning in think-aloud protocols. In Proceedings of the 17th International Conference on Educational Data Mining (EDM), 2024

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.