REVIEW 3 major objections 6 minor 37 references
The Impact of Example Selection in Few-Shot Prompting on Automated Essay Scoring Using GPT Models
T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Example selection in few-shot prompts shifts GPT essay scores, with GPT-3.5 most affected.
desk verdict The descriptive QWK results are worth a look, but the regression-based bias claims collapse on perfect multicollinearity. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a controlled family of prompts combined with regression. Each few-shot prompt is named by its sequence of example scores such as H, ML, or LMH, with all one-, two-, and three-shot combinations represented, and QWK is computed after mapping ratings to high, medium, and low. To isolate bias, the paper regresses GPT scores and QWK on the counts of high, medium, and low examples ($H_n, M_n, L_n$) and on indicators for the last example's label ($H_l, M_l, L_l$). The signs and significance of these coefficients are what the majority-label and recency-bias claims rest on.
What would settle it
Because every 3-shot prompt satisfies $H_n+M_n+L_n=3$ and $H_l+M_l+L_l=1$, the reported coefficients are not uniquely identified; a re-estimation with one count and one indicator omitted would settle whether the bias effects are real.
Extended reading notes
Core claim
The central discovery is that the examples in a few-shot prompt are not a neutral scaffold for AES: the score labels included and their order shift the distribution of GPT ratings and the agreement with human raters. The paper names two mechanisms: majority label bias, where more examples of a particular score pull ratings toward that score, and recency bias, where the last example's score has extra influence. These appear in GPT-3.5 snapshots; GPT-4 shows weaker effects, and the June 2023 GPT-4 version is both the most accurate and least sensitive to example variation. The paper argues that this makes example selection a prompt-engineering lever, especially for cost-sensitive deployments of lower-tier models.
Load-bearing premise
The bias findings assume the regression predictors are linearly independent, so each coefficient can be uniquely estimated; in the 3-shot prompts the count variables sum to 3 and the last-label indicators sum to 1, making the coefficients non-identifiable as specified.
Editorial extensions
If this is right
- Prompt design becomes a tunable factor in AES: the set and order of examples can be optimized per model rather than treated as fixed.
- Cost savings are available: with careful example selection, GPT-3.5 can match or beat some GPT-4 configurations.
- Reports of LLM-based AES should state exact minor versions, since behavior shifts between releases such as 0613, 1106, and 0125.
- Examples affect models differently, so conclusions about example selection from one model cannot be assumed to transfer to another.
- Because recency effects are comparable to or larger than majority effects, simply reordering examples may be a low-cost intervention.
Reading between the lines
- One extension the paper does not run is a within-prompt permutation study: holding the multiset of example labels fixed while permuting their order would isolate recency from majority effects, which the current prompt sets vary together.
- The same audit could be applied to other constructed-response scoring tasks and to non-GPT LLM families, since neither the prompt mechanism nor the bias hypotheses are GPT-specific.
- In practice, the findings suggest a cheap calibration step: evaluate a few label and order combinations on a small validation set and freeze the best prompt for each model snapshot before large-scale scoring.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper studies how the choice and order of few-shot examples affect automated essay scoring (AES) with several versions of GPT-3.5 and GPT-4. The authors construct 118 prompts (117 few-shot with one to three examples, plus one zero-shot prompt), score 72 TOEFL11 essays with six GPT models, and quantify agreement with human ratings using quadratic weighted kappa (QWK). They then use linear regression on 3-shot prompts to detect 'majority label bias' and 'recency bias' in both raw GPT scores and QWK. The reported findings are that example selection affects QWK, that the effect is stronger for GPT-3.5 than for GPT-4, that majority and recency biases are present and more pronounced in GPT-3.5, and that GPT-4 (Jun23) is the most stable and highest-performing model in their setup. The descriptive, raw-variation part of the paper is clear and useful; however, the regression-based evidence for the bias claims is compromised by a rank-deficient design, as detailed below.
Significance. The descriptive contribution is practically relevant: documenting that example selection has a much larger effect on GPT-3.5 than on GPT-4, and that a minor-version checkpoint (GPT-4 Jun23) outperforms newer checkpoints, is useful for practitioners and for reporting standards in LLM research. The paper also has positive features: it names exact API model identifiers, fixes temperature to 0, reports the number of prompts and essays, and makes the raw QWK variation visible in figures. If the bias results were statistically valid, they would extend known findings from sentiment analysis (Zhao et al.) to AES. Unfortunately, the regression equations used to support the majority-bias and recency-bias claims are not identifiable as specified, so the quantitative bias conclusions in the abstract, Section 4.2, and Section 5 are not supported by the reported analysis.
major comments (3)
- [3.2, Eqs. (1)-(2)] The regression models are not identifiable because of perfect multicollinearity. For every 3-shot prompt, H_n + M_n + L_n = 3 by construction, and exactly one of H_l, M_l, L_l equals 1, so H_l + M_l + L_l = 1. After the stated standardization, these exact linear dependencies remain (the centered versions sum to zero). Thus the six predictors in Eqs. (1) and (2) are linearly dependent, the design matrix is rank-deficient, and the OLS coefficient vector is non-unique. Any coefficient vector differing by a vector in the null space of X'X produces identical fitted values. The specific coefficients and p-values reported in Tables 1 and 2 therefore depend on arbitrary identifiability conventions rather than on the data. The one-sample t-tests on these coefficients are not valid tests of majority label bias or recency bias. The bias claims in Section 4.2 and Section 5 are load-bearing for the paper's central message and need to be re-derived. A concrete fix is to drop one count predictor and one last-label indicator (e.g., use L_n and L_l as baseline categories) or to use sum-to-zero/orthogonal contrast coding, and then to report which parameterization was used.
- [3.2, Table 2] The QWK regression in Eq. (2) does not specify the observational unit or sample size. If QWK is computed per prompt across the 72 essays, the regression has only 27 distinct 3-shot prompt variants (or 81 if the three random example sets are treated as separate observations), and the t-tests treat these as independent even though the same 72 essays are used in every QWK value. The error terms are therefore correlated across prompts, and the reported p-values in Table 2 are not trustworthy even after resolving the collinearity issue. The paper should state the number of observations used in each regression, and should either use cluster-robust standard errors by essay set or a mixed-effects model with prompt and essay-set as random effects.
- [3.1] The scores substituted for high/medium/low examples were tuned to [4,3,2] using GPT-3.5 (Jun23) on 200 prior essays, and this fixed mapping was then used for all six models. Since the optimal numerical substitution may differ across model versions, this choice can confound cross-model comparisons of both QWK levels and the size of the bias coefficients. The paper should either re-tune the substitution per model (on a separate validation set) or show that QWK and the inferred bias coefficients are robust to the substitution values. At minimum, the sensitivity of the main results to the [4,3,2] choice should be reported.
minor comments (6)
- [Abstract, 3.1] The abstract states '119 prompts' while Section 3.1 says 117 few-shot prompts plus one zero-shot prompt, for a total of 118. Please correct the inconsistency.
- [3.1, GPT Ratings] The sentence 'Two responses within the category LM by GPT-3.5 (Nov23) did not yield ratings' is unclear: were these API calls that returned empty output, or outputs that could not be parsed? This matters for whether the exclusion is a missing-data mechanism or a model failure.
- [3.1, Agreement] The mapping of GPT scores to three levels ('above 3 as high, 3 as medium, below 3 as low') is asymmetric around the example substitution [4,3,2]; please justify this choice and state how ties or non-integer outputs were handled.
- [4.2] The sentence explaining why a meta-analysis was not possible ('essays where all coefficients were zero') is confusing; coefficients being zero for all predictors would be a special case, not a general obstacle to meta-analysis. Please clarify what was meant.
- [Figures 2 and 3] The captions should specify what the error bars represent (standard deviation across the three example sets? across prompts?) and the sample size underlying each average QWK.
- [3.1] The paper does not state whether the 72 evaluation essays and the essays used as examples were drawn from the same TOEFL11 essay prompts. If example essays come from the same prompt topics as evaluated essays, prompt-content leakage could inflate QWK for few-shot conditions; please clarify the sampling procedure.
Circularity Check
No significant circularity: the central claims are empirical comparisons and regressions on independently defined prompt-composition variables, with the single tuning choice disclosed as a pre-experiment hyperparameter selection.
full rationale
The paper is an empirical study rather than a derivation; its central claims are that example selection affects QWK and that majority-label and recency biases appear in GPT scoring. These claims are supported by directly measured QWK values and by regressions whose independent variables are simple counts and last-example indicators, not quantities derived from the outcomes they are used to explain. The only fitted element is the example-score substitution [4, 3, 2], which the paper explicitly states was chosen in a pre-experiment on 200 essays with GPT-3.5 (Jun23) before the main experiment; this is transparent hyperparameter tuning, not a fitted parameter renamed as a prediction, and it does not by construction produce the reported QWK differences or bias coefficients. The paper does not rely on load-bearing self-citations: Yancey et al. is external prior work used as a prompt template, and Zhao et al. is external prior work supplying the bias concepts being tested in a new domain. The rank deficiency in Eqs. (1)-(2) noted by the reviewer is a genuine statistical identifiability concern for the t-tests, but it is not circularity: the regression coefficients are not defined in terms of the quantities they purport to establish, and the descriptive claim that examples affect QWK does not depend on those regressions. No circular step is exhibited.
Assumptions & free parameters
free parameters (2)
- Example score substitutions for high/medium/low =
4/3/2
- Threshold for mapping GPT 1-5 scores to three levels =
>3 high, 3 medium, <3 low
assumptions (3)
- domain assumption TOEFL11 expert ratings on a three-point scale are the ground truth for essay quality.
- ad hoc to paper The linear regression models in Equations (1) and (2) are identifiable, meaning the predictor variables are linearly independent.
- domain assumption GPT API responses at temperature 0 are deterministic and stable across API calls.
Cite this review
Pith. "Pith review of The Impact of Example Selection in Few-Shot Prompting on Automated Essay Scoring Using GPT Models." pith.science (2026). https://pith.science/paper/4AKKGJF4
@misc{pith2026241118924,
author = {Pith},
title = {Pith review of: The Impact of Example Selection in Few-Shot Prompting on Automated Essay Scoring Using GPT Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/4AKKGJF4}},
note = {Machine review of arXiv:2411.18924}
}
read the original abstract
This study investigates the impact of example selection on the performance of au-tomated essay scoring (AES) using few-shot prompting with GPT models. We evaluate the effects of the choice and order of examples in few-shot prompting on several versions of GPT-3.5 and GPT-4 models. Our experiments involve 119 prompts with different examples, and we calculate the quadratic weighted kappa (QWK) to measure the agreement between GPT and human rater scores. Regres-sion analysis is used to quantitatively assess biases introduced by example selec-tion. The results show that the impact of example selection on QWK varies across models, with GPT-3.5 being more influenced by examples than GPT-4. We also find evidence of majority label bias, which is a tendency to favor the majority la-bel among the examples, and recency bias, which is a tendency to favor the label of the most recent example, in GPT-generated essay scores and QWK, with these biases being more pronounced in GPT-3.5. Notably, careful example selection enables GPT-3.5 models to outperform some GPT-4 models. However, among the GPT models, the June 2023 version of GPT-4, which is not the latest model, exhibits the highest stability and performance. Our findings provide insights into the importance of example selection in few-shot prompting for AES, especially in GPT-3.5 models, and highlight the need for individual performance evaluations of each model, even for minor versions.
Reference graph
Works this paper leans on
-
[1]
Min, B., et al.: Recent Advances in Natural Language Processing via Large Pre-trained Lan- guage Models: A Survey. ACM Comput. Surv. 56(2), 30 (2023)
work page 2023
- [2]
-
[3]
Available at SSRN 4337484 (2023)
Baidoo-Anu, D., Ansah, L.O.: Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learn - ing. Available at SSRN 4337484 (2023)
work page 2023
-
[4]
Kasneci, E., et al.: ChatGPT for good? On opportunities and challenges of large lan -guage models for education. Learn. Individ. Differ. 103, 102274 (2023)
work page 2023
-
[5]
Lo, C.K.: What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature. Educ. Sci. 13(4), 410 (2023)
work page 2023
-
[6]
Mizumoto, A., Eguchi, M.: Exploring the Potential of Using an Ai Language Model for Automated Essay Scoring. Res. Methods in Appl. Linguist. 2(2), 100050 (2023)
work page 2023
-
[7]
Yancey, K.P., Laflair, G., Verardi, A., Burstein, J.: Rating Short L2 Essays on the CEFR Scale with GPT -4. In: Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 576-584. Association for Computational Linguistics, Toronto, Canada (2023)
work page 2023
-
[8]
Naismith, B., Mulcaire, P., Burstein, J.: Automated evaluation of written discourse coher- ence using GPT -4. In: Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 394-403. Association for Computational Linguistics, Toronto, Canada (2023)
work page 2023
Show all 37 references
-
[9]
In: Advances in Neural Infor- mation Processing Systems 33 (NeurIPS 2020), pp
Brown, T., et al.: Language Models are Few -Shot Learners. In: Advances in Neural Infor- mation Processing Systems 33 (NeurIPS 2020), pp. 1877 -1901. Curran Associates, Inc., Vancouver, Canada (2020)
2020
-
[10]
In: Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp
Gu, Y., Han, X., Liu, Z., Huang, M.: PPT: Pre -trained Prompt Tuning for Few-shot Learn- ing. In: Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 8410 -8423. Association for Computational Linguis- tics, Dub...
2022
-
[11]
In: Proceedings of the 38th International Con- ference on Machine Learning (ICML 2021), pp
Zhao, T.Z., Wallace, E., Feng, S., Klein, D., Singh, S.: Calibrate Before Use: Improving Few-Shot Performance of Language Models. In: Proceedings of the 38th International Con- ference on Machine Learning (ICML 2021), pp. 12697-12706. PMLR, Online (2021)
2021
-
[12]
In: Advances in Neural Information Processing Systems 35 (NeurIPS 2022) , pp
Wei, J., et al.: Chain -of-Thought Prompting Elicits Reasoning in Large Language Models. In: Advances in Neural Information Processing Systems 35 (NeurIPS 2022) , pp. 24824- 24837. Curran Associates, Inc., New Orleans, USA (2022)
2022
-
[13]
arXiv preprint arXiv:2210.03493 (2022)
Zhang, Z., Zhang, A., Li, M., Smola, A.: Automatic Chain of Thought Prompting in Large Language Models. arXiv preprint arXiv:2210.03493 (2022)
2022 arXiv
-
[14]
In: Advances in Neural Information Processing Systems 35 (NeurIPS 2022), pp
Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large Language Models are Zero- Shot Reasoners. In: Advances in Neural Information Processing Systems 35 (NeurIPS 2022), pp. 22199-22213. Curran Associates, Inc., New Orleans, USA (2023)
2023
-
[15]
arXiv preprint arXiv:2110.08207 (2022)
Sanh, V., et al.: Multitask Prompted Training Enables Zero-Shot Task Generalization. arXiv preprint arXiv:2110.08207 (2022)
2022 arXiv
-
[16]
arXiv preprint arXiv:2312.16549 (2023)
Gupta, K., et al.: How Robust are LLMs to In-Context Majority Label Bias?. arXiv preprint arXiv:2312.16549 (2023)
2023 arXiv
-
[17]
arXiv preprint arXiv:2302.11042 (2023)
Nguyen, T., Wong, E.: In -context Example Selection with Influences. arXiv preprint arXiv:2302.11042 (2023)
2023 arXiv
-
[18]
Grading Essays by Computer
Page, E.B.: The Imminence of... Grading Essays by Computer. The Phi Delta Kappan. 47(5), 238-243 (1966) 12
1966
-
[19]
Assessment in Education: Principles, Policy Pract
Landauer, T.K.: Automatic Essay Assessment. Assessment in Education: Principles, Policy Pract. 10(3), 295-308 (2003)
2003
-
[20]
Attali, Y., Burstein, J.: Automated Essay Scoring With e-rater® V.2. J. Technol. Learn. As- sess. 4(3) (2006)
2006
-
[21]
In: Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
Dong, F., Zhang, Y.: Automatic Features for Essay Scoring - An Empirical Study. In: Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pp. 1072-1077. Association for Computational Linguistics, Austin, Texas (2016)
2016
-
[22]
In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp
Alikaniotis, D., Yannakoudakis, H., Rei, M.: Automatic Text Scoring Using Neural Net- works. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 715-725. Association for Computational Linguis- tics, Berlin, G...
2016
-
[23]
In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp
Taghipour, K., Ng, H.T.: A Neural Approach to Automated Essay Scoring. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1882 -
2016
-
[24]
In: Advances in Neural Information Pro- cessing Systems 30 (NIPS 2017), Curran Associates, Inc., California, USA (2017)
Vaswani, A., et al.: Attention is All you Need. In: Advances in Neural Information Pro- cessing Systems 30 (NIPS 2017), Curran Associates, Inc., California, USA (2017)
2017
-
[25]
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technolog...
2019
-
[26]
Behaviormetrika
Uto, M.: A review of deep-neural automated essay scoring models. Behaviormetrika. 48(2), 459-484 (2021)
2021
-
[27]
In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp
Yang, R., Cao, J., Wen, Z., Wu, Y., He, X.: Enhancing Automated Essay Scoring Perfor- mance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1560-1569. Associa...
2020
-
[28]
In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp
Wang, Y., Wang, C., Li, R., Lin, H.: On the Use of BERT for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno...
2022
-
[29]
Ramesh, D., Sanampudi, S., Kumar: An automated essay scoring systems: a systematic lit- erature review. Artif. Intell. Rev. 55(3), 2495-2527 (2022)
2022
-
[30]
ETS Research Report Series
Blanchard, D., Tetreault, J., Higgins, D., Cahill, A., Chodorow, M.: TOEFL11: A Corpus of Non-Native English. ETS Research Report Series. 2013(2), i-15 (2013)
2013
-
[31]
In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence Survey track, pp
Ke, Z., Ng, V.: Automated Essay Scoring: A Survey of the State of the Art. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence Survey track, pp. 6300 -6308. International Joint Conferences on Artificial Intelligence , Macao, China (2019)
2019
-
[32]
Ramnarain-Seetohul, V., Bassoo, V., Rosunally, Y.: Similarity measures in automated essay scoring systems: A ten-year review. Educ. Inf. Technol. 27(4), 5573-5604 (2022)
2022
-
[33]
-P., Miranskyy, A.: On Sarcasm Detection with OpenAI GPT - based Models
Gole, M., Nwadiugwu, W. -P., Miranskyy, A.: On Sarcasm Detection with OpenAI GPT - based Models. arXiv:2312.04642 (2023)
2023 arXiv
-
[34]
Large Model Systems Organization: LMSYS Chatbot Arena Leaderboard, https://hugging- face.co/spaces/lmsys/chatbot-arena-leaderboard, last accessed 2024/02/05
2024
-
[35]
Shermis, M.D.: State -of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration. Assess. Writ. 20, 53-76 (2014) The Impact of Example Selection in Few-shot Prompting on AES Using GPT Models 13
2014
-
[36]
In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pp
Yannakoudakis, H., Briscoe, T., Medlock, B.: A New Dataset and Method for Automatically Grading ESOL Texts. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pp. 180-189. Association for Computational Lin...
2011
-
[1891]
Association for Computational Linguistics, Austin, Texas (2016)
2016
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.