Pith. sign in

REVIEW 4 major objections 5 minor 21 references

Fine-tuning for Better Few Shot Prompting: An Empirical Comparison for Short Answer Grading

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that fine-tuning GPT-4o-mini on about 150 examples raises short-answer grading F1 from 0.68 to 0.73, and that synthetic examples can make open Llama models competitive.

desk verdict A useful low-resource recipe for ASAG, but the headline numbers rest on a shaky statistical foundation and the central comparison is less clean than it looks. read the letter →

arxiv 2508.04063 v1 pith:ADIQ6AGJ submitted 2025-08-06 cs.LG

Joel Walsh , Siddarth Mamidanna , Benjamin Nye , Mark Core , Daniel Auerbach This is my paper
classification cs.LG
keywords automatedshortanswergradingfew-shotpromptingfine-tuningQLoRAGPT-4o-miniLlama3.18B-InstructsynthetictrainingdatastructuredJSONoutput
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether fine-tuning is worthwhile for automated short-answer grading when only a small labeled set (148 examples) and a single GPU are available. It claims that for the closed model GPT-4o-mini, fine-tuning on that set beats a strong few-shot prompting baseline, lifting F1 from 0.68 to 0.73. It claims the same small dataset is not enough for the open-weight Llama 3.1 8B-Instruct fine-tuned with QLoRA, which reaches only 0.408 at its best epoch; performance becomes competitive (0.653) only after roughly a thousand cheap synthetic examples are added to training. If these results hold, small-scale fine-tuning is a practical route for closed models, and synthetic data can open the same route for open models without large compute clusters. The paper also finds that gains are not uniform across subject domains, with the largest improvements in content-specific lessons.

What carries the argument

Structured JSON grading outputs plus an N-shot evaluation harness. For each concept in a student answer, the model must emit is_known, confidence, and justification as valid JSON. Training mixes labeled responses from 42 lessons, and evaluation draws N=0..40 graded examples from the same withheld lesson as context, reshuffling them across ten trials while never letting test items appear in context. The secondary mechanism is synthetic seeding: randomly selected real graded examples are given to Gemini 1.5 Flash as few-shot prompts to generate new labeled examples, which are then mixed into the fine-tuning set. The work of this machinery is to measure whether fine-tuning improves a model's ab

What would settle it

Re-run the comparison scoring each test response once per N under a disjoint set of few-shot examples, and select Llama epochs on a validation split instead of the test set; if the GPT-4o-mini gain (0.735 vs 0.681) or the synthetic-data gain (0.653 vs 0.408) falls within trial-to-trial variance, the paper's central fine-tuning advantage is not established.

Watch

Extended reading notes

Core claim

The central discovery is that the data appetite of fine-tuning for structured grading output differs sharply between the two model families tested. Fine-tuned GPT-4o-mini reaches F1 0.735 versus 0.681 for the base model over 17,820 prompt evaluations, with the largest gains in specialized domains such as military leadership and suicide prevention rather than the technical diode lesson. QLoRA fine-tuning of Llama 3.1 8B-Instruct on the same 148 real examples is described as unsuccessful: early epochs collapse to predicting mostly false, the 6-epoch model reaches 0.408, and 9 epochs degrade further. Adding 1,000 synthetic examples generated from real seeds by Gemini 1.5 Flash lifts the 6-epoch

Load-bearing premise

The evaluation protocol in Section 2.5 reuses the same test responses under overlapping few-shot contexts across trials; the reported gains treat the resulting 17,820 evaluations as independent samples, which makes the effective sample size much smaller than the headline number.

Editorial extensions

If this is right

  • Fine-tuning a closed model like GPT-4o-mini on roughly 150 examples can improve structured grading F1 by about 0.05 over few-shot prompting alone.
  • Open-weight models fine-tuned with QLoRA on small real data alone are not viable for this task; synthetic data, not more epochs, is what makes them competitive.
  • Because the synthetic expansion costs under one US dollar and runs on one consumer-grade GPU, the recipe makes private, offline grading systems attainable for low-resource organizations.
  • The fine-tuned model's F1 improves more steeply as N-shots increase, indicating that fine-tuning improves how well the model uses in-context rubric examples.
  • The size of the fine-tuning benefit depends on domain: specialized rubric-heavy lessons benefit more, so blanket claims about fine-tuning for ASAG should be domain-aware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment is sweeping the synthetic-data count (e.g., 250, 500, 1,000, 2,000) to map where the F1 gain saturates; the paper fixed 1,000 and notes the choice was arbitrary.
  • The same small-real-seed plus cheap-synthetic-expansion recipe could transfer to other rubric-based structured classifiers where labeled examples are scarce, such as medical or legal answer scoring, since nothing in the mechanism is specific to the three test lessons.
  • The domain dependence hints that fine-tuning is chiefly teaching rubric conventions; a testable prediction is that fine-tuning gains shrink on lessons where general knowledge already aligns with the rubric.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents an empirical comparison of fine-tuning methods for automated short-answer grading (ASAG) with structured JSON outputs. Using 148 human-labeled training examples, the authors fine-tune GPT-4o-mini via OpenAI's API and Llama 3.1 8B-Instruct via QLoRA on a single A40 GPU, and evaluate them on three held-out 'gold' lessons (Diode Breakdown, Reaching Out, Suicide Prevention) across N-shot prompting settings (N=0,5,...,40) and 10 reshuffled trials. They report that fine-tuning GPT-4o-mini improves F1 from 0.681 to 0.735, that QLoRA fine-tuning of Llama alone yields only 0.408 F1 at 6 epochs, and that adding 1,000 Gemini-generated synthetic examples raises Llama to 0.653, nearly matching GPT-4o-mini. The paper concludes that small-scale fine-tuning is useful for closed models and that synthetic data can make open-weight models competitive, with implications for low-resource educational deployments.

Significance. The practical question addressed here is important: can users with modest compute and annotation budgets improve ASAG over few-shot prompting? The study's strengths are its realistic constraints (single GPU, ~150 real examples), systematic variation of N-shot context sizes, inclusion of multiple content domains, and a concrete, low-cost synthetic-data pipeline. If the central F1 differences were statistically reliable, the findings would be useful guidance for practitioners. However, the current evidence is not yet sufficient: the headline numbers are aggregate F1 computed over 17,820 prompt evaluations that are highly correlated within each of only 220 unique gold responses, no confidence intervals or significance tests are provided, and the Llama epoch selection appears to use the test set. These issues are load-bearing because every abstract claim depends on the aggregate numbers being trustworthy.

major comments (4)
  1. [Sec. 2.5, Table 1] The aggregate F1 statistics pool 17,820 prompt-level evaluations, but these are not independent: each gold response (50+50+120, Sec. 2.5) is evaluated 81 times under overlapping N-shot contexts and shuffled trials. The effective sample size for comparing conditions is at most 220 responses, not 17,820. The paper's Introduction asks whether gains are 'statistically and practically meaningful,' but no confidence intervals or significance tests are reported, so the observed gaps (GPT-4o-mini 0.681→0.735; Llama 0.408→0.653) are not established as reliable. Please compute response-level bootstrap CIs or use a mixed-effects model with response as a random effect. Also resolve the inconsistency between the '148 distinct examples' in Sec. 2.1 and the 220 gold responses described in Sec. 2.5.
  2. [Sec. 3, Table 1] The Llama epoch selection appears to be made on the same test set. The text states 'The best model in both instances was 6 epochs' after reporting F1 for 3, 6, and 9 epochs. Choosing the epoch that performs best on the held-out test set and presenting that as the headline result inflates the reported F1 and biases the synthetic-data comparison upward. Use a separate validation split for model selection, or at least report test performance for all epochs and clearly state how the best epoch was chosen.
  3. [Sec. 2.1, Sec. 2.3] Fine-tuning hyperparameters are not reported for either model family. The paper notes it could not ablate LoRA rank/hidden dimensions, but it never states the chosen values for QLoRA (rank, alpha, learning rate, batch size, optimizer) or for the OpenAI fine-tuning job (number of epochs, learning-rate multiplier, etc.). Without these, the core comparison is not reproducible, and the conclusion that QLoRA with small data 'has limited utility' could be an artifact of a poorly chosen configuration rather than a property of the method. Please add a hyperparameter table and any available training details.
  4. [Sec. 2.4, Table 1] The synthetic-data comparison confounds data quantity with data source. The 'with synthetic data' model is trained on 148 real + 1,000 synthetic examples, while the 'without synthetic' model is trained on only 148 real examples. The jump from 0.408 to 0.653 may be due primarily to the larger training set size, not to the synthetic nature of the data. To support the claim that synthetic data 'seeding' is what drives the improvement, a control trained on the same total number of real (or real+real) examples is needed, or the interpretation should be explicitly limited to 'more training data.'
minor comments (5)
  1. [Sec. 2.1 vs Sec. 2.5] The number of evaluation examples is inconsistent: Sec. 2.1 says 148 distinct labeled examples, while Sec. 2.5 lists 50+50+120=220 gold responses. Please correct this discrepancy.
  2. [Figures 2 and 3] All F1/precision/recall plots lack error bars or confidence bands, making it impossible to judge whether apparent differences across N-shot values are meaningful. Adding response-level bootstrap intervals would help considerably.
  3. [Sec. 3] The phrase 'best model' in 'The best model in both instances was 6 epochs' is ambiguous. Specify whether 'best' is determined on a validation split or on the test set; if on the test set, this is a selection bias issue (see Major Comment 2).
  4. [References] Reference [19] appears to be a duplicate of reference [18] (same OpenTutor paper). Please consolidate or replace with the correct citation.
  5. [Throughout] The model name is rendered inconsistently as 'Llama' and 'LLama'; please standardize. Also, the Declared Generative AI statement conflicts with the paper's reported use of GPT-4o and Gemini for data generation; clarify that the statement refers only to the writing of the manuscript.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: held-out empirical fine-tuning comparison; minor self-citation for auxiliary confidence calibration is not load-bearing.

full rationale

The paper is an empirical comparison rather than a derivation. The central claims—fine-tuned GPT-4o-mini F1 increasing from 0.681 to 0.735 and QLoRA-tuned Llama-3-8B with synthetic data increasing from 0.408 to 0.653—are obtained by fine-tuning on training lessons and evaluating on three withheld 'gold' lessons (Section 2.5). Test responses are never used in training, so the improvements are not circular by construction. The only self-referential element is the use of GPT-4o-generated justifications and confidence scores, averaged with a logistic regression classifier 'previously trained on the same dataset' (refs. [18]/[19], overlapping with the present authors), to enrich training examples (Section 2.3). This is a minor self-citation but is not load-bearing for the F1 comparisons: the concept labels used for scoring are expert human labels, and the confidence-calibration step is an auxiliary training-signal detail rather than the source of the reported outcome. The paper's statistical concerns—treating 17,820 correlated prompt evaluations as independent, and selecting the reported Llama epoch based on the test set—are validity and bias issues, not circularity: they do not reduce any reported quantity to an input by definition. No equations are derived from assumptions, and no uniqueness theorem or ansatz is imported via self-citation. The findings are self-contained against held-out labels, so the circularity score is 1 for the minor self-citation only.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on a private labeled dataset, assumed label quality, and a repeated-sampling evaluation protocol. Several design choices, including the number of epochs and the amount of synthetic data, were selected without validation-based justification, and no code or data was released.

free parameters (5)
  • Number of fine-tuning epochs for Llama = 6 epochs, selected from 3, 6, and 9
    The best epoch appears to be selected using test-set performance, with no validation-based early stopping; this makes the reported F1 optimistic.
  • Amount of synthetic training data = 1,000 examples
    The authors explicitly state the amount was 'chosen somewhat arbitrarily' in Sec. 2.4.
  • N-shot context sizes and trial count = N = 0, 5, 10, ..., 40 and 10 trials
    These evaluation design choices determine the variance and effective independence of the reported F1 scores.
  • Confidence calibration blend weight = 0.5 equal average
    GPT-4o confidence scores are averaged with logistic-regression scores without justification for the 50/50 weighting.
  • QLoRA rank/alpha and OpenAI fine-tuning hyperparameters = Not disclosed
    LoRA configuration is not reported, and OpenAI hides fine-tuning details, so the comparison is not fully reproducible.
assumptions (5)
  • domain assumption Expert labels in the OpenTutor data are correct and consistent
    All training and evaluation labels come from subject matter experts, but no inter-annotator agreement or label reliability statistics are reported.
  • domain assumption GPT-4o-generated justifications and confidence scores are accurate enough to train on
    Sec. 2.3 uses GPT-4o to generate justifications and confidence scores, averaged with a logistic regression, without measuring the quality of these synthetic training signals.
  • domain assumption Repeated evaluations of the same test responses are independent
    Sec. 2.5 expands 148 gold responses into 17,820 evaluations by chunking, overlapping N-shot contexts, and multiple trials; the implied independence is not tested.
  • domain assumption Gemini 1.5 Flash synthetic examples are representative of the target ASAG domains
    Sec. 2.4 generates examples from random training examples with a broad domain prompt; there is no check that the generated distribution matches the three gold lessons.
  • domain assumption Three gold lessons are representative of ASAG
    The authors acknowledge the evaluation set is limited; generalizing to other subject areas is an unverified assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fine-tuning for Better Few Shot Prompting: An Empirical Comparison for Short Answer Grading." pith.science (2026). https://pith.science/paper/ADIQ6AGJ

@misc{pith2026250804063,
  author       = {Pith},
  title        = {Pith review of: Fine-tuning for Better Few Shot Prompting: An Empirical Comparison for Short Answer Grading},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ADIQ6AGJ}},
  note         = {Machine review of arXiv:2508.04063}
}
read the original abstract

Research to improve Automated Short Answer Grading has recently focused on Large Language Models (LLMs) with prompt engineering and no- or few-shot prompting to achieve best results. This is in contrast to the fine-tuning approach, which has historically required large-scale compute clusters inaccessible to most users. New closed-model approaches such as OpenAI's fine-tuning service promise results with as few as 100 examples, while methods using open weights such as quantized low-rank adaptive (QLORA) can be used to fine-tune models on consumer GPUs. We evaluate both of these fine-tuning methods, measuring their interaction with few-shot prompting for automated short answer grading (ASAG) with structured (JSON) outputs. Our results show that finetuning with small amounts of data has limited utility for Llama open-weight models, but that fine-tuning methods can outperform few-shot baseline instruction-tuned LLMs for OpenAI's closed models. While our evaluation set is limited, we find some evidence that the observed benefits of finetuning may be impacted by the domain subject matter. Lastly, we observed dramatic improvement with the LLama 3.1 8B-Instruct open-weight model by seeding the initial training examples with a significant amount of cheaply generated synthetic training data.

Figures

Figures reproduced from arXiv: 2508.04063 by the authors.

Figure 1
Figure 1. Visualization of evaluation structure for N=5 and 50 examples. All examples in the active test set consist of only the answer, while the n-shot examples contain the graded answers. occurs in the Reaching Out and Suicide Prevention domains, which are based on highly specific content domains and whose evaluations require using labeled examples rather than prior knowledge. The objective of fine-tuning the model on a se… view at source ↗
Figure 2
Figure 2. F1 scores for baseline and fine-tuned models (Using best Llama model, 6 epoch) on the baseline model (see [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Performance metrics across different text domains within the training data This process is thought to be much more sample efficient than supervised finetuning. In terms of training with synthetic data, this process can optimized substantially. The prompt could be optimized, the amount of data could be increased, and much more advanced models than Gemini 1.5 Flash now exist. However, the iterative nature of data anno… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 13 canonical work pages

  1. [1]

    Alikaniotis, H

    D. Alikaniotis, H. Yannakoudakis, M. Rei, Automatic text scoring using neural networks, in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, 2016, p. 715–725. URL: https://www. aclweb.org/anthology/P16-1068

    work page 2016

  2. [2]

    Kumar, S

    S. Kumar, S. Chakrabarti, S. Roy, Earth mover’s distance pooling over siamese LSTMs for automatic short answer grading, in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2017, p. 2046–2052. URL: https: //aclanthology.org/D17-1217

    work page 2017

  3. [3]

    S. Saha, T. I. Dhamecha, S. Marvaniya, R. Sindhgatta, B. Sengupta, Sentence level or token level features for automatic short answer grading?: Use both, in: Natural Language Processing and Information Systems, volume 10859 of Lecture Notes in Computer Science , Springer International Publishing, Cham, 2018, pp. 35–47. doi:10.1007/978-3-319-91947-8_3

    work page doi:10.1007/978-3-319-91947-8_3 2018

  4. [4]

    T. Liu, W. Ding, Z. Wang, J. Tang, G. Y. Huang, Z. Liu, Automatic short answer grading via multiway attention networks, arXiv preprint arXiv:1909.10166 (2019). URL: http://arxiv.org/abs/1909.10166

    work page Pith review arXiv 1909

  5. [5]

    Z. Wang, A. S. Lan, A. E. Waters, P. J. Grimaldi, R. G. Baraniuk, A meta-learning augmented bidirectional transformer model for automatic short answer grading, in: Proceedings of the 12th International Conference on Educational Data Mining (EDM 2019), International Educational Data Mining Society (IEDMS), 2019, pp. 156–163. URL: https://educationaldatamin...

    work page 2019

  6. [6]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, ...

    arXiv 2005

  7. [7]

    Short Answer Grading Using One-shot Prompting and Text Similarity Scoring Model

    S.-Y. Yoon, Short answer grading using one-shot prompting and text similarity scoring model, arXiv preprint arXiv:2305.18638 (2023). URL: http://arxiv.org/abs/2305.18638

    work page Pith review arXiv 2023

  8. [8]

    Schneider, B

    J. Schneider, B. Schenk, C. Niklaus, Towards LLM-based autograding for short textual answers, arXiv preprint arXiv:2309.11508 (2024). URL: http://arxiv.org/abs/2309.11508

    arXiv 2024

Show all 21 references
  1. [9]

    Ivanova, S

    R. Ivanova, S. Handschuh, Evaluating LLMs’ performance at automatic short-answer grading, in: Proceedings of the Workshop on Automatic Evaluation of Learning and Assessment Content (EvalLAC 2024), volume 3772 of CEUR Workshop Proceedings, CEUR-WS.org, Recife, Brazil, 2024, pp....

    2024

  2. [10]

    Chamieh, T

    I. Chamieh, T. Zesch, K. Giebermann, LLMs in short answer scoring: Limitations and promise of zero-shot and few-shot approaches, in: Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Association for Computational Lingui...

    2024

  3. [11]

    J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, Q. V. Le, Finetuned language models are zero-shot learners, arXiv preprint arXiv:2109.01652 (2022). URL: http://arxiv. org/abs/2109.01652

    2022 arXiv

  4. [12]

    Z. Chen, Y. Deng, H. Yuan, K. Ji, Q. Gu, Self-play fine-tuning converts weak language models to strong language models, arXiv preprint arXiv:2401.01335 (2024). URL: http://arxiv.org/abs/2401. 01335

    2024 arXiv

  5. [13]

    A. Zhu, P. Asawa, J. Q. Davis, L. Chen, L. Hanin, I. Stoica, J. E. Gonzalez, M. Zaharia, BARE: Leverag- ing base language models for few-shot synthetic data generation, arXiv preprint arXiv:2502.01697 (2025). URL: http://arxiv.org/abs/2502.01697

    2025 arXiv

  6. [14]

    OpenAI, Introducing improvements to the fine-tuning api and ex- panding our custom models program, https://openai.com/index/ introducing-improvements-to-the-fine-tuning-api-and-expanding-our-custom-models-program/,

  7. [15]

    OpenAI platform, https://platform.openai.com, ???? Accessed on 2025-07-08

    2025

  8. [16]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, ..., Z. Ma, The llama 3 herd of models, arXiv preprint arXiv:2407.21783 (2024). URL: http://arxiv.org/abs/2407.21783

    2024 arXiv

  9. [17]

    Dettmers, A

    T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer, QLoRA: Efficient finetuning of quantized LLMs, arXiv preprint arXiv:2305.14314 (2023). URL: http://arxiv.org/abs/2305.14314

    2023 arXiv

  10. [19]

    B. D. Nye, R. Sanghrajka, V. Bodhwani, M. Acob, D. Budziwojski, K. Carr, W. R. Swartout, Opentutor: Designing a rapid-authored tutor that learns as you grade, in: The International FLAIRS Conference Proceedings, volume 34, 2021. URL: https://doi.org/10.32473/flairs.v34i1.12857...

    2021 doi

  11. [20]

    Z. R. Tam, C.-K. Wu, Y.-L. Tsai, C.-Y. Lin, H. yi Lee, Y.-N. Chen, Let me speak freely? a study on the impact of format restrictions on performance of large language models, arXiv preprint arXiv:2408.02442 (2024). URL: http://arxiv.org/abs/2408.02442

    2024 arXiv

  12. [21]

    DeepSeek-AI, D. Guo, D. Yang, H. Zhang, et al., Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501.12948 (2025). URL: http://arxiv.org/abs/ 2501.12948

    2025 arXiv

  13. [2023]

    Accessed on 2025-07-08

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.