Pith. sign in

REVIEW 3 major objections 5 minor 26 references

Automated Assignment Grading with Large Language Models: Insights From a Bioinformatics Course

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read With structured rubrics and graded examples, LLM graders matched human graders on scores and on feedback quality.

desk verdict A genuinely useful classroom case study whose headline accuracy claim is undermined by an in-sample evaluation; the student-preference results and open-source comparison are the more solid contributions. read the letter →

arxiv 2501.14499 v1 pith:LCXY5JPD submitted 2025-01-24 cs.LG cs.CY

Pavlin G. Poličar , Martin Špendl , Tomaž Curk , Blaž Zupan This is my paper · ORCID
classification cs.LGcs.CY
keywords bioinformaticseducationautomaticevaluationlargelanguagemodelsassignmentgradingfeedbackqualityrubricsfew-shotexamplesopen-sourceLLMs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a semester-long classroom experiment in which written answers from more than 100 students to 36 bioinformatics questions were graded by one of six large language models or by human teaching assistants, without the students knowing which was which. The authors' central claim is that when the prompt contains a structured grading rubric and several TA-graded examples, LLM graders match human graders: on average they matched human judgments on 85–90% of rubric criteria, and students rated machine feedback overall as satisfying as TA feedback. The paper also claims that larger open-source models perform as well as the commercial model tested, that 4-bit quantized versions of 70B models lose little accuracy, and that the only clearly unsuitable model was the smallest 8B one. If these claims hold, large courses could deliver fast, consistent, personalized feedback on written work while freeing teaching assistants for direct interaction with students.

What carries the argument

The carrying mechanism is a two-part prompt. A fixed system prompt supplies grading guidelines and anti-cheating instructions; an exercise-specific user prompt supplies the question, a model answer, the student submission, a structured grading rubric of point-valued criteria with optional explanations, and up to 10 TA-graded example submissions. The model must return a JSON object listing which criteria the submission satisfies, the summed score, and written feedback. The rubric anchors scoring to explicit criteria, the examples calibrate the model's leniency and feedback style, and requiring the model to list satisfied criteria is reported to improve accuracy. Ablations show that rubric-only prompts grade too strictly, example-only prompts grade too leniently, and the combination lands closest to human TA judgments.

What would settle it

Grade a fresh batch of student answers that played no role in rubric refinement or example selection, using the same prompts, and compare against independent TA grades; if accuracy on these held-out answers falls materially below the reported 85–90% range, the claimed parity with human graders is an artifact of prompt tuning rather than a general capability.

Watch

Extended reading notes

Core claim

The paper's claim is that an LLM can take over the grading of written student answers without lowering grading quality, provided the prompt carries a structured rubric and several TA-graded examples. Against 670 manually graded submissions used as the reference standard, the LLM graders correctly matched 85–90% of the rubric criteria on average; the smallest 8B model fell to 75%, and models tended to be more lenient than TAs on hard questions. In the blind feedback survey, students showed no overall preference against LLM feedback, and for correctly answered questions they slightly preferred it, apparently because LLMs wrote longer elaborations while TAs often wrote terse praise. A tone-control group, in which TA feedback was rewritten in a ChatGPT style without changing content, was rated no differently, suggesting that reactions tracked feedback content rather than style. The authors conclude that the prompt design, not the specific model, is the main driver, and that large open-source models with quantization are viable for institutions that want to grade locally.

Load-bearing premise

The load-bearing premise is that the 670 TA-graded submissions used as the gold standard were not also used to improve the grading prompts; the study used the same submissions to refine the rubrics and select examples, so the reported accuracy is an in-sample fit rather than an out-of-sample estimate.

Editorial extensions

If this is right

  • Large courses can shift much of the written-answer grading workload to LLMs without measurably changing the quality of scores or feedback from the student's point of view.
  • Institutions can run grading locally with large open-source models instead of sending student work to commercial APIs, since the largest open-source model matched the commercial model and quantization cost little accuracy.
  • Grading prompts need both an explicit, point-valued rubric and graded examples; supplying only one of the two biases the model toward strictness or leniency.
  • Student acceptance of LLM grading is contingent on keeping a human review option, because models still make occasional errors and students strongly value the ability to request re-evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported accuracy is best read as an in-sample fit: the same 670 TA-graded submissions were used to refine the rubrics, select the examples, and serve as the gold standard, so a held-out test set could give lower numbers. (Editorial inference.)
  • The preference for LLM feedback on correct answers suggests that feedback length, not authorship, drives satisfaction; asking TAs to write longer praise on correct answers would directly test this. (Editorial inference.)
  • The rubric-plus-examples recipe should transfer to other text-heavy courses only where correct answers are enumerable; the paper's own hardest and open-ended questions show the largest accuracy drops. (Editorial inference.)
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports a semester-long deployment of LLM-based grading for written answers in an introductory bioinformatics course, comparing six LLMs against TA-assigned scores on 670 submissions and collecting student satisfaction ratings under a blind design. The authors claim that, with structured rubrics and TA-graded examples, LLMs achieve 85% to 90% classification accuracy on grading criteria and feedback quality comparable to human TAs, with open-source Llama models performing on par with GPT-4o.

Significance. The study addresses a timely and practical problem: automated grading of open-ended student answers. Its strengths include a real classroom setting, a blind student-preference survey, a systematic comparison of six commercial and open-source models, and explicit ethical review. If the accuracy claim were properly validated out-of-sample and against a human-human agreement baseline, the results would be a valuable case study for universities considering LLM grading. However, the current evaluation is in-sample and lacks a human agreement reference, so the central accuracy claim is not yet established.

major comments (3)
  1. [Sec. 3.1 and Sec. 4.1] The evaluation of LLM grading accuracy in Sec. 4.1 is performed on the same 670 TA-graded submissions that were used to refine the grading rubrics (Sec. 3.1) and to select the grading examples (Sec. 3). This makes the reported classification accuracy an in-sample estimate: the rubrics were revised until GPT-4o's errors on these specific submissions were resolved, so the later agreement between LLMs and TA grades reflects the success of that fitting process rather than expected performance on new student answers. The manuscript itself acknowledges the overfitting risk in Sec. 3.1 and Recommendation 3 recommends validation on a new sample, but no such validation is reported. The central claim of "grading accuracy comparable to human graders" therefore needs to be supported by a held-out evaluation (e.g., a train/test split of the 670 submissions) or explicitly re-scoped as an in-sample proof-of-concept.
  2. [Sec. 4.1] The claim that LLM grading accuracy is comparable to human graders is not interpretable without a human-human agreement baseline. The paper reports only LLM-to-TA agreement; if human TAs agree with each other at, say, 95%, then an 85% to 90% LLM-to-TA agreement would indicate substantially worse grading, while if TA-TA agreement is around 80%, the LLM would be at par. The authors should report inter-rater reliability between human TAs on a subsample (e.g., Cohen's kappa or exact score agreement) and compare the LLM-to-TA agreement to that reference. Without this, the comparative statement in the Abstract and Sec. 4.1 is unsupported.
  3. [Sec. 4.1] The accuracy analysis is reported per grading criterion as binary classification, but the grading outcome for students is a numeric score. The paper does not report the distribution of score-level errors, such as exact-match rate, mean absolute score difference, or the proportion of submissions with a grade change large enough to affect a student's overall assignment score. Such metrics would directly support the practical claim that LLM-assigned grades are acceptable for classroom use, and their absence makes it difficult to assess whether the reported 85% to 90% criterion accuracy translates into acceptable scoring behavior.
minor comments (5)
  1. [Sec. 5, point 4] The sentence "open-source LLMs perform as well as their commercial counterpart.s" contains a typographical error; "counterpart.s" should be "counterparts."
  2. [Sec. 4.3] The sentence "We the effects of grader preferences on correctly vs incorrectly graded answers" appears incomplete; it should likely read "We examine the effects...".
  3. [Sec. 2 and Sec. 4.1] The paper uses both "36 text-based questions" and "36 text-based exercises" for the same quantity; please standardize the terminology.
  4. [Sec. 2] The number of human TAs involved and the grading procedure per submission are not described (e.g., whether each answer was graded by a single TA or by multiple TAs). Clarifying this would also help the reader interpret the human-human agreement point raised above.
  5. [Sec. 4.4] The final survey reports responses from 42 students out of 119 in the course; the paper does not discuss potential non-response bias or how representative this subsample may be.

Circularity Check

1 steps flagged · score 6.0 of 10

The 85–90% grading accuracy is measured on the same 670 TA-graded submissions used to refine rubrics and select few-shot examples, making it an in-sample fit rather than an independent prediction.

  1. fitted input called prediction [Section 3.1 (Preparing Grading Rubrics) and Section 4.1 (LLM Grading Accuracy)]
    "To verify that the grading rubric is understandable to LLMs, we evaluate these same submissions using GPT-4o and manually inspect any mismatches between TA-assigned and LLM-assigned scores. In case of systematic differences ... we revise the grading rubric as needed. ... This subset accounts for 25% of total student submissions, resulting in a training set of 670 manually evaluated submissions ... These manually assigned grades were then used as the gold standard for evaluating the performance of different LLMs."

    The reported classification accuracy against TA labels is computed on the same 670 submissions that were used, in Section 3.1, to iteratively revise the grading rubric until GPT-4o mismatches were resolved, and, in Section 3, to sample the up-to-10 TA-graded few-shot examples shown to every LLM. The gold-standard labels therefore shaped the prompt content and example selection for the very submissions on which agreement is measured. The 85–90% accuracy is an in-sample fit, not an out-of-sample estimate of agreement with human graders. The paper itself acknowledges this risk in Recommendation 3 ('Any refinements should further be validated, ideally on a new sample of student submissions, to avoid overfitting'), but the headline accuracy in Section 4.1 does not perform such a validation.

full rationale

The central headline claim—that LLM graders achieve grading accuracy comparable to human teaching assistants—relies on the Section 4.1 classification accuracy measured against 670 TA-graded submissions. Those same submissions were not independent of the prompts: Section 3.1 describes refining the rubrics by evaluating GPT-4o on exactly these submissions and revising until systematic errors were corrected, and Section 3 describes drawing the few-shot grading examples from the same TA-graded pool. Consequently, the reported accuracy is an in-sample evaluation of a prompt that was tuned on the evaluation set. This is a methodological circularity of the 'fitted input called prediction' kind: the quantity presented as evidence of grading accuracy was fitted, at least in part, to the very labels used as ground truth. The paper's own recommendation to validate refinements on a new sample of submissions confirms that the authors recognize the need for out-of-sample evaluation, but the study does not provide it. The student-preference results from the blind study are more independent and provide some support for feedback quality, and the ablation comparisons among prompt variants are informative, but they do not rescue the accuracy claim. The self-citation to the authors' prior course-design paper [17] is not load-bearing for the grading claim. The absence of a human-human agreement baseline is a correctness risk rather than a circularity, since it weakens the interpretation of 'comparable to human graders' without making the derivation circular. Overall, the evaluation is partially circular: the central accuracy result is compromised by in-sample prompt fitting, while other findings retain independent content.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The evaluation depends on hand-tuned grading rubrics and selected examples that are not released; the key assumption that the gold standard is independent of prompt construction is violated by the stated workflow.

free parameters (3)
  • Grading rubric criteria and point allotments = varies per exercise
    Hand-crafted by TAs and refined via GPT-4o dry run on the evaluation set (Sec 3.1); these directly determine scores.
  • Grading examples per exercise = up to 10 per exercise
    Selected from the TA-graded set to maximize diversity; used in the prompt and influence grading (Sec 3).
  • Difficulty categories = trivial/easy/medium/hard/open-ended
    Post hoc bins based on mean scores of the gold standard (Sec 4.1); used in subgroup analysis.
assumptions (3)
  • domain assumption TA grades are the gold standard for correctness
    The paper treats TA-assigned scores as ground truth for LLM evaluation without measuring inter-rater reliability (Sec 4.1).
  • domain assumption The 670-submission set used for evaluation is representative and independent of prompt design
    The paper assumes this independence, but Sec 3.1 shows the set was used to refine rubrics and select examples, violating the assumption.
  • domain assumption Student satisfaction ratings reflect feedback quality
    The paper equates student Likert ratings with feedback quality (Sec 4.3), controlling for scores but not for other confounders.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Assignment Grading with Large Language Models: Insights From a Bioinformatics Course." pith.science (2026). https://pith.science/paper/LCXY5JPD

@misc{pith2026250114499,
  author       = {Pith},
  title        = {Pith review of: Automated Assignment Grading with Large Language Models: Insights From a Bioinformatics Course},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LCXY5JPD}},
  note         = {Machine review of arXiv:2501.14499}
}
read the original abstract

Providing students with individualized feedback through assignments is a cornerstone of education that supports their learning and development. Studies have shown that timely, high-quality feedback plays a critical role in improving learning outcomes. However, providing personalized feedback on a large scale in classes with large numbers of students is often impractical due to the significant time and effort required. Recent advances in natural language processing and large language models (LLMs) offer a promising solution by enabling the efficient delivery of personalized feedback. These technologies can reduce the workload of course staff while improving student satisfaction and learning outcomes. Their successful implementation, however, requires thorough evaluation and validation in real classrooms. We present the results of a practical evaluation of LLM-based graders for written assignments in the 2024/25 iteration of the Introduction to Bioinformatics course at the University of Ljubljana. Over the course of the semester, more than 100 students answered 36 text-based questions, most of which were automatically graded using LLMs. In a blind study, students received feedback from both LLMs and human teaching assistants without knowing the source, and later rated the quality of the feedback. We conducted a systematic evaluation of six commercial and open-source LLMs and compared their grading performance with human teaching assistants. Our results show that with well-designed prompts, LLMs can achieve grading accuracy and feedback quality comparable to human graders. Our results also suggest that open-source LLMs perform as well as commercial LLMs, allowing schools to implement their own grading systems while maintaining privacy.

Figures

Figures reproduced from arXiv: 2501.14499 by the authors.

Figure 2
Figure 2. Prompt structure with a grading rubric and TA-graded examples. The system prompt remains unchanged [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. LLM performance on predicting grading criteria. TA grades represent the gold standard. 95% credible [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Importance of grading rubric and graded-examples on LLM performance. The scale relates to systematic [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Group factor in relation to student preference. Due to the correlation between grading group sample values, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Results of the final survey. Questions are asked on a five-point Likert scale. Red bar colors correspond to [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 17 canonical work pages

  1. [1]

    Challenges and applications of large language models, 2023

    Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models, 2023

    work page 2023

  2. [2]

    Large language models: A survey, 2024

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, 2024

    2024

  3. [3]

    Evaluating the effectiveness of LLMs in introductory computer science education: A semester-long field study

    Wenhan Lyu, Yimeng Wang, Tingting Chung, Yifan Sun, and Yixuan Zhang. Evaluating the effectiveness of LLMs in introductory computer science education: A semester-long field study. In Proceedings of the Eleventh ACM Conference on Learning@ Scale, pages 63–74, 2024

    work page 2024

  4. [4]

    Contextualizing problems to student interests at scale in intelligent tutoring system using large language models, 2023

    Gautam Yadav, Ying-Jui Tseng, and Xiaolin Ni. Contextualizing problems to student interests at scale in intelligent tutoring system using large language models, 2023

    work page 2023

  5. [5]

    Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937, 2023

    Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937, 2023

    arXiv 2023

  6. [6]

    G-EV AL: NLG evaluation using GPT-4 with better human alignment

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-EV AL: NLG evaluation using GPT-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023

    arXiv 2023

  7. [7]

    What are students’ needs and preferences for academic feedback in higher education? a systematic review

    Catherine Paterson, Nathan Paterson, William Jackson, and Fiona Work. What are students’ needs and preferences for academic feedback in higher education? a systematic review. Nurse Education Today, 85:104236, 2020

    work page 2020

  8. [8]

    The failure of a decision support system: inconsistency in test grading by teachers

    Joseph Klein. The failure of a decision support system: inconsistency in test grading by teachers. Teaching and Teacher Education, 18(8):1023–1033, 2002

    work page 2002

Show all 26 references
  1. [9]

    Madigan, Lisa E

    Daniel J. Madigan, Lisa E. Kim, Hanna L. Glandorf, and Owen Kavanagh. Teacher burnout and physical health: A systematic review. International Journal of Educational Research, 119:102173, January 2023

    2023

  2. [10]

    The use of the computer in analyzing student essays

    Ellis B Page. The use of the computer in analyzing student essays. International review of education, pages 210–225, 1968

    1968

  3. [11]

    Learning to grade short answer questions using semantic similarity measures and dependency graph alignments

    Michael Mohler, Razvan Bunescu, and Rada Mihalcea. Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologi...

    2011

  4. [12]

    Investigating neural architectures for short answer scoring

    Brian Riordan, Andrea Horbach, Aoife Cahill, Torsten Zesch, and Chungmin Lee. Investigating neural architectures for short answer scoring. In Proceedings of the 12th workshop on innovative use of NLP for building educational applications, pages 159–168, 2017

    2017

  5. [13]

    LLMs in Automated Essay Evaluation: A Case Study

    Milan Kostic, Hans Friedrich Witschel, Knut Hinkelmann, and Maja Spahic-Bogdanovic. LLMs in Automated Essay Evaluation: A Case Study. Proceedings of the AAAI Symposium Series, 3(1):143–147, May 2024

    2024

  6. [14]

    Large language model as an assignment evaluator: Insights, feedback, and challenges in a 1000+ student course

    Cheng-Han Chiang, Wei-Chih Chen, Chun-Yi Kuan, Chienchou Yang, and Hung-yi Lee. Large language model as an assignment evaluator: Insights, feedback, and challenges in a 1000+ student course. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024...

    2024

  7. [15]

    Using large language models for automated grading of student writing about science

    Chris Impey, Matthew Wenger, Nikhil Garuda, Shahriar Golchin, and Sarah Stamer. Using large language models for automated grading of student writing about science. arXiv preprint arXiv:2412.18719, 2024

    2024 arXiv

  8. [16]

    Can large language models provide feedback to students? a case study on chatgpt

    Wei Dai, Jionghao Lin, Hua Jin, Tongguang Li, Yi-Shan Tsai, Dragan Gaševi´c, and Guanliang Chen. Can large language models provide feedback to students? a case study on chatgpt. In 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), pages 323–325, 2023

    2023

  9. [17]

    Teaching bioinformatics through the analysis of SARS-CoV-2: project-based training for computer science students

    Pavlin G Poli ˇcar, Martin Špendl, Tomaž Curk, and Blaž Zupan. Teaching bioinformatics through the analysis of SARS-CoV-2: project-based training for computer science students. Bioinformatics, 40(Supplement_1):i20–i29, 06 2024

    2024

  10. [18]

    GPT-4o System Card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o System Card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  11. [19]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024

    2024

  12. [20]

    HelpSteer2-Preference: Complementing ratings with preferences, 2024

    Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. HelpSteer2-Preference: Complementing ratings with preferences, 2024

    2024

  13. [21]

    A comprehensive evaluation of quantization strategies for large language models

    Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong. A comprehensive evaluation of quantization strategies for large language models. arXiv preprint arXiv:2402.16775, 2024

    2024 arXiv

  14. [22]

    Kruschke

    John K. Kruschke. Doing Bayesian Data Analysis (Second Edition). Academic Press, 2015. 11 PREPRINT

    2015

  15. [23]

    Stan reference manual, 2.36.0

    Stan Development Team. Stan reference manual, 2.36.0. https://mc-stan.org, 2025

    2025

  16. [24]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  17. [25]

    Investigating automatic scoring and feedback using large language models

    Gloria Ashiya Katuka, Alexander Gain, and Yen-Yun Yu. Investigating automatic scoring and feedback using large language models. arXiv preprint arXiv:2405.00602, 2024

    2024 arXiv

  18. [26]

    Evaluating students’ open-ended written responses with LLMs: Using the RAG framework for GPT-3.5, GPT-4, Claude-3, and Mistral-Large.arXiv preprint arXiv:2405.05444, 2024

    Jussi S Jauhiainen and Agustín Garagorry Guerra. Evaluating students’ open-ended written responses with LLMs: Using the RAG framework for GPT-3.5, GPT-4, Claude-3, and Mistral-Large.arXiv preprint arXiv:2405.05444, 2024. 12

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.