Enhancing Security and Strengthening Defenses in Automated Short-Answer Grading Systems

Brian Clauser; Peter Baldwin; Polina Harikeo; Saed Rezayi; Sahar Yarmohammadtoosky; Victoria Yaneva; Yiyun Zhou

REVIEW 3 major objections 5 minor 17 references

Enhancing Security and Strengthening Defenses in Automated Short-Answer Grading Systems

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mixed-answer gaming false positives cut from 43.5% to 4.1%

desk verdict Useful systematic comparison of countermeasures against gaming in medical short-answer grading, but the headline FPR drops are not all like-for-like because before/after numbers come from different test sets. read the letter →

arxiv 2505.00061 v1 pith:AEPI2C3U submitted 2025-04-30 cs.CL cs.CR

Sahar Yarmohammadtoosky , Yiyun Zhou , Victoria Yaneva , Peter Baldwin , Saed Rezayi , Brian Clauser , Polina Harikeo This is my paper · ORCID

classification cs.CLcs.CR

keywords automatedshort-answergradingadversarialtraininggamingstrategiesfalsepositiveratetransformer-basedscoringmedicaleducationassessmentpromptengineeringensemblemethods

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether automated short-answer grading systems for medical exams can be tricked into giving credit for answers that deserve none, and whether the trick can be taught away. Working with ACTA, a transformer-based grader that matches a student response to the closest human-scored answer, the authors simulate three gaming strategies an unprepared examinee might try: sampling words from the question stem, submitting a ChatGPT summary of the clinical vignette, and listing multiple plausible answers at once. All three fool the untrained system, with the multiple-answer tactic producing a false positive rate of 43.5%. Adding simulated gaming responses to the training data cuts that rate to 4.1% when the same strategy appears in training and to 6.7% when training covers only the other strategies, while accuracy on real responses stays at F1 = 0.98. The authors conclude that adversarial training, ensemble scoring, and LLM-based prompt engineering can each meaningfully harden such systems, supporting the reliability and fairness of AI-based grading in high-stakes settings.

What carries the argument

The central object is ACTA, a transformer-based short-answer grader that embeds each response with Sentence-BERT and assigns the label of the most similar human-scored response when similarity exceeds a threshold. The paper's core mechanism is adversarial training by data augmentation: simulated gaming responses -- random stem words, ChatGPT vignette summaries, and mixed correct-plus-incorrect answers -- are added to the training set so the similarity-matching model learns to label them incorrect. The defense is reinforced by ensemble aggregation, majority vote and ridge regression over five medical BERT embedding models, and by a separate prompt-engineering route in which GPT-4 is asked to score responses given the question context.

What would settle it

Collect authentic gaming attempts from real test-takers, for example from low-stakes practice administrations or from responses flagged during operational grading, and run the adversarially trained ACTA system on them. If the false positive rate on those authentic gaming responses is close to the untrained baseline (for example, above 0.20 for mixed-answer responses), the paper's defense claim would not transfer to practice.

Watch

Extended reading notes

Core claim

The central claim is that transformer-based short-answer graders are substantially gameable but can be hardened. ACTA, a similarity scorer built on Sentence-BERT embeddings, awards credit to 43.5% of "mixed" responses that list multiple plausible diagnoses, 18.9% of ChatGPT-generated vignette summaries, and 6.1% of random words sampled from the question stem. Adversarial training on simulated gaming responses reduces the mixed-response false positive rate to 4.1% when the same strategy is seen in training and to 6.7% when the model is trained only on the other two strategies, while keeping F1 on authentic responses at 0.98. Combining five medical-domain BERT embeddings through majority vote or ridge regression lowers false positives further than any single model, and prompting GPT-4 to score with the question supplied recognises mixed-answer gaming with a 1% false positive rate. The authors take these results as evidence that both data augmentation and LLM-based scoring can materially reduce the damage from known gaming tactics.

Load-bearing premise

The simulated gaming responses are representative enough of how real examinees would actually game the system that the measured false-positive reductions would hold during operational use.

Editorial extensions

If this is right

Adversarial training on simulated gaming responses cuts the false positive rate for the most damaging strategy (mixed responses) from 0.435 to 0.041 when the strategy is included in training, with no drop in real-response accuracy (F1 = 0.98).
Cross-strategy adversarial training transfers partial protection: training on two gaming strategies lowers the mixed-response FPR to 0.067, suggesting that defenses built on known tactics generalise to some degree to unseen tactics.
Ensemble scoring with majority vote, and especially ridge regression across five medical BERT embeddings, yields lower false positive rates than any individual embedding model, making ensembles a useful second line of defense.
Prompt-engineering GPT-4 to score responses with the question supplied recognises mixed-answer gaming with FPR 0.01 and vignette summaries with FPR 0.03, outperforming adversarial training on those strategies, while remaining weaker on random stem-word sampling (FPR 0.11).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If real test-takers game the system in ways resembling the simulated strategies, the measured FPR reductions suggest operational ASAG systems could avoid awarding undeserved credit for the most common gaming attempts, but the transfer depends on the authenticity of the simulations — a limitation the paper itself flags.
The finding that training on "strong" gaming examples transfers better to unseen strategies than training on "weak" ones suggests a testable design principle: when building adversarial training sets, prioritise collecting or synthesising the most effective known attacks, not the easiest ones.
The GPT-4 result on mixed responses hints that an LLM scoring layer could act as a complement to similarity-based graders where those are most gameable, though the LLM's weaker performance on random word sampling shows no single model should be the only defense.
A direct next step would be to collect authentic gaming attempts from low-stakes or practice exams and measure how well adversarially trained models score those naturally occurring responses; the paper's Section 7 limitation makes that the natural confirmation test.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: this is a useful empirical study of countermeasures against gaming in a medical short-answer grading system, with a real dataset and a sensible experiment suite, but the headline FPR reductions are not all computed on the same test sets, so treat the specific numbers with care.

What is new: the paper takes gaming strategies identified in prior work by the same group (Baldwin et al. 2025) and systematically evaluates adversarial training, five embedding models, majority-vote and ridge ensembles, and GPT-4 prompting. That systematic comparison is the contribution. The main positive result is credible in direction: after adversarial training, the model keeps F1 around 0.98 on real responses while lowering false positives on simulated gaming responses. The cross-strategy experiment (AdvT2) is a good design and suggests partial transfer. The GPT-4 prompt engineering results, especially for mixed responses, are also worth attention.

Soft spots, in proportion. The most important is an internal-validity mismatch in the central before/after comparison. The 'before' FPRs are computed on all simulated responses (e.g., all 584 Mixed Responses), while the 'after within-strategy' FPRs (AdvT1) are computed on a 30% held-out subset (about 175). So the drop from 0.435 to 0.041 for Mixed Responses is not a same-test-set comparison; sampling variability alone could move those numbers. The AdvT2 cross-validation numbers are on the full held-out strategy and are comparable to 'before', but the paper does not clearly separate that. No confidence intervals or significance tests are reported anywhere, which matters when test sets are this small. The acknowledged limitation in Section 7—simulated responses are not authentic examinee behavior—is real and honestly stated, but it means operational transfer is unknown. Minor issues: the prompt-engineering section reports only the best of three strategies and uses only 100 samples per condition.

Who this is for: people building or defending automated short-answer grading systems, particularly in medical education. It deserves a serious referee; the flaws are addressable with same-test-set comparisons, confidence intervals, and ideally a code/data release. I would accept it for review but not without revision.

Referee Report

3 major / 5 minor

Summary. The paper studies vulnerability of the ACTA transformer-based automated short-answer grading system to three simulated gaming strategies (random words from the stem, ChatGPT summaries, and mixed correct/incorrect responses) using real medical exam data. It reports that adversarial training, both within-strategy (AdvT1) and cross-strategy (AdvT2), substantially reduces false positive rates on simulated gaming responses while maintaining an F1 of 0.98 on authentic responses, and that ensemble methods (majority vote, ridge regression) and GPT-4 prompting can further improve detection. The central quantitative claim is that the Mixed Responses FPR drops from 0.435 before training to 0.041 after within-strategy training and 0.067 after cross-strategy training.

Significance. If the reported effects are robust, the paper addresses a real and important problem: maintaining integrity of automated scoring in high-stakes medical education. The study uses a large authentic dataset (36,735 responses), describes the simulation pipeline in detail, and compares several defense strategies across five embedding models and two ensemble methods, which is a useful contribution to the ASAG security literature. The authors are appropriately transparent about the simulated nature of the adversarial examples and the single-domain scope. However, the headline FPR reductions are weakened by an internal-validity issue in the before/after comparison and by the absence of uncertainty quantification, so the quantitative conclusions are not yet firmly established.

major comments (3)

[§4.1 vs. §4.2] The 'before' and 'after' FPRs for the within-strategy experiment (AdvT1) are computed on different test sets, so the reported reductions are not like-for-like. Section 4.1 states that pre-training evaluation was done on 'the remaining 30% (10,890) combined with all artificial responses,' meaning the 'before' FPR for each strategy is based on the full set of simulated responses (e.g., all 584 Mixed Responses). Section 4.2 states that Experiment 1 uses 'the remaining 30% of both artificial and authentic responses' for testing, so the 'after within-strategy' FPR is based on only about 175 Mixed Responses. Thus Table 2's headline reduction from 0.435 to 0.041 could be inflated or deflated by sampling variability, and the same issue affects Tables 3–4 and Figures 4–6 for AdvT1. The authors should recompute the 'before' FPR on the same 30% held-out subset, or report both evaluations on the full set and the subset, with confidence intervals.
[§4.2 and Tables 2–4] No confidence intervals or significance tests accompany any FPR estimate. With roughly 175 graded Mixed Responses in the AdvT1 test set, a difference between 0.041 and 0.067 corresponds to about 7 versus 12 false positives, which is not statistically distinguishable. The claim that ridge regression 'outperformed' majority vote (e.g., 0.035 vs. 0.035 for Mixed Responses in Table 4) is similarly unsupported. The authors should report exact counts, binomial or bootstrap confidence intervals, and pairwise tests where comparisons are made.
[§7] The paper explicitly concedes that the adversarial examples are 'simulated approximations of gaming strategies, rather than authentic, organically derived examples from real-world test-takers.' This concession is appropriate, but it directly limits the operational conclusion that adversarial training 'significantly reduces' the system's susceptibility. The reported gains are measured on the same simulation distribution used for training, so they may not transfer to real examinee gaming behavior. The authors should either temper the general claim (e.g., restrict it to simulated gaming responses) or provide a concrete out-of-distribution test, such as evaluating on a separate set of human-generated gaming attempts or on responses from a different exam cohort.

minor comments (5)

[§4.3 and Tables 3–4] There are typographical inconsistencies, e.g., 'Majority V ote' appears in Table 4 and Section 6, and the column header 'Acta Model' in Table 3 should be 'ACTA Model.'
[§4.3] The text says 'Figures 5 and 6 and Tables 3 and 4,' but Figures 4–6 are all referenced; please correct the cross-references.
[§4.4] The prompt engineering experiment is based on only 100 samples of each data type, and only the results of the first prompting strategy are reported. Please state the sample size limitation explicitly in the table or text, and explain why the other two strategies are omitted.
[§2] When describing Baldwin et al. (2025), the text says 'the first five strategies lead to a success rate between 6% to 16%,' but the reporting of strategy 6 (multiple responses) is the most effective; it would be clearer to present these as a list or table.
[Appendix A] Appendix A contains only the sentence 'This is an appendix.' It should be removed or filled with actual content.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FPR reductions are measured on held-out simulated responses; self-citations supply data-generation and system descriptions rather than the conclusion.

full rationale

The paper's central claim is empirical rather than derivational: adversarial training on simulated gaming responses reduces false positive rates while preserving accuracy on real responses. The within-strategy condition trains on 70% of simulated responses and evaluates on the remaining 30%, and the cross-strategy condition uses 3-fold validation testing on a strategy not seen in training, so the reported FPR reductions are measurements on held-out data rather than quantities forced by construction. The Section 4.1 baseline is computed on all artificial responses, whereas the AdvT1 condition in Section 4.2 uses a 30% subset of artificial responses; this is a like-for-like comparability concern that may affect the magnitude of the reported drop, but it is not circularity because the 'after' numbers are still evaluated on data not used in that condition's training, and the AdvT2 numbers are computed on fully held-out strategies. The gaming strategies are adopted from Baldwin et al. (2025), which shares authors with this paper, but that citation provides the data-generation recipe and prior evidence of vulnerability, not the conclusion about adversarial training efficacy; the FPR reductions are measured rather than derived from the citation. Section 7 explicitly concedes that the simulated gaming responses are approximations rather than authentic real-world gaming, which limits external validity but does not make the within-paper measurements self-justifying. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported, and no known result is relabeled. The paper is therefore self-contained with respect to its own evaluation protocol, and no circular step is identified.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities or fitted constants. The central claims rest on domain assumptions about the realism of the simulated gaming strategies, the representativeness of the ACTA system, and the generalizability of the single medical dataset; all three are acknowledged in Section 7. These are honest limitations rather than hidden free parameters.

assumptions (3)

domain assumption Simulated gaming responses are representative of real examinee gaming behavior.
Stated in Section 7 (Limitations): 'the adversarial examples used in our experiments are simulated approximations of gaming strategies, rather than authentic... may not fully reflect the diversity and nuance of actual test-taker behaviors.'
domain assumption The ACTA system is representative of transformer-based ASAG systems.
The study evaluates only ACTA; Section 7 notes 'The extent to which these results generalize to other transformer-based or few-shot scoring systems is an open question.'
domain assumption The 71 SAQ dataset is representative of high-stakes medical exam responses.
Single domain and dataset; Section 7 notes generalizability to other domains is uncertain.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enhancing Security and Strengthening Defenses in Automated Short-Answer Grading Systems." pith.science (2026). https://pith.science/paper/AEPI2C3U

@misc{pith2026250500061,
  author       = {Pith},
  title        = {Pith review of: Enhancing Security and Strengthening Defenses in Automated Short-Answer Grading Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AEPI2C3U}},
  note         = {Machine review of arXiv:2505.00061}
}

read the original abstract

This study examines vulnerabilities in transformer-based automated short-answer grading systems used in medical education, with a focus on how these systems can be manipulated through adversarial gaming strategies. Our research identifies three main types of gaming strategies that exploit the system's weaknesses, potentially leading to false positives. To counteract these vulnerabilities, we implement several adversarial training methods designed to enhance the systems' robustness. Our results indicate that these methods significantly reduce the susceptibility of grading systems to such manipulations, especially when combined with ensemble techniques like majority voting and ridge regression, which further improve the system's defense against sophisticated adversarial inputs. Additionally, employing large language models such as GPT-4 with varied prompting techniques has shown promise in recognizing and scoring gaming strategies effectively. The findings underscore the importance of continuous improvements in AI-driven educational tools to ensure their reliability and fairness in high-stakes settings.

Figures

Figures reproduced from arXiv: 2505.00061 by the authors.

**Figure 1.** PCA of Response Embedding for Item 1 2 1 0 1 2 3 4 Principal Component 1 2 1 0 1 2 3 4 Principal Component 2 PCA of Responses Embeddings MCN0955 wrong_response right_response adversarial [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

**Figure 3.** Adversarial Defense Workflow. Gaming data is combined with real data for training and testing purposes. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 5.** FPR Across Different Models - Part 2 Gaming Strategy FPR Before Adv Training FPR Adv Training #1 FPR Adv Training #2 Information from the Stem .061 .017 .067 Clinical Case Summary .189 .036 .04 Mixed Responses .435 .041 .067 [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗

Figures from the paper (1 more)

**Figure 6.** Figure 6: False Positive Rates for Majority Vote and [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 10 canonical work pages

[1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[2]

Peter Baldwin, Victoria Yaneva, Kai North, Le An Ha, Yiyun Zhou, Alex J Mechaber, and Brian E Clauser. 2025. The vulnerability of ai-based scoring systems to gaming strategies: A case study. Journal of Educational Measurement

work page 2025
[3]

Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676

arXiv 2019
[4]

Marie Bexte, Andrea Horbach, and Torsten Zesch. 2023. Similarity-based content scoring-a more classroom-suitable alternative to instance-based scoring? In Findings of the association for computational linguistics: Acl 2023, pages 1892--1903

work page 2023
[5]

Sridevi Bonthu, S Rama Sree, and MHM Krishna Prasad. 2021. Automated short answer grading using deep learning: A survey. In Machine Learning and Knowledge Extraction: 5th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2021, Virtual Event, August 17--20, 2021, Proceedings 5, pages 61--78. Springer

2021
[6]

nswvtnvakgxpm

Yuning Ding, Brian Riordan, Andrea Horbach, Aoife Cahill, and Torsten Zesch. 2020. Don’t take “nswvtnvakgxpm” for an answer--the surprising vulnerability of automatic content scoring systems to adversarial input. In Proceedings of the 28th international conference on computational linguistics, pages 882--892

work page 2020
[7]

Anna Filighera, Sebastian Ochs, Tim Steuer, and Thomas Tregel. 2023. Cheating automatic short answer grading with the adversarial usage of adjectives and adverbs. International Journal of Artificial Intelligence in Education, pages 1--31

work page 2023
[8]

Anna Filighera, Tim Steuer, and Christoph Rensing. 2020. Fooling automatic short answer grading systems. In International conference on artificial intelligence in education, pages 177--190. Springer

work page 2020

Show all 17 references

[9]

Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342

2019 arXiv
[10]

Yaman Kumar, Swati Aggarwal, Debanjan Mahata, Rajiv Ratn Shah, Ponnurangam Kumaraguru, and Roger Zimmermann. 2019. Get it scored using autosas—an automated system for scoring short answers. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 9662--9669

2019
[11]

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234--1240

2020
[12]

Behzad Naderalvojoud and Tina Hernandez-Boussard. 2023. Improving machine learning with ensemble learning on observational healthcare data. In AMIA Annual Symposium Proceedings, volume 2023, page 521. American Medical Informatics Association

2023
[13]

Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer learning in biomedical natural language processing: an evaluation of bert and elmo on ten benchmarking datasets. arXiv preprint arXiv:1906.05474

2019 arXiv
[14]

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084

2019 arXiv
[15]

Johannes Schneider, Robin Richner, and Micha Riser. 2022. Towards trustworthy autograding of short, multi-lingual, multi-type answers. International Journal of Artificial Intelligence in Education, pages 1--31

2022
[16]

King Yiu Suen, Victoria Yaneva, Janet Mee, Yiyun Zhou, Polina Harik, et al. 2023. Acta: Short-answer grading in high-stakes medical exams. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 443--447

2023
[17]

Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading esol texts. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 180--189

2011

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[2] [2]

Peter Baldwin, Victoria Yaneva, Kai North, Le An Ha, Yiyun Zhou, Alex J Mechaber, and Brian E Clauser. 2025. The vulnerability of ai-based scoring systems to gaming strategies: A case study. Journal of Educational Measurement

work page 2025

[3] [3]

Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676

arXiv 2019

[4] [4]

Marie Bexte, Andrea Horbach, and Torsten Zesch. 2023. Similarity-based content scoring-a more classroom-suitable alternative to instance-based scoring? In Findings of the association for computational linguistics: Acl 2023, pages 1892--1903

work page 2023

[5] [5]

Sridevi Bonthu, S Rama Sree, and MHM Krishna Prasad. 2021. Automated short answer grading using deep learning: A survey. In Machine Learning and Knowledge Extraction: 5th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2021, Virtual Event, August 17--20, 2021, Proceedings 5, pages 61--78. Springer

2021

[6] [6]

nswvtnvakgxpm

Yuning Ding, Brian Riordan, Andrea Horbach, Aoife Cahill, and Torsten Zesch. 2020. Don’t take “nswvtnvakgxpm” for an answer--the surprising vulnerability of automatic content scoring systems to adversarial input. In Proceedings of the 28th international conference on computational linguistics, pages 882--892

work page 2020

[7] [7]

Anna Filighera, Sebastian Ochs, Tim Steuer, and Thomas Tregel. 2023. Cheating automatic short answer grading with the adversarial usage of adjectives and adverbs. International Journal of Artificial Intelligence in Education, pages 1--31

work page 2023

[8] [8]

Anna Filighera, Tim Steuer, and Christoph Rensing. 2020. Fooling automatic short answer grading systems. In International conference on artificial intelligence in education, pages 177--190. Springer

work page 2020