{"id":"bc6967fb-a4ae-4dbc-81d6-6dbed7902c19","arxiv_id":"2505.00061","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Adversarial training on simulated student gaming responses reduces false-positive rates in a medical short-answer grading system, with ensembles and GPT-4 prompting adding further defense.","lead":"Researchers tested how easily an automated medical exam grader could be fooled by students who do not know the answer, and whether training the system on those tricks makes it more robust. They found that adversarial training and model ensembles sharply cut false positives, and that GPT-4 can also catch many of the gaming attempts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's before/after FPR comparison for within-strategy training uses different test sets: 'before' is on all simulated responses, 'after' on a 30% subset, so the Mixed Responses drop from 0.435 to 0.041 may not be a like-for-like comparison.","rationale":"The reader's weakest_assumption focused on the representativeness of simulated gaming responses, which is explicitly acknowledged in Section 7 and is a valid external-validity concern. However, the most load-bearing concern is internal: the headline numbers for within-strategy adversarial training are not computed on the same test set before and after training. Section 4.1 uses all artificial responses for the 'before' FPR, while Section 4.2 uses a 30% subset for AdvT1. This confound directly affects the specific values quoted in the strongest_claim (0.435 to 0.041). The cross-strategy AdvT2 result is on a comparable test set and still shows a large reduction (0.435 to 0.067), so the qualitative conclusion that adversarial training helps is likely robust. But the paper should either present paired before/after FPRs on identical test sets or explicitly acknowledge the mismatch and provide the NoAdvT FPR on the 30% subset. This is a concrete, actionable correction that does not change the overall conditional verdict: the paper should be accepted only if the recomputation confirms the reported reductions. I disagree with the reader's choice of weakest_assumption because the test-set mismatch is a more direct threat to the central numerical claim and is not flagged in the manuscript.","tokens_in":10037,"tokens_out":11611,"duration_ms":111480,"concrete_test":"Recompute the non-adversarially trained ACTA model's FPR on the exact 30% held-out simulated responses used in Experiment 1 (same random seed/split as the AdvT1 condition). For each strategy, compare this 'NoAdvT-on-30%' FPR to the reported AdvT1 FPR. If the NoAdvT-on-30% FPR for Mixed Responses is substantially lower than the reported 0.435, the claimed within-strategy reduction is overstated; the paper should then report paired before/after FPRs on identical test sets and recompute the corresponding bars in Figures 4–6.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central numerical claim in the strongest_claim is that within-strategy adversarial training drops the Mixed Responses FPR from 0.435 to 0.041. However, these two numbers are computed on different test sets. Section 4.1 states that the pre-training evaluation was done on 'the remaining 30% (10,890) combined with all artificial responses,' meaning the 'before' FPR for each strategy is computed on the full set of simulated responses (e.g., all 584 Mixed Responses). Section 4.2 states that Experiment 1 uses 'the remaining 30% of both artificial and authentic responses' for testing, so the 'after within-strategy' FPR is computed on only a 30% subset (about 175 Mixed Responses). Thus the reported reduction from 0.435 to 0.041 is not a paired, same-test-set comparison; it could be inflated or deflated by sampling variability in the 30% subset. The same mismatch affects Tables 3–4 and Figures 4–6 for the AdvT1 condition. The cross-strategy AdvT2 numbers are computed on the full held-out strategy set and are therefore comparable to the 'before' numbers, but the paper does not clearly distinguish this. This is an internal-validity issue distinct from the acknowledged representativeness limitation in Section 7, and it directly bears on the headline quantitative claim.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies vulnerability of the ACTA transformer-based automated short-answer grading system to three simulated gaming strategies (random words from the stem, ChatGPT summaries, and mixed correct/incorrect responses) using real medical exam data. It reports that adversarial training, both within-strategy (AdvT1) and cross-strategy (AdvT2), substantially reduces false positive rates on simulated gaming responses while maintaining an F1 of 0.98 on authentic responses, and that ensemble methods (majority vote, ridge regression) and GPT-4 prompting can further improve detection. The central quantitative claim is that the Mixed Responses FPR drops from 0.435 before training to 0.041 after within-strategy training and 0.067 after cross-strategy training.","tokens_in":10270,"tokens_out":2936,"duration_ms":29967,"significance":"If the reported effects are robust, the paper addresses a real and important problem: maintaining integrity of automated scoring in high-stakes medical education. The study uses a large authentic dataset (36,735 responses), describes the simulation pipeline in detail, and compares several defense strategies across five embedding models and two ensemble methods, which is a useful contribution to the ASAG security literature. The authors are appropriately transparent about the simulated nature of the adversarial examples and the single-domain scope. However, the headline FPR reductions are weakened by an internal-validity issue in the before/after comparison and by the absence of uncertainty quantification, so the quantitative conclusions are not yet firmly established.","major_comments":[{"comment":"The 'before' and 'after' FPRs for the within-strategy experiment (AdvT1) are computed on different test sets, so the reported reductions are not like-for-like. Section 4.1 states that pre-training evaluation was done on 'the remaining 30% (10,890) combined with all artificial responses,' meaning the 'before' FPR for each strategy is based on the full set of simulated responses (e.g., all 584 Mixed Responses). Section 4.2 states that Experiment 1 uses 'the remaining 30% of both artificial and authentic responses' for testing, so the 'after within-strategy' FPR is based on only about 175 Mixed Responses. Thus Table 2's headline reduction from 0.435 to 0.041 could be inflated or deflated by sampling variability, and the same issue affects Tables 3–4 and Figures 4–6 for AdvT1. The authors should recompute the 'before' FPR on the same 30% held-out subset, or report both evaluations on the full set and the subset, with confidence intervals.","section":"§4.1 vs. §4.2"},{"comment":"No confidence intervals or significance tests accompany any FPR estimate. With roughly 175 graded Mixed Responses in the AdvT1 test set, a difference between 0.041 and 0.067 corresponds to about 7 versus 12 false positives, which is not statistically distinguishable. The claim that ridge regression 'outperformed' majority vote (e.g., 0.035 vs. 0.035 for Mixed Responses in Table 4) is similarly unsupported. The authors should report exact counts, binomial or bootstrap confidence intervals, and pairwise tests where comparisons are made.","section":"§4.2 and Tables 2–4"},{"comment":"The paper explicitly concedes that the adversarial examples are 'simulated approximations of gaming strategies, rather than authentic, organically derived examples from real-world test-takers.' This concession is appropriate, but it directly limits the operational conclusion that adversarial training 'significantly reduces' the system's susceptibility. The reported gains are measured on the same simulation distribution used for training, so they may not transfer to real examinee gaming behavior. The authors should either temper the general claim (e.g., restrict it to simulated gaming responses) or provide a concrete out-of-distribution test, such as evaluating on a separate set of human-generated gaming attempts or on responses from a different exam cohort.","section":"§7"}],"minor_comments":[{"comment":"There are typographical inconsistencies, e.g., 'Majority V ote' appears in Table 4 and Section 6, and the column header 'Acta Model' in Table 3 should be 'ACTA Model.'","section":"§4.3 and Tables 3–4"},{"comment":"The text says 'Figures 5 and 6 and Tables 3 and 4,' but Figures 4–6 are all referenced; please correct the cross-references.","section":"§4.3"},{"comment":"The prompt engineering experiment is based on only 100 samples of each data type, and only the results of the first prompting strategy are reported. Please state the sample size limitation explicitly in the table or text, and explain why the other two strategies are omitted.","section":"§4.4"},{"comment":"When describing Baldwin et al. (2025), the text says 'the first five strategies lead to a success rate between 6% to 16%,' but the reporting of strategy 6 (multiple responses) is the most effective; it would be clearer to present these as a list or table.","section":"§2"},{"comment":"Appendix A contains only the sentence 'This is an appendix.' It should be removed or filled with actual content.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper relies heavily on self-cited prior work (Baldwin et al., 2025) to define the gaming strategies; this is not inherently problematic, but the authors should ensure no circularity in the evaluation—for instance, by noting that the simulation procedure is independently implemented. The scope fits cs.CL and educational NLP venues. The data and code are not mentioned as available; sharing the simulated responses and evaluation scripts would strengthen the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a useful empirical study of countermeasures against gaming in a medical short-answer grading system, with a real dataset and a sensible experiment suite, but the headline FPR reductions are not all computed on the same test sets, so treat the specific numbers with care.\n\nWhat is new: the paper takes gaming strategies identified in prior work by the same group (Baldwin et al. 2025) and systematically evaluates adversarial training, five embedding models, majority-vote and ridge ensembles, and GPT-4 prompting. That systematic comparison is the contribution. The main positive result is credible in direction: after adversarial training, the model keeps F1 around 0.98 on real responses while lowering false positives on simulated gaming responses. The cross-strategy experiment (AdvT2) is a good design and suggests partial transfer. The GPT-4 prompt engineering results, especially for mixed responses, are also worth attention.\n\nSoft spots, in proportion. The most important is an internal-validity mismatch in the central before/after comparison. The 'before' FPRs are computed on all simulated responses (e.g., all 584 Mixed Responses), while the 'after within-strategy' FPRs (AdvT1) are computed on a 30% held-out subset (about 175). So the drop from 0.435 to 0.041 for Mixed Responses is not a same-test-set comparison; sampling variability alone could move those numbers. The AdvT2 cross-validation numbers are on the full held-out strategy and are comparable to 'before', but the paper does not clearly separate that. No confidence intervals or significance tests are reported anywhere, which matters when test sets are this small. The acknowledged limitation in Section 7—simulated responses are not authentic examinee behavior—is real and honestly stated, but it means operational transfer is unknown. Minor issues: the prompt-engineering section reports only the best of three strategies and uses only 100 samples per condition.\n\nWho this is for: people building or defending automated short-answer grading systems, particularly in medical education. It deserves a serious referee; the flaws are addressable with same-test-set comparisons, confidence intervals, and ideally a code/data release. I would accept it for review but not without revision.","headline":"Useful systematic comparison of countermeasures against gaming in medical short-answer grading, but the headline FPR drops are not all like-for-like because before/after numbers come from different test sets.","tokens_in":10862,"tokens_out":2126,"would_cite":true,"duration_ms":22918,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mixed-answer gaming false positives cut from 43.5% to 4.1%","keywords":["automated short-answer grading","adversarial training","gaming strategies","false positive rate","transformer-based scoring","medical education assessment","prompt engineering","ensemble methods"],"falsifier":"Collect authentic gaming attempts from real test-takers, for example from low-stakes practice administrations or from responses flagged during operational grading, and run the adversarially trained ACTA system on them. If the false positive rate on those authentic gaming responses is close to the untrained baseline (for example, above 0.20 for mixed-answer responses), the paper's defense claim would not transfer to practice.","tokens_in":9818,"feed_emoji":"🛡️","tokens_out":10716,"duration_ms":87621,"temperature":0.7,"pith_summary":"This paper asks whether automated short-answer grading systems for medical exams can be tricked into giving credit for answers that deserve none, and whether the trick can be taught away. Working with ACTA, a transformer-based grader that matches a student response to the closest human-scored answer, the authors simulate three gaming strategies an unprepared examinee might try: sampling words from the question stem, submitting a ChatGPT summary of the clinical vignette, and listing multiple plausible answers at once. All three fool the untrained system, with the multiple-answer tactic producing a false positive rate of 43.5%. Adding simulated gaming responses to the training data cuts that rate to 4.1% when the same strategy appears in training and to 6.7% when training covers only the other strategies, while accuracy on real responses stays at F1 = 0.98. The authors conclude that adversarial training, ensemble scoring, and LLM-based prompt engineering can each meaningfully harden such systems, supporting the reliability and fairness of AI-based grading in high-stakes settings.","feed_headline":"Mixed-answer gaming false positives cut from 43.5% to 4.1%","feed_subtitle":"Simulated gaming responses in training block the most effective tactic while real-response accuracy stays at F1 0.98.","key_machinery":"The central object is ACTA, a transformer-based short-answer grader that embeds each response with Sentence-BERT and assigns the label of the most similar human-scored response when similarity exceeds a threshold. The paper's core mechanism is adversarial training by data augmentation: simulated gaming responses -- random stem words, ChatGPT vignette summaries, and mixed correct-plus-incorrect answers -- are added to the training set so the similarity-matching model learns to label them incorrect. The defense is reinforced by ensemble aggregation, majority vote and ridge regression over five medical BERT embedding models, and by a separate prompt-engineering route in which GPT-4 is asked to score responses given the question context.","core_discovery":"The central claim is that transformer-based short-answer graders are substantially gameable but can be hardened. ACTA, a similarity scorer built on Sentence-BERT embeddings, awards credit to 43.5% of \"mixed\" responses that list multiple plausible diagnoses, 18.9% of ChatGPT-generated vignette summaries, and 6.1% of random words sampled from the question stem. Adversarial training on simulated gaming responses reduces the mixed-response false positive rate to 4.1% when the same strategy is seen in training and to 6.7% when the model is trained only on the other two strategies, while keeping F1 on authentic responses at 0.98. Combining five medical-domain BERT embeddings through majority vote or ridge regression lowers false positives further than any single model, and prompting GPT-4 to score with the question supplied recognises mixed-answer gaming with a 1% false positive rate. The authors take these results as evidence that both data augmentation and LLM-based scoring can materially reduce the damage from known gaming tactics.","pith_inferences":["If real test-takers game the system in ways resembling the simulated strategies, the measured FPR reductions suggest operational ASAG systems could avoid awarding undeserved credit for the most common gaming attempts, but the transfer depends on the authenticity of the simulations — a limitation the paper itself flags.","The finding that training on \"strong\" gaming examples transfers better to unseen strategies than training on \"weak\" ones suggests a testable design principle: when building adversarial training sets, prioritise collecting or synthesising the most effective known attacks, not the easiest ones.","The GPT-4 result on mixed responses hints that an LLM scoring layer could act as a complement to similarity-based graders where those are most gameable, though the LLM's weaker performance on random word sampling shows no single model should be the only defense.","A direct next step would be to collect authentic gaming attempts from low-stakes or practice exams and measure how well adversarially trained models score those naturally occurring responses; the paper's Section 7 limitation makes that the natural confirmation test."],"forward_implications":["Adversarial training on simulated gaming responses cuts the false positive rate for the most damaging strategy (mixed responses) from 0.435 to 0.041 when the strategy is included in training, with no drop in real-response accuracy (F1 = 0.98).","Cross-strategy adversarial training transfers partial protection: training on two gaming strategies lowers the mixed-response FPR to 0.067, suggesting that defenses built on known tactics generalise to some degree to unseen tactics.","Ensemble scoring with majority vote, and especially ridge regression across five medical BERT embeddings, yields lower false positive rates than any individual embedding model, making ensembles a useful second line of defense.","Prompt-engineering GPT-4 to score responses with the question supplied recognises mixed-answer gaming with FPR 0.01 and vignette summaries with FPR 0.03, outperforming adversarial training on those strategies, while remaining weaker on random stem-word sampling (FPR 0.11)."],"supporting_citations":[{"why":"Defines the gaming strategies the paper simulates and reports their success rates, including the 57% success of listing multiple responses.","marker":"Baldwin et al. (2025)"},{"why":"Demonstrates that a nonsensical string can receive a correct score, motivating the vulnerability analysis.","marker":"Ding et al. (2020)"},{"why":"Introduces ACTA, the transformer-based similarity scoring system evaluated throughout.","marker":"Suen et al. (2023)"},{"why":"Provides Sentence-BERT, the embedding model used by ACTA for response matching.","marker":"Reimers and Gurevych (2019)"},{"why":"Documents GPT-4, the model used for the prompt-engineering experiments.","marker":"Achiam et al. (2023)"},{"why":"Supplies Clinical-BERT, one of the five medical-domain embeddings combined in the ensemble defenses.","marker":"Huang et al. (2019)"}],"fun_headline_variants":["Grading AI fraud drops from 43.5% to 4.1% with adversarial training","Hardening short-answer graders: 43.5% exploit rate cut to 4.1%","Medical grading AI becomes 10x more robust to gaming attacks","Adversarial training slashes false positives in automated grading","GPT-4 plus ensemble defense thwarts short-answer grading cheats"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The simulated gaming responses are representative enough of how real examinees would actually game the system that the measured false-positive reductions would hold during operational use.","fun_headline_variants_meta":{"raw":{"variants":["Grading AI fraud drops from 43.5% to 4.1% with adversarial training","Hardening short-answer graders: 43.5% exploit rate cut to 4.1%","Medical grading AI becomes 10x more robust to gaming attacks","Adversarial training slashes false positives in automated grading","GPT-4 plus ensemble defense thwarts short-answer grading cheats"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000617,"raw_usage":{"total_tokens":2842,"prompt_tokens":902,"completion_tokens":1940,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1836}},"tokens_in":518,"tokens_out":1940,"duration_ms":12019,"temperature":1.0,"reasoning_tokens":1836,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:55:05.938902+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect authentic gaming attempts from real test-takers, for example from low-stakes practice administrations or from responses flagged during operational grading, and run the adversarially trained ACTA system on them. If the false positive rate on those authentic gaming responses is close to the untrained baseline (for example, above 0.20 for mixed-answer responses), the paper's defense claim would not transfer to practice.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the gaming strategies the paper simulates and reports their success rates, including the 57% success of listing multiple responses."},{"cited_title":"nswvtnvakgxpm","cited_arxiv_id":null,"evidence_quote":"Demonstrates that a nonsensical string can receive a correct score, motivating the vulnerability analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces ACTA, the transformer-based similarity scoring system evaluated throughout."}],"review_version":1}