{"id":"85787e5c-ed1f-4e39-9eb4-7860e49e62a8","arxiv_id":"2607.16085","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A rank-correlation penalty on the training loss of text- and audio-based L2 speaking auto-markers reduces their over-reliance on word count and speaking time to human-rater levels while keeping overall accuracy competitive.","lead":"Automated graders of spoken English exams can learn to lean heavily on response length, letting test-takers raise scores by simply talking more. This paper adds a training penalty that caps how strongly a grader may reward exploitable signals like word count, letting operators dial reliance down to the level human raters show.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The malpractice-suppression claim is untested: no attack or adversarial-input experiment shows that lowering proxy correlation makes the grader harder to exploit.","rationale":"The reader's weakest assumption is exactly the exploitability bridge. The paper demonstrates a controllable reduction in a specific, externally computable correlation, but the abstract and conclusions translate this into reduced malpractice. Since no attack or adversarial-input experiment is run, the central practical claim is conditional. This does not overturn the positive-proxy results for the text and audio graders (ModernBERT and wav2vec), where the reduction in ρ(ŷ,ẑ) is real and selectivity is partially checked; it means the paper should be accepted with the caveat that 'malpractice-suppression' requires additional validation. A secondary internal inconsistency—Eq. 8 is not sign-agnostic for negative proxies, since penalizing λρ(ŷ,ẑ) drives a negative correlation further negative rather than toward the human reference—further supports keeping the verdict conditional, but the primary load-bearing gap is the missing exploitability test. Both concerns are addressable and do not require rejecting the paper.","tokens_in":12882,"tokens_out":6189,"duration_ms":60726,"concrete_test":"Train/load the ModernBERT and wav2vec graders at λ=0 and at the reported human-alignment λ (≈0.13 and ≈0.17). Construct paired manipulation inputs from evaluation responses: for text, duplicate clauses to increase #words while keeping content/topic fixed; for audio, insert silence/pauses to increase VAD time. Measure the score change Δŷ per unit Δproxy for each model. If the human-alignment model's score gain per word/second is not significantly smaller than the λ=0 baseline (or remains large), the correlation drop did not reduce exploitability and the malpractice-suppression claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central practical claim—that the method yields a 'malpractice-suppression mode'—rests on identifying shortcut reliance with the output-level Spearman gap ρ(ŷ,ẑ) vs ρ(y,ẑ) (Eq. 5). But Eq. 8 directly optimizes this same quantity, so the observed reduction in Figures 2–3 is expected and does not by itself establish that the grader is less gameable. A learner could exploit the margin: Spearman is computed over the whole n=300 evaluation distribution, so a model with low overall rank correlation may still give large score gains for adding words/pauses at the high end. Exploitation could also move to a correlated but unmeasured proxy (e.g., template vocabulary, pause structure); the ASR-confidence selectivity check covers only one non-targeted feature. The paper motivates the problem with the repeated-paragraph attack on e-rater [5] but never runs the spoken analogue or any attack. Thus the 'human-alignment' and 'malpractice-suppression' operating modes are defined by correlation matching, not by demonstrated resistance to score manipulation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-time penalty for fine-tuned encoder-based L2 spoken English auto-markers: Eq. (8) adds λ_j ρ(ŷ, ẑ_j) to the negative rank-correlation loss, so that the model is simultaneously optimized for agreement with human scores and for low correlation with an externally computable proxy (word count for the ModernBERT text grader, VAD time for the wav2vec 2.0 audio grader). Experiments on the Speak & Improve 2025 corpus show that both baselines have higher proxy correlations than human raters, and that increasing λ reduces the targeted correlation monotonically, crossing the human-reference level at λ_WC≈0.13 and λ_VAD≈0.17. The authors interpret these as human-alignment and malpractice-suppression operating modes.","tokens_in":13012,"tokens_out":4477,"duration_ms":45517,"significance":"The baseline diagnostic—quantifying implicit shortcut reliance as output-level Spearman correlation with proxies—is useful, and the proposed output-level penalty is a simple, model-agnostic intervention that does not require explicit feature access. The paper is strengthened by reproducible methodology, a 10-seed ensemble for the text grader, and the cross-modal consistency of the λ-trends. However, the central practical claim, that the method yields a 'malpractice-suppression mode' that limits score gaming, is not validated by any attack or adversarial-input experiment. The observed reduction in ρ(ŷ, ẑ_j) is largely a mathematical consequence of Eq. (8), so the paper currently demonstrates control of a diagnostic quantity rather than demonstrated resistance to exploitation.","major_comments":[{"comment":"The monotone decrease in ρ(ŷ, ẑ_j) with λ_j is expected by construction because L = −ρ(ŷ,y) + λ_j ρ(ŷ,ẑ_j) is minimized. The novel practical claim is the 'malpractice-suppression mode', but no attack experiment (the spoken analogue of the repeated-paragraph attack in [5]), no marginal/tail analysis, and no test on unmeasured proxies is provided. A low overall Spearman correlation over n=300 does not preclude large score gains from adding words/pauses in the high-scoring tail, and exploitation could shift to a correlated but unmeasured feature (e.g., pause structure, template vocabulary). The selectivity check on ASR confidence covers only one non-targeted feature. I request either an adversarial/manipulation experiment or a clearly stated weakening of the malpractice claim in the abstract and conclusions.","section":"Section VI-A/B, Figs. 2-3, Eq. (8)"},{"comment":"All headline quantities are point estimates without uncertainty. The evaluation set has n=300 submissions, giving a Spearman standard error of roughly 0.06, and the ModernBERT results are reported as the mean over 10 seeds without a spread, while wav2vec uses a single seed. The λ-crossing points (≈0.13, ≈0.17) are therefore not statistically grounded, and the two operating modes are defined by point estimates. Moreover, the objective in Eq. (8) is optimized within small batches (16–64) with differentiable soft ranks, but the reported figures show global evaluation-set Spearman correlations; the monotone transfer from batch-level optimization to the full test set is asserted rather than checked. Please report bootstrap confidence intervals, per-seed ranges, and per-part evaluation curves.","section":"Sections V-B/V-C, Tables II-III"},{"comment":"The submission-level combination uses per-part z-score normalization and equal part weights, but the proxy distributions (word count, VAD time) differ substantially across parts. This arbitrary equal weighting can affect the combined Spearman correlations with the proxies and hence the location of the λ-crossing points. The paper should report part-level correlations or pooled raw predictions and justify the weighting, or show that the qualitative trends are invariant to the combination scheme.","section":"Section V-C, Eqs. (9)-(10)"}],"minor_comments":[{"comment":"The '≫' criterion for shortcut presence is informal. A threshold or effect-size convention would make the diagnostic reproducible.","section":"Eq. (5)"},{"comment":"Typo: 'introduced wih a controllable weight' should read 'with'.","section":"Section VI-A"},{"comment":"Typo: 'minimal cost to accuracy,;' contains a stray comma.","section":"Section VII"},{"comment":"Grammar: 'it applicable to feature-based graders' should be 'it is applicable'.","section":"Section IV"},{"comment":"The dashed human-reference lines are informative, but adding uncertainty bands or error bars around the model curves would greatly improve the interpretability of the λ-crossings.","section":"Figures 2-3"}],"recommendation":"major_revision","confidential_remarks":"The central claim of 'malpractice suppression' currently rests on a correlation-matching argument rather than on demonstrated robustness to score manipulation. This is fixable within the manuscript's scope by adding an adversarial or controlled-manipulation evaluation, or by reframing the contribution as output-level shortcut-correlation control. If the authors are unwilling to do either, I would not recommend acceptance for a venue that emphasizes practical assessment integrity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Gao, Gales, and Knill propose a training-time penalty on the Spearman correlation between model predictions and an externally computable proxy (word count, VAD time), for fine-tuned encoder graders where the proxy is not an explicit input. That is a real gap in the mitigation literature: RRR, COMI, FiLoRA all need differentiable feature access. The paper shows clearly that both a ModernBERT text grader (ρ(ŷ,#words)=0.874 vs human 0.659) and a wav2vec audio grader (ρ(ŷ,VAD)=0.705 vs 0.450) over-rely on these proxies, and that sweeping λ reduces the targeted correlation toward and below the human reference while ρ(ŷ,y) stays competitive. The selectivity check (ASR confidence largely unaffected) and the 10-seed ensemble for ModernBERT are genuine good practice. The two operating modes — human-alignment and malpractice-suppression — are a useful way to frame the dial.\n\nThe soft spots are real but proportionate. First, the headline effect is partly definitional: Eq. 8 directly optimizes the quantity plotted in Figures 2 and 3, so the monotone decline is expected. The non-circular content is that the effect transfers to the held-out set, does not crater accuracy, and is selective; those are worth reporting, but the paper should say the decline itself is by construction. Second, the 'malpractice-suppression' mode is never tested as such. The paper motivates with the repeated-paragraph essay attack but runs no spoken analogue or any adversarial input; lowering bulk Spearman correlation could leave margin exploitability intact or shift exploitation to an unmeasured proxy. The claim should be softened to 'reduces measured proxy correlation' unless attacked. Third, there are no uncertainty bounds anywhere — n=300 gives SE≈0.06 on the headline correlations, so the crossing points λ≈0.13/0.17 are given without error bars, and the wav2vec results are single-seed. Fourth, the sign-agnosticism claim is wrong as written: +λρ(ŷ,ẑ) pushes correlations downward, which is the wrong direction for a negative proxy like disfluency count; you'd need absolute value or a signed λ. Minor fix, but it should be fixed before publication.\n\nWho this is for: anyone building or auditing end-to-end automated speaking or essay scoring systems, and researchers working on shortcut mitigation in encoder-based models. The core method and baseline findings are worth a serious referee; the paper needs revisions on the malpractice claim, uncertainty, and the sign issue, not a desk reject.","headline":"Genuinely new output-level rank penalty for shortcut reliance in encoder graders, well-executed on two models — but the malpractice-suppression claim is correlation matching, not attack resistance.","tokens_in":13656,"tokens_out":4940,"would_cite":true,"duration_ms":40221,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single training term lets spoken-English auto-markers shed their hidden dependence on word count and speaking time.","keywords":["automated speaking assessment","shortcut learning","rank correlation penalty","Spearman correlation","ModernBERT","wav2vec 2.0","malpractice suppression","L2 English proficiency"],"falsifier":"Take the text grader at λ≈0.13, where word-count correlation matches the human reference, and feed it a response that repeats the same content to inflate length. If the inflated response scores substantially above a human rating while the correlation diagnostic says reliance is at human level, the central claim collapses. Equivalently, an attack that raises word count without improving proficiency and still produces a score jump at the 'aligned' setting would falsify the malpractice-suppression conclusion.","tokens_in":12609,"feed_emoji":"🎯","tokens_out":3842,"duration_ms":36067,"temperature":0.7,"pith_summary":"The paper argues that fine-tuned encoder-based graders of L2 spoken English quietly over-rely on surface proxies — word count for text-based graders, speaking duration for audio-based graders — and that this over-reliance is both measurable and correctable. It introduces a training objective that adds a weighted rank-correlation penalty between predictions and any chosen proxy to the usual score-correlation loss. Sweeping the penalty weight produces two useful operating points: a human-alignment mode, where the grader's proxy dependence matches human raters' at small accuracy cost, and a malpractice-suppression mode, where dependence is pushed below the human reference at an accepted accuracy cost. The experiments show the penalty mainly affects the targeted proxy and its close relatives, leaving unrelated features such as average ASR confidence largely untouched.","feed_headline":"One training term pulls spoken-English auto-markers off their length shortcut","feed_subtitle":"A rank-correlation penalty trims word-count and speaking-time dependence, letting graders match human raters — or suppress exploitability ha","key_machinery":"The key object is a differentiable Spearman rank correlation used as both task loss and regulariser. Because the discrete rank operator is non-differentiable, ranks are replaced by a soft, differentiable approximation, making the objective L = −ρ(ŷ,y) + Σ_{j∈J} λ_j ρ(ŷ,ẑ_j) trainable end-to-end. The penalty acts purely on predicted outputs and externally computable proxy features, so it requires no access to the encoder's internal representations and no gradient path from score to feature — which is exactly what previous shortcut-mitigation methods required. The weight λ controls how far the proxy correlation is pushed down, and the same formulation works for both text and audio graders.","core_discovery":"The central claim is that implicit shortcut reliance in end-to-end auto-markers can be controlled at the output level, without explicit feature inputs or encoder modification, by optimising L = −ρ(ŷ,y) + Σ λ_j ρ(ŷ,ẑ_j), where ρ is a differentiable Spearman rank correlation. For a ModernBERT text grader with word count as proxy, the word-count correlation falls from about 0.87 to cross the human reference of 0.66 at λ≈0.13; for a wav2vec 2.0 audio grader with VAD time as proxy, the correlation falls from about 0.71 to cross the human reference of 0.45 at λ≈0.17. At those crossing points overall accuracy stays competitive, and the penalty is selective: non-targeted features remain nearly uncha","pith_inferences":["The evidence is correlational; the malpractice-suppression claim would be directly testable by running adversarial submissions — e.g., repeated content or artificially lengthened audio — and checking whether score inflation actually disappears at the human-alignment λ. The paper itself does not run such an attack.","Because only one proxy is penalised at a time, a determined test-taker could shift exploitation to a correlated or unmeasured proxy; the paper notes that highly correlated neighbour features are also suppressed, but a multi-proxy version with J>1 is a natural next step.","The method's guarantee is about aggregate correlation, not worst-case exploitability; a grader tuned to match human-rater correlation on the bulk of responses could still be gameable at the margin, so deployment should pair this penalty with explicit robustness checks.","Since the penalty never touches the encoder, it could likely be applied as a post-training adjustment to already-deployed scoring systems, lowering the cost of remediation for assessment providers."],"forward_implications":["If correct, operators of high-stakes spoken English tests can tune a single scalar to make auto-markers' reliance on word count or speaking time match human raters, closing the 'keep talking' malpractice route.","The same output-level penalty applies to any computable, human-interpretable feature, so the method can be repurposed as new exploitable shortcuts are identified.","Because the penalty is selective, graders can suppress targeted shortcuts without broadly degrading performance, as shown by stable average-ASR-confidence correlations at moderate λ.","The framework gives a principled cross-modal comparison: the same diagnostic and penalty work for text-based and audio-based graders under one formulation."],"fun_headline_variants":["New loss term makes AI English graders stop gaming word count","Auto-markers taught to ignore length, match human ratings","Rank-correlation penalty kills shortcut exploits in spoken English scoring","End-to-end graders retrained to avoid word-count shortcut","One tweak: auto-markers stop relying on how long you talk"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the gap between the model's rank correlation with a proxy and the human raters' rank correlation with that same proxy faithfully measures how much a test-taker can exploit the proxy to inflate scores; the paper never tests actual attempts to game the grader.","fun_headline_variants_meta":{"raw":{"variants":["New loss term makes AI English graders stop gaming word count","Auto-markers taught to ignore length, match human ratings","Rank-correlation penalty kills shortcut exploits in spoken English scoring","End-to-end graders retrained to avoid word-count shortcut","One tweak: auto-markers stop relying on how long you talk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000785,"raw_usage":{"total_tokens":3314,"prompt_tokens":771,"completion_tokens":2543,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":2459}},"tokens_in":515,"tokens_out":2543,"duration_ms":17098,"temperature":1.0,"reasoning_tokens":2459,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T21:24:35.944357+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the text grader at λ≈0.13, where word-count correlation matches the human reference, and feed it a response that repeats the same content to inflate length. If the inflated response scores substantially above a human rating while the correlation diagnostic says reliance is at human level, the central claim collapses. Equivalently, an attack that raises word count without improving proficiency and still produces a score jump at the 'aligned' setting would falsify the malpractice-suppression conclusion.","supporting_citations":[],"review_version":1}