Pith. sign in

REVIEW 3 major objections 5 minor 43 references

Controlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single training term lets spoken-English auto-markers shed their hidden dependence on word count and speaking time.

desk verdict Genuinely new output-level rank penalty for shortcut reliance in encoder graders, well-executed on two models — but the malpractice-suppression claim is correlation matching, not attack resistance. read the letter →

arxiv 2607.16085 v1 pith:QNWWYUEV submitted 2026-07-17 cs.CL eess.AS

Shilin Gao , Mark J. F. Gales , Kate M. Knill This is my paper · ORCID
classification cs.CLeess.AS
keywords automatedspeakingassessmentshortcutlearningrankcorrelationpenaltySpearmanModernBERTwav2vec2.0malpracticesuppressionL2Englishproficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that fine-tuned encoder-based graders of L2 spoken English quietly over-rely on surface proxies — word count for text-based graders, speaking duration for audio-based graders — and that this over-reliance is both measurable and correctable. It introduces a training objective that adds a weighted rank-correlation penalty between predictions and any chosen proxy to the usual score-correlation loss. Sweeping the penalty weight produces two useful operating points: a human-alignment mode, where the grader's proxy dependence matches human raters' at small accuracy cost, and a malpractice-suppression mode, where dependence is pushed below the human reference at an accepted accuracy cost. The experiments show the penalty mainly affects the targeted proxy and its close relatives, leaving unrelated features such as average ASR confidence largely untouched.

What carries the argument

The key object is a differentiable Spearman rank correlation used as both task loss and regulariser. Because the discrete rank operator is non-differentiable, ranks are replaced by a soft, differentiable approximation, making the objective L = −ρ(ŷ,y) + Σ_{j∈J} λ_j ρ(ŷ,ẑ_j) trainable end-to-end. The penalty acts purely on predicted outputs and externally computable proxy features, so it requires no access to the encoder's internal representations and no gradient path from score to feature — which is exactly what previous shortcut-mitigation methods required. The weight λ controls how far the proxy correlation is pushed down, and the same formulation works for both text and audio graders.

What would settle it

Take the text grader at λ≈0.13, where word-count correlation matches the human reference, and feed it a response that repeats the same content to inflate length. If the inflated response scores substantially above a human rating while the correlation diagnostic says reliance is at human level, the central claim collapses. Equivalently, an attack that raises word count without improving proficiency and still produces a score jump at the 'aligned' setting would falsify the malpractice-suppression conclusion.

Watch

Extended reading notes

Core claim

The central claim is that implicit shortcut reliance in end-to-end auto-markers can be controlled at the output level, without explicit feature inputs or encoder modification, by optimising L = −ρ(ŷ,y) + Σ λ_j ρ(ŷ,ẑ_j), where ρ is a differentiable Spearman rank correlation. For a ModernBERT text grader with word count as proxy, the word-count correlation falls from about 0.87 to cross the human reference of 0.66 at λ≈0.13; for a wav2vec 2.0 audio grader with VAD time as proxy, the correlation falls from about 0.71 to cross the human reference of 0.45 at λ≈0.17. At those crossing points overall accuracy stays competitive, and the penalty is selective: non-targeted features remain nearly uncha

Load-bearing premise

The load-bearing premise is that the gap between the model's rank correlation with a proxy and the human raters' rank correlation with that same proxy faithfully measures how much a test-taker can exploit the proxy to inflate scores; the paper never tests actual attempts to game the grader.

Editorial extensions

If this is right

  • If correct, operators of high-stakes spoken English tests can tune a single scalar to make auto-markers' reliance on word count or speaking time match human raters, closing the 'keep talking' malpractice route.
  • The same output-level penalty applies to any computable, human-interpretable feature, so the method can be repurposed as new exploitable shortcuts are identified.
  • Because the penalty is selective, graders can suppress targeted shortcuts without broadly degrading performance, as shown by stable average-ASR-confidence correlations at moderate λ.
  • The framework gives a principled cross-modal comparison: the same diagnostic and penalty work for text-based and audio-based graders under one formulation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The evidence is correlational; the malpractice-suppression claim would be directly testable by running adversarial submissions — e.g., repeated content or artificially lengthened audio — and checking whether score inflation actually disappears at the human-alignment λ. The paper itself does not run such an attack.
  • Because only one proxy is penalised at a time, a determined test-taker could shift exploitation to a correlated or unmeasured proxy; the paper notes that highly correlated neighbour features are also suppressed, but a multi-proxy version with J>1 is a natural next step.
  • The method's guarantee is about aggregate correlation, not worst-case exploitability; a grader tuned to match human-rater correlation on the bulk of responses could still be gameable at the margin, so deployment should pair this penalty with explicit robustness checks.
  • Since the penalty never touches the encoder, it could likely be applied as a post-training adjustment to already-deployed scoring systems, lowering the cost of remediation for assessment providers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a training-time penalty for fine-tuned encoder-based L2 spoken English auto-markers: Eq. (8) adds λ_j ρ(ŷ, ẑ_j) to the negative rank-correlation loss, so that the model is simultaneously optimized for agreement with human scores and for low correlation with an externally computable proxy (word count for the ModernBERT text grader, VAD time for the wav2vec 2.0 audio grader). Experiments on the Speak & Improve 2025 corpus show that both baselines have higher proxy correlations than human raters, and that increasing λ reduces the targeted correlation monotonically, crossing the human-reference level at λ_WC≈0.13 and λ_VAD≈0.17. The authors interpret these as human-alignment and malpractice-suppression operating modes.

Significance. The baseline diagnostic—quantifying implicit shortcut reliance as output-level Spearman correlation with proxies—is useful, and the proposed output-level penalty is a simple, model-agnostic intervention that does not require explicit feature access. The paper is strengthened by reproducible methodology, a 10-seed ensemble for the text grader, and the cross-modal consistency of the λ-trends. However, the central practical claim, that the method yields a 'malpractice-suppression mode' that limits score gaming, is not validated by any attack or adversarial-input experiment. The observed reduction in ρ(ŷ, ẑ_j) is largely a mathematical consequence of Eq. (8), so the paper currently demonstrates control of a diagnostic quantity rather than demonstrated resistance to exploitation.

major comments (3)
  1. [Section VI-A/B, Figs. 2-3, Eq. (8)] The monotone decrease in ρ(ŷ, ẑ_j) with λ_j is expected by construction because L = −ρ(ŷ,y) + λ_j ρ(ŷ,ẑ_j) is minimized. The novel practical claim is the 'malpractice-suppression mode', but no attack experiment (the spoken analogue of the repeated-paragraph attack in [5]), no marginal/tail analysis, and no test on unmeasured proxies is provided. A low overall Spearman correlation over n=300 does not preclude large score gains from adding words/pauses in the high-scoring tail, and exploitation could shift to a correlated but unmeasured feature (e.g., pause structure, template vocabulary). The selectivity check on ASR confidence covers only one non-targeted feature. I request either an adversarial/manipulation experiment or a clearly stated weakening of the malpractice claim in the abstract and conclusions.
  2. [Sections V-B/V-C, Tables II-III] All headline quantities are point estimates without uncertainty. The evaluation set has n=300 submissions, giving a Spearman standard error of roughly 0.06, and the ModernBERT results are reported as the mean over 10 seeds without a spread, while wav2vec uses a single seed. The λ-crossing points (≈0.13, ≈0.17) are therefore not statistically grounded, and the two operating modes are defined by point estimates. Moreover, the objective in Eq. (8) is optimized within small batches (16–64) with differentiable soft ranks, but the reported figures show global evaluation-set Spearman correlations; the monotone transfer from batch-level optimization to the full test set is asserted rather than checked. Please report bootstrap confidence intervals, per-seed ranges, and per-part evaluation curves.
  3. [Section V-C, Eqs. (9)-(10)] The submission-level combination uses per-part z-score normalization and equal part weights, but the proxy distributions (word count, VAD time) differ substantially across parts. This arbitrary equal weighting can affect the combined Spearman correlations with the proxies and hence the location of the λ-crossing points. The paper should report part-level correlations or pooled raw predictions and justify the weighting, or show that the qualitative trends are invariant to the combination scheme.
minor comments (5)
  1. [Eq. (5)] The '≫' criterion for shortcut presence is informal. A threshold or effect-size convention would make the diagnostic reproducible.
  2. [Section VI-A] Typo: 'introduced wih a controllable weight' should read 'with'.
  3. [Section VII] Typo: 'minimal cost to accuracy,;' contains a stray comma.
  4. [Section IV] Grammar: 'it applicable to feature-based graders' should be 'it is applicable'.
  5. [Figures 2-3] The dashed human-reference lines are informative, but adding uncertainty bands or error bars around the model curves would greatly improve the interpretability of the λ-crossings.

Circularity Check

2 steps flagged · score 5.0 of 10

Targeted proxy-correlation reduction is written into the training objective; the human-alignment operating point is chosen by fitting λ, so the headline demonstration is partly definitional.

  1. self definitional [Section IV, Eq. 8; Section VI-A, Figure 2]
    "The full training objective to minimise is: L=−ρ( ˆy,y) + X_{j∈J} λ_j ρ(ˆy, ˆzj) ... As λ_WC increases, correlation with word count decreases steadily, crossing the human reference level at λ_WC ≈0.13"

    Equation 8 explicitly adds the penalty term Σ λ_j ρ(ŷ,ẑ_j) to the loss being minimized. Therefore the observed decline in the targeted proxy correlation ρ(ŷ,ẑ) as λ_j increases is a direct consequence of the optimization objective, not an independent behavioral discovery. The 'human-alignment' point is then defined by sweeping λ until ρ(ŷ,ẑ)≈ρ(y,ẑ), so the matching is imposed by the selected hyperparameter rather than demonstrated as a property of the trained grader.

  2. fitted input called prediction [Section VI-A, Figure 2; Section VI-B, Figure 3]
    "crossing the human reference level at λ_WC ≈0.13, which defines the human-alignment operating mode, where shortcut reliance on word count matches the level exhibited by human raters (ρ( ˆy, ˆzWC)≈ρ(y, ˆzWC))"

    The 'human-alignment operating mode' is not an out-of-sample prediction; it is the value of the regularization weight λ at which the measured correlation exactly equals the human reference. Selecting λ to satisfy ρ(ŷ,ẑ)=ρ(y,ẑ) and then reporting that this equality holds is a tautological construction, akin to fitting a parameter to a target and then announcing that the target is met.

full rationale

The main circular component is the headline demonstration that the proposed penalty reduces the targeted proxy correlation: that reduction is explicitly encoded in Eq. 8, so the reported monotone decrease in Figures 2–3 is the optimization target itself. The 'human-alignment' crossing is likewise a fitted λ value, not an independent finding. However, the paper also contains non-circular empirical content: the baseline over-reliance relative to human raters (Tables II–III), the competitive accuracy at the alignment point, and the selectivity of the penalty (ASR confidence remaining stable) are not forced by the loss. The broader practical claim of 'malpractice-suppression' is not circular but is under-supported, since no adversarial or attack experiment is run; that is a validity/correctness concern, not a circularity one. No load-bearing self-citation chain was found. Overall, the central quantitative demonstration is partly definitional, so a moderate score is appropriate.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical or ontological entities are introduced; the 'human-alignment' and 'malpractice-suppression' modes are labels for λ ranges, not entities. The free parameters are the penalty weight λ (which defines the operating modes and the quantitative claims) and the soft-sort temperature. The four axioms record that the paper's shortcut construct, the precision of the human reference, the rank surrogate, and the batch-to-global transfer are assumed rather than demonstrated — each is load-bearing for the stated conclusions.

free parameters (2)
  • λ_j (λ_WC, λ_VAD) penalty weight = swept over [0, 0.30]; operating points λ_WC≈0.13, λ_VAD≈0.17
    Penalty strength in Eq. 8 that selects the operating mode. The two headline operating points are read off the evaluation-set sweep curves, not derived; the central quantitative claims ('matches human reference', 'malpractice-suppression mode') depend on this hand-chosen knob.
  • soft-sort regularization strength = 0.1
    Default temperature of the differentiable rank approximation [40] used for both the task and penalty Spearman terms (Section V-B); it controls gradient fidelity of the entire objective.
assumptions (4)
  • domain assumption Output-level Spearman correlation with an interpretable proxy, benchmarked against human-rater correlation (Eq. 5), measures shortcut reliance and malpractice risk.
    Section III-A defines shortcut reliance this way and Section IV turns it into the training loss. The paper validates neither that ρ(ŷ,ẑ) captures real gameability nor that reducing it to the human level removes exploitable reward.
  • domain assumption Human-rater feature correlations (Table I) are a precise, error-free reference target for auto-marker behavior.
    Figures 2–3 draw horizontal lines at these values to define the 'human-alignment' operating point; no uncertainty is reported even though n=300 evaluation submissions implies SE≈0.06 on each Spearman ρ.
  • standard math The fast-soft-sort differentiable rank approximation faithfully substitutes for Spearman rank correlation during gradient training.
    Section IV replaces the discrete rank operator with the approximation of [40] at temperature 0.1; fidelity is assumed from the cited open-source implementation.
  • domain assumption Within-batch rank-loss training transfers to the global evaluation-set rank correlations reported in the paper.
    Training optimizes per-batch SRC (batch sizes 16/64, tied part-level labels for Parts 1 and 5, Section V-B); all headline numbers (Tables II–III, Figures 2–3) are global evaluation-set Spearman values. The monotone λ-sweep behavior is assumed to transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Controlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers." pith.science (2026). https://pith.science/paper/QNWWYUEV

@misc{pith2026260716085,
  author       = {Pith},
  title        = {Pith review of: Controlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QNWWYUEV}},
  note         = {Machine review of arXiv:2607.16085}
}
read the original abstract

Increasingly, speech and language processing tasks take either audio or text directly rather than extracting features from these as the input to the classifier or regressor. Often these systems make use of complex, for example transformer-based, processes that have the ability to derive highly non-linear mappings between the input and the output. Unfortunately these systems can also learn ''shortcuts'' where the classifier is overly reliant on particular aspects of the input to yield the output. For the task of language proficiency assessment, this over-reliance can enable learners to increase their score by exploiting the shortcut rather than improving their ability. This paper introduces a novel training criterion that is able to reduce the classifier's reliance on shortcuts, thus for example limiting this option for malpractice in language assessment. This process is illustrated on two forms of assessment system, one based on the audio the other on the speech recognition text. The results show that, for both systems, there is higher correlations with features that could be exploited for malpractice than expected from the human reference, indicating an over-reliance on these features. By introducing the modified training criterion, this correlation can be reduced to be closer to the reference correlation.

Figures

Figures reproduced from arXiv: 2607.16085 by the authors.

Figure 1
Figure 1. Three typical spoken language assessment system architectures [26]: [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Spearman ρ between wav2vec-based model predictions and reference features across λ values. Dashed horizontal lines indicate the Spearman ρ between reference scores and each feature. (S&I evaluation set) VII. CONCLUSIONS AND FUTURE WORK This paper introduced a rank correlation penalty that dis￾courages fine-tuned encoder-based graders from over-relying on externally computable proxy features, operating at the output … view at source ↗
Figure 2
Figure 2. Spearman ρ between ModernBERT-based models’ predictions and reference features across λ values. Dashed horizontal lines indicate the Spearman ρ between reference scores and each feature. (S&I evaluation set) B. Audio as input: wav2vec 2.0 Table III shows that wav2vec 2.0 exhibits higher correlation with VAD time and word count than the human reference, confirming over-reliance consistent with Equation 5. As with the… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 3 canonical work pages

  1. [5]

    Stumping e-rater: challenging the validity of automated essay scoring,

    D. E. Powers, J. C. Burstein, M. Chodorow, M. E. Fowles, and K. Kukich, “Stumping e-rater: challenging the validity of automated essay scoring,”Computers in Human Behavior, vol. 18, no. 2, pp. 103–134, 2002. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S0747563201000528

    2002

  2. [1]

    BERT: Pre- training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language understanding,” inProc. NAACL-HLT, 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423/

    2019

  3. [2]

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference,

    B. Warner, A. Chaffin, B. Clavi ´e, O. Weller, O. Hallstr ¨om, S. Taghadouini, A. Gallagher, R. Biswas, F. Ladhak, T. Aarsen, G. T. Adams, J. Howard, and I. Poli, “Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference,” inProc. the 63rd Annual Meeting of the Association for Co...

    2025

  4. [3]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 12 449–12 460. [Online]. Available: https://proceedings.neurips.cc/ paper/2020/hash/92d1e1eb1cd6f9fba3227870bb6d7f07-Abstract.html

    2020

  5. [4]

    On the foundations of shortcut learning,

    K. Hermann, H. Mobahi, T. Fel, and M. Mozer, “On the foundations of shortcut learning,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 43 832–43 868. [Online]. Available: https://proceedings.iclr.cc/paper files/paper/2024/ hash/bf215fa7fe70a38c5e967e59c44a99d0-Abstract-Conference.html

    2024

  6. [6]

    Beyond essay length: Evaluating e-rater®’s performance on TOEFL® essays,

    M. Chodorow and J. Burstein, “Beyond essay length: Evaluating e-rater®’s performance on TOEFL® essays,” Educational Testing Service, Tech. Rep. RR-04-04, 2004. [Online]. Available: https: //www.ets.org/Media/Research/pdf/RR-04-04.pdf

    2004

  7. [7]

    Construct validity of e-rater in scoring TOEFL essays,

    Y . Attali, “Construct validity of e-rater in scoring TOEFL essays,” Educational Testing Service, Tech. Rep. RR-07-21,

  8. [8]

    Is a long essay always a good essay? The effect of text length on writing assessment,

    J. Fleckenstein, J. Meyer, T. Jansen, S. Keller, and O. K ¨oller, “Is a long essay always a good essay? The effect of text length on writing assessment,”Frontiers in Psychology, vol. 11, p. 562462, 2020. [Online]. Available: https://doi.org/10.3389/fpsyg.2020.562462

    arXiv 2020

Show all 43 references
  1. [9]

    Countering the influence of essay length in neural essay scoring,

    S. Jeon and M. Strube, “Countering the influence of essay length in neural essay scoring,” inProc. the Second Workshop on Simple and Efficient Natural Language Processing, N. S. Moosavi, I. Gurevych, A. Fan, T. Wolf, Y . Hou, A. Marasovi ´c, and S. Ravi, Eds. Virtual: Associat...

    2021

  2. [10]

    Beyond agreement: Diagnosing the rationale alignment of automated essay scoring methods based on linguistically-informed counterfactuals,

    Y . Wang, R. Hu, and Z. Zhao, “Beyond agreement: Diagnosing the rationale alignment of automated essay scoring methods based on linguistically-informed counterfactuals,” inFindings of the Association for Computational Linguistics: EMNLP 2024, Y . Al-Onaizan, M. Bansal, and Y ....

    2024

  3. [11]

    Neural automated essay scoring and coherence modeling for adversarially crafted input,

    Y . Farag, H. Yannakoudakis, and T. Briscoe, “Neural automated essay scoring and coherence modeling for adversarially crafted input,” inProc. the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume...

    2018

  4. [12]

    Enhancing marker scoring accuracy through ordinal confidence modelling in educational assessments,

    A. Chakravarty, M. Brenchley, T. Breakspear, I. Lewin, and Y . Huang, “Enhancing marker scoring accuracy through ordinal confidence modelling in educational assessments,” inProc. the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track...

    2025

  5. [13]

    Towards automatic assessment of spontaneous spoken english,

    Y . Wang, M. J. Gales, K. Knill, K. Kyriakopoulos, A. Malinin, R. C. van Dalen, and M. Rashid, “Towards automatic assessment of spontaneous spoken english,”Speech Communication, vol. 104, pp. 47–56, 2018. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/...

    2018

  6. [14]

    Automated speech scoring system under the lens: evaluating and interpreting the linguistic cues for language proficiency,

    P. Bamdev, M. S. Grover, Y . K. Singla, P. Vafaee, M. Hama, and R. R. Shah, “Automated speech scoring system under the lens: evaluating and interpreting the linguistic cues for language proficiency,”International Journal of Artificial Intelligence in Education, vol. 33, no. 1,...

    2023 doi

  7. [15]

    What do the measures of utterance fluency employed in automatic speech evaluation (ase) tell us about oral proficiency?

    Z. L. Handley and H. Wang, “What do the measures of utterance fluency employed in automatic speech evaluation (ase) tell us about oral proficiency?”Language Assessment Quarterly, vol. 21, no. 1, pp. 3–32, 2024. [Online]. Available: https://www.tandfonline.com/doi/pdf/ 10.1080/...

    2024

  8. [16]

    Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations,

    A. S. Ross, M. C. Hughes, and F. Doshi-Velez, “Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations,” inProc. the Twenty-Sixth International Joint Conference on Artificial Intelligence, 2017, pp. 2662–2670. [Online]. Available: https:/...

    2017 doi

  9. [17]

    Right for Better Reasons: Training Differentiable Models by Constraining their Influence Functions

    X. Shao, A. Skryagin, W. Stammer, P. Schramowski, and K. Kersting, “Right for Better Reasons: Training Differentiable Models by Constraining their Influence Functions.” inProc. the AAAI Conference on Artificial Intelligence, vol. 35, no. 11, 2021, pp. 9533–9540. [Online]. Avai...

    2021 doi

  10. [18]

    Axiomatic attribution for deep networks,

    M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” inInternational conference on machine learning. PMLR, 2017, pp. 3319–3328. [Online]. Available: https://proceedings.mlr.press/ v70/sundararajan17a.html

    2017

  11. [19]

    Automatic essay scoring systems are both overstable and oversensitive: Explaining why and proposing defenses,

    Y . Kumar, S. Parekh, S. Singh, J. J. Li, R. R. Shah, and C. Chen, “Automatic essay scoring systems are both overstable and oversensitive: Explaining why and proposing defenses,”Dialogue & Discourse, vol. 14, pp. 1–33, Apr. 2023. [Online]. Available: https://aclanthology.org/2...

    2023

  12. [20]

    The Speak & Improve Corpus 2025: an L2 English Speech Corpus for Language Assessment and Feedback,

    K. Knill, D. Nicholls, M. J. Gales, M. Qian, and P. Stroinski, “The Speak & Improve Corpus 2025: an L2 English Speech Corpus for Language Assessment and Feedback,” 2025. [Online]. Available: https://doi.org/10.17863/CAM.114333

    2025 doi

  13. [21]

    Introducing the speak & improve corpus 2025: an l2 english speech corpus for language assessment and feedback,

    ——, “Introducing the speak & improve corpus 2025: an l2 english speech corpus for language assessment and feedback,” in10th Workshop on Speech and Language Technology in Education (SLaTE), 2025, pp. 167–171. [Online]. Available: https://www.isca-archive.org/slate 2025/ knill25...

    2025

  14. [22]

    Universal Adversarial Attacks on Spoken Language Assessment Systems,

    V . Raina, M. J. Gales, and K. M. Knill, “Universal Adversarial Attacks on Spoken Language Assessment Systems,” inInterspeech 2020, 2020, pp. 3855–3859

    2020

  15. [23]

    Speak & Improve Challenge 2025,

    M. Qian, K. M. Knill, S. Bann `o, S. Tang, P. Karanasou, M. J. Gales, and D. Nicholls, “Speak & Improve Challenge 2025,” in10th Workshop on Speech and Language Technology in Education (SLaTE), 2025, pp. 41–45

    2025

  16. [24]

    Proficiency assessment of L2 spoken English using wav2vec 2.0,

    S. Bann `o and M. Matassoni, “Proficiency assessment of L2 spoken English using wav2vec 2.0,” in2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 1088–1095. [Online]. Available: https://ieeexplore.ieee.org/document/10023019

    2023

  17. [25]

    Assessment of L2 oral proficiency using speech large language models,

    R. Ma, M. Qian, S. Tang, S. Bann `o, K. Knill, and M. J. Gales, “Assessment of L2 oral proficiency using speech large language models,” inProc. Interspeech 2025, 2025, pp. 5078–5082. [Online]. Available: https://www.isca-archive.org/interspeech 2025/ma25b interspeech.pdf

    2025

  18. [26]

    Assessment of L2 oral proficiency using self-supervised speech representation learning,

    S. Bann `o, K. Knill, M. Matassoni, V . Raina, and M. J. Gales, “Assessment of L2 oral proficiency using self-supervised speech representation learning,” inProc. 9th Workshop on Speech and Language Technology in Education (SLaTE), 2023, pp. 126–130. [Online]. Available: https:...

    2023

  19. [27]

    Automated scoring of spontaneous speech using SpeechRater v1.0,

    X. Xi, D. Higgins, K. Zechner, and D. M. Williamson, “Automated scoring of spontaneous speech using SpeechRater v1.0,” Educational Testing Service, Princeton, NJ, Tech. Rep. ETS RR-08-62, 2008

    2008

  20. [28]

    Automatic scoring of non-native spontaneous speech in tests of spoken english,

    K. Zechner, D. Higgins, X. Xi, and D. M. Williamson, “Automatic scoring of non-native spontaneous speech in tests of spoken english,”Speech communication, vol. 51, no. 10, pp. 883–895, 2009. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0167639309000703

    2009

  21. [29]

    A three-stage approach to the automated scoring of spontaneous spoken responses,

    D. Higgins, X. Xi, K. Zechner, and D. Williamson, “A three-stage approach to the automated scoring of spontaneous spoken responses,” Computer Speech and Language, vol. 25, no. 2, pp. 282–306, 2011. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S088523...

    2011

  22. [30]

    Automated scoring of nonnative speech using the SpeechRater SM v. 5.0 engine,

    L. Chen, K. Zechner, S.-Y . Yoon, K. Evanini, X. Wang, A. Loukina, J. Tao, L. Davis, C. M. Lee, M. Ma, R. Mundkowsky, C. Lu, C. W. Leong, and B. Gyawali, “Automated scoring of nonnative speech using the SpeechRater SM v. 5.0 engine,”ETS Research Report Series, vol. 2018, no. 1...

    2018 doi

  23. [31]

    Assessing L2 English speaking using automated scoring technology: examining automarker reliability,

    J. Xu, E. Jones, V . Laxton, and E. Galaczi, “Assessing L2 English speaking using automated scoring technology: examining automarker reliability,”Assessment in Education: Principles, Policy & Practice, vol. 28, no. 4, pp. 411–436, 2021

    2021

  24. [32]

    Detecting aberrant responses in automated L2 spoken English assessment,

    S. Gao, M. J. Gales, and J. Xu, “Detecting aberrant responses in automated L2 spoken English assessment,” inExploring Artificial Intelligence in Applied Linguistics, C. A. Chapelle, G. H. Beckett, and J. Ranalli, Eds. Iowa State University Digital Press, 2024, pp. 96–117. [Onl...

    2024

  25. [33]

    Building an auto-marker for assessing spontaneous L2 English speech,

    K. Knill and M. J. Gales, “Building an auto-marker for assessing spontaneous L2 English speech,” inLanguage Test Validation in a Digital Age, ser. Studies in Language Testing, G. Yu and J. Xu, Eds. Cambridge University Press & Assessment, 2024, vol. 52, pp. 309–334

    2024

  26. [34]

    Longformer: The long-document transformer,

    I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,”arXiv preprint arXiv:2004.05150, 2020. [Online]. Available: https://arxiv.org/abs/2004.05150

    2004 arXiv

  27. [35]

    Automatic assessment of conversational speaking tests,

    S. W. McKnight, A. Civelekoglu, M. J. Gales, S. Bann `o, A. Liusie, and K. Knill, “Automatic assessment of conversational speaking tests,” inProc. SLaTE 2023, 2023, pp. 99–103. [Online]. Available: https://www.isca-archive.org/slate 2023/mcknight23 slate.html

    2023

  28. [36]

    Natural Language-based Assessment of L2 Oral Proficiency using LLMs,

    S. Bann `o, R. Ma, M. Qian, S. Tang, K. Knill, and M. J. Gales, “Natural Language-based Assessment of L2 Oral Proficiency using LLMs,”arXiv preprint arXiv:2507.10200, 2025. [Online]. Available: https://arxiv.org/abs/2507.10200

    2025 arXiv

  29. [37]

    Comi: Correct and mitigate shortcut learning behavior in deep neural networks,

    L. Zhao, Q. Liu, L. Yue, W. Chen, L. Chen, R. Sun, and C. Song, “Comi: Correct and mitigate shortcut learning behavior in deep neural networks,” inProc. the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, ser. SIGIR ’24. New York, ...

    2024

  30. [38]

    Filora: Focus-and-ignore lora for controllable feature reliance,

    H. Chung, C. Han, Y . Choi, S. Ji, J. Kim, E.-J. Holden, and K. Han, “Filora: Focus-and-ignore lora for controllable feature reliance,”arXiv preprint arXiv:2602.02060, 2026. [Online]. Available: https://arxiv.org/abs/2602.02060

    2026

  31. [39]

    Who Needs Decoders? Efficient Estimation of Sequence-Level Attributes with Proxies,

    Y . Fathullah, P. Radmard, A. Liusie, and M. J. Gales, “Who Needs Decoders? Efficient Estimation of Sequence-Level Attributes with Proxies,” inProc. the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Mar. 2024,...

    2024

  32. [40]

    Fast Differentiable Sorting and Ranking,

    M. Blondel, O. Teboul, Q. Berthet, and J. Djolonga, “Fast Differentiable Sorting and Ranking,” inProc. the 37th International Conference on Machine Learning, vol. 119. PMLR, 2020, pp. 950–959. [Online]. Available: https://proceedings.mlr.press/v119/blondel20a.html

    2020

  33. [41]

    Qwen2.5 technical report,

    Qwen Team, “Qwen2.5 technical report,” Alibaba Group, Tech. Rep.,

  34. [2007]

    Available: https://www.ets.org/research/policy research reports/publications/report/2007/hsmn.html

    [Online]. Available: https://www.ets.org/research/policy research reports/publications/report/2007/hsmn.html

    2007

  35. [2025]

    Available: https://arxiv.org/abs/2412.15115

    [Online]. Available: https://arxiv.org/abs/2412.15115

    arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.