Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Natural Language-based Assessment of L2 Oral Proficiency using LLMs

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM prompted with CEFR descriptors and fed speech transcripts grades L2 speaking with no training data, beating a fine-tuned BERT model.

desk verdict NLA is a worthwhile zero-shot text-only approach to L2 speaking assessment, but the paper's headline claim that it surpasses BERT is statistically unsupported as reported. read the letter →

arxiv 2507.10200 v1 pith:DGT3FV2U submitted 2025-07-14 eess.AS cs.AIcs.CL

Stefano Bannò , Rao Ma , Mengjie Qian , Siyuan Tang , Kate Knill , Mark Gales This is my paper · ORCID
classification eess.AScs.AIcs.CL
keywords automaticspeakingassessmentsecondlanguageproficiencyCEFRcan-dodescriptorszero-shotlargemodelsnaturallanguage-basedASRtranscriptionsinterpretablescoring
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a large language model, prompted in a zero-shot setting with the official CEFR analytic can-do descriptors meant for human examiners, can grade spontaneous second-language speech from text transcriptions nearly as well as systems trained for the task. On the public S&I Corpus, Qwen 2.5 72B with NLA reaches 0.761 Pearson correlation on the evaluation set, beating a fine-tuned BERT grader (0.727) and matching a speech LLM fine-tuned on read-aloud data (0.750), while a speech LLM fine-tuned on matched spontaneous data remains best (0.821). The result matters because it suggests that neither task-specific training data nor access to audio is a prerequisite for a useful automated speaking grader, and because the scores are anchored to interpretable descriptors rather than a black box. The authors position NLA as especially valuable when spontaneous training data are unavailable, and report that it generalises to conversational speech and written data.

What carries the argument

The central mechanism is the pairing of CEFR analytic can-do descriptors with the Fair Average scoring rule. For each of ten aspects, the descriptor set is presented as an option list (A–F or A–E) with CEFR level labels fixed but descriptor order randomised over three runs; the LLM's logit probabilities over the options are softmaxed into a distribution over levels, and the Fair Average $\sum_k p_k v_k$ with $v = 1,\dots,6$ turns that distribution into a continuous score. Averaging the ten analytic scores reconstructs the holistic score, mimicking the human rater's aggregation step. This construction replaces learned task-specific scoring heads with an instruction-following step plus an explicit expectation over ordinal levels, which is what allows a zero-shot text-only model to score speech.

What would settle it

Two observations would settle the claim: pairs of recordings whose ASR transcripts are identical but whose human scores differ by more than one CEFR level, since Q+NLA would necessarily assign the same score to both; and a correlation comparison of Q+NLA run on manual versus Whisper transcripts, where a large drop would show the ASR bottleneck is the binding constraint.

Watch

Extended reading notes

Core claim

The authors' central discovery is that the interpretative process human raters apply to CEFR can-do descriptors can be delegated to an open-source LLM working only from ASR transcriptions. For each of ten analytic aspects (grammatical accuracy, fluency, coherence and cohesion, and others), Qwen 2.5 72B is asked, under three randomised descriptor orders to control positional bias, to select the most appropriate CEFR level; the softmax over the model's logits is converted to a numeric score by the Fair Average, an expected value over the ordinal A1–C2 scale, and the analytic scores are averaged across aspects and exam parts to reconstruct a holistic score. On the S&I dev and eval sets this zero-shot text-only pipeline outperforms a BERT grader fine-tuned on the same transcripts, stays competitive with a speech LLM fine-tuned on read-aloud data (the mismatched condition), and trails only a speech LLM fine-tuned on matched spontaneous speech. Adding Ridge regression on the analytic scores (Q+NLA+RR) gives a small further gain (0.771 PCC on eval), and the regression coefficients vary sensibly across exam parts, indicating the analytic dimensions capture distinct aspects of proficiency.

Load-bearing premise

Whisper-small transcriptions preserve enough of what makes speech proficient—including partial fluency cues such as hesitations—that scoring from text alone is a valid proxy for scoring the spoken performance.

Editorial extensions

If this is right

  • Automated speaking assessment can be deployed for a new task type or language without collecting and annotating spontaneous speech, since the method is zero-shot and relies only on widely available CEFR descriptors.
  • A text-only grader now provides a stronger baseline than the standard fine-tuned BERT approach, which matters for any application where audio access is restricted or expensive.
  • In mismatched settings with only read-aloud training data, NLA performs on par with a speech LLM fine-tuned on that read-aloud data, making it a practical option when matched spontaneous data are scarce.
  • The analytic scores can serve as feedback: the regression coefficients show which competences dominate each exam part, e.g., fluency in short personal questions and sociolinguistic appropriateness in opinion tasks.
  • Because CEFR descriptors are language-independent, the same pipeline should transfer to non-English assessments, subject to the LLM's competence in the target language.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not spell out is to report the ten analytic scores to learners as a diagnostic profile rather than collapsing them into one holistic number; the pairwise differences among analytic scores (mostly significant under the Nemenyi tests) would give learners per-dimension feedback grounded in CEFR language.
  • A testable follow-up is to strip disfluencies from the transcripts before prompting: if the fluency analytic score drops sharply, the model is reading fluency from orthographic cues, which would clarify exactly what survives the ASR bottleneck.
  • The strong mismatched-condition result suggests an operational ordering the authors do not state explicitly: run zero-shot NLA first for any new language or test format, and invest in fine-tuned audio models only when the text-only ceiling is insufficient.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Natural Language-based Assessment (NLA), a zero-shot method that feeds ASR transcriptions of L2 English speech together with CEFR analytic can-do descriptors to the open-weight LLM Qwen 2.5 72B, extracts per-aspect scores via logit probabilities and a Fair Average transform, and averages them into a holistic score. On the public S&I Corpus, the authors compare this method with a fine-tuned BERT grader and with Qwen2Audio in matched and mismatched training conditions, reporting Pearson and Spearman correlations on dev and eval sets. The central claim is that NLA is competitive with fine-tuned speech LLMs and surpasses the BERT baseline, with added interpretability from the analytic scores and a Ridge regression analysis.

Significance. If the headline results are statistically robust, the paper's significance is real: it would demonstrate that an open-source LLM prompted with publicly available CEFR descriptors can produce useful holistic L2 speaking scores without task-specific training data, and it would provide a transparent, interpretable scoring rationale. The use of the public S&I Corpus, an open-weight model, and a disclosed separate calibration condition are strengths, as is the analytic-score analysis that connects parts of the exam to different proficiency aspects. However, the current lack of confidence intervals or significance tests on the key comparisons means the main 'surpasses BERT' claim is not yet established.

major comments (3)
  1. [Section 5, Tables 3 and 4] The ranking claims in the abstract and conclusions are not supported by any uncertainty quantification. On the eval set (N=300), the Q+NLA vs BERT differences are ΔPCC=0.034 and ΔSRC=0.027, and vs Q2Audio-LNG the differences are ΔPCC=0.011 and ΔSRC=−0.011; with N=300 the approximate standard error of a correlation of 0.75 is on the order of 0.04–0.05, so these gaps are within sampling noise. The sentence in Section 5 that Q+NLA 'significantly surpasses' Q2Audio on the dev set is also made without a significance test. Please add bootstrap confidence intervals for the correlations and/or tests for the difference between dependent correlations (e.g., Steiger's test), and soften claims that depend on those differences.
  2. [Section 4] The prompt template is not provided. The method's core is the instruction to the LLM with CEFR descriptors, yet the exact prompt text, the mapping of option labels (A-F, A-E) to CEFR levels, and the procedure for extracting the logit probabilities for those labels are unspecified. Without this information the experiment is not reproducible. Please include the full prompt(s) and extraction procedure in an appendix or supplementary material.
  3. [Section 5, Table 5 and following paragraph] The interpretation of the Ridge regression coefficients as the contribution of each analytic aspect to the holistic score is not well supported. As the paper acknowledges in Footnote 4, the analytic predictors are multicollinear, and Ridge coefficients are biased; the observation that vocabulary-related features 'consistently receive lower weights' may be an artifact of the shrinkage and collinearity rather than a true property of the assessment. Please temper these claims or support them with a more appropriate feature-importance analysis.
minor comments (6)
  1. [Section 4, Figure 1] The figure is referenced but not shown in the text; please ensure the final version includes the figure with a caption that explains the pipeline steps.
  2. [Section 4, Fair Average] The ordinal values v_k are given only as an example (A1=1...C2=6); please state explicitly how aspects without an A1 descriptor are coded (e.g., A2=2 ... C2=6).
  3. [Section 5] The statement that Q+NLA+RR improves PCC by 'roughly 1 point' and SRC by '0.5 points' is ambiguous; the values in Table 4 differ by 0.010 and 0.005, so please state whether these are percentage points or units of correlation.
  4. [Section 3, Table 1] The word error rate of the Whisper transcriptions on S&I is not reported; footnote 2 defers to [30], but a one-line WER figure would help the reader gauge the ASR quality.
  5. [Section 5, Table 2] The text says positional bias does not significantly affect behavior except for glr, ga, soc, but no threshold for 'significantly' is defined for the JSD values; please state a criterion.
  6. [Section 1] The word 'fortes' (in 'highlighting their fortes') is non-standard in English; 'strengths' would be more natural.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the target human scores are external, the headline Q+NLA result is zero-shot with no fitted parameters, and the only calibrated variant (Q+NLA+RR) is explicitly labelled and reported separately.

full rationale

The paper's central target is the external human holistic rating from the S&I Corpus, while the proposed Q+NLA score is produced by prompting Qwen 2.5 72B with public CEFR analytic descriptors over ASR transcripts, extracting softmax probabilities, applying the Fair Average formula, and averaging across analytic aspects and exam parts. None of these operations uses the human scores as an input or fits a parameter to them; the only fitted variant, Q+NLA+RR, trains Ridge regressions on the dev set and is reported separately in Table 4, so it does not masquerade as the zero-shot result. The self-citations ([27], [28], [30]) supply implementation details, baseline systems, or prior related work rather than load-bearing justifications: in particular, the Qwen2Audio baseline is an independently trained system, and no 'uniqueness theorem' or unverified premise is imported from the authors' previous papers. Concerns about the lack of confidence intervals and significance tests on the Table 4 differences are statistical-evidence issues, not circularity, because the reported correlations are not forced by construction. The derivation is therefore self-contained, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central empirical claim rests on external human labels and public descriptors; no new entities are posited. The unstated premises that would invalidate the result are transcription fidelity, logit calibration, reliability of human labels, and absence of test-corpus contamination. The Q+NLA+RR variant also depends on ridge coefficients fit to the dev set.

free parameters (2)
  • Ridge regression coefficients, one model per exam part (Q+NLA+RR) = See Table 5, e.g., Part 1 beta_flu=1.12, beta_pp=-0.59
    Trained on S&I dev predicted analytic scores to map analytic scores to the holistic target; these fitted coefficients determine the improved Q+NLA+RR eval result.
  • Ridge regularization alpha = 1
    Set by hand without a reported search; affects coefficient estimates and the Q+NLA+RR scores.
assumptions (5)
  • domain assumption CEFR can-do descriptors are interpretable by LLMs as scoring rubrics.
    The entire NLA pipeline presupposes Qwen can map transcript content to CEFR levels, as described in Section 2 and Figure 1.
  • domain assumption Whisper-small ASR transcriptions preserve enough proficiency-relevant information.
    Text is the only input; fluency is only partially inferable and phonological control is excluded, as stated in Section 4.
  • standard math Softmax over option-label logits yields meaningful level probabilities.
    Fair Average treats p_k as probabilities to compute an expected level, using the equation in Section 4.
  • domain assumption The S&I human scores are reliable ground truth.
    PCC and SRC are computed against these labels without a human-rater reliability baseline, as used in Section 3.
  • domain assumption Qwen 2.5 72B was not contaminated by the public S&I Corpus during pretraining.
    Zero-shot validity assumes Qwen did not memorize S&I examples; the paper does not discuss or test this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Natural Language-based Assessment of L2 Oral Proficiency using LLMs." pith.science (2026). https://pith.science/paper/DGT3FV2U

@misc{pith2026250710200,
  author       = {Pith},
  title        = {Pith review of: Natural Language-based Assessment of L2 Oral Proficiency using LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DGT3FV2U}},
  note         = {Machine review of arXiv:2507.10200}
}
read the original abstract

Natural language-based assessment (NLA) is an approach to second language assessment that uses instructions - expressed in the form of can-do descriptors - originally intended for human examiners, aiming to determine whether large language models (LLMs) can interpret and apply them in ways comparable to human assessment. In this work, we explore the use of such descriptors with an open-source LLM, Qwen 2.5 72B, to assess responses from the publicly available S&I Corpus in a zero-shot setting. Our results show that this approach - relying solely on textual information - achieves competitive performance: while it does not outperform state-of-the-art speech LLMs fine-tuned for the task, it surpasses a BERT-based model trained specifically for this purpose. NLA proves particularly effective in mismatched task settings, is generalisable to other data types and languages, and offers greater interpretability, as it is grounded in clearly explainable, widely applicable language descriptors.

Figures

Figures reproduced from arXiv: 2507.10200 by the authors.

Figure 1
Figure 1. Proposed NLA pipeline for speaking assessment. A related approach was explored in the context of L2 writ￾ing assessment by [26], where the authors examined the use of GPT-3.5 and GPT-4 with holistic CEFR-based descriptors. The models were prompted to select the most appropriate CEFR￾based descriptor on a scale from A1 to C2, and the generated responses were converted into numerical scores on a scale from 0 to 6. How… view at source ↗
Figure 2
Figure 2. Heatmap of pairwise Nemenyi test between predicted analytic and ground truth scores (overall) on S&I dev set. 6. Conclusions and future work In this paper, we presented our initial experiments on L2 speaking assessment using LLMs in combination with natu￾ral language-based analytic descriptors. Our results demon￾strate that this approach, relying exclusively on textual informa￾tion in a zero-shot setting, achieves c… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Controlling Implicit Shortcut Reliance in L2 Spoken English Auto-markers

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A rank-correlation penalty on the training loss of text- and audio-based L2 speaking auto-markers reduces their over-reliance on word count and speaking time to human-rater levels while keeping overall accuracy competitive.

Reference graph

Works this paper leans on

42 extracted references · 36 canonical work pages · cited by 1 Pith paper

  1. [1]

    Introduction The automatic evaluation of language proficiency is gaining in- creasing attention and significance in the domain of second lan- guage (L2) assessment, driven by the steady global rise in the number of L2 learners of English [1]. Language proficiency exams typically include dedicated sections for assessing speak- ing, listening, writing, and ...

    work page Pith review arXiv 2025

  2. [2]

    It aims to assess whether LLMs can interpret and apply these descriptors in a manner comparable to human raters

    Natural language-based assessment Natural language-based assessment (NLA) is an approach to L2 assessment that uses instructions, formulated as scoring criteria or can-do descriptors, originally designed for human examin- ers. It aims to assess whether LLMs can interpret and apply these descriptors in a manner comparable to human raters. We aim to have th...

    work page

  3. [3]

    Both the LNG and S&I language tests comprise five sections, each designed to assess a different as- pect of a candidate’s language proficiency

    Data This study utilises two datasets: a private dataset, Linguaskill (LNG) [29], and a public dataset, the Speak & Improve Cor- pus 2025 (S&I) [21]. Both the LNG and S&I language tests comprise five sections, each designed to assess a different as- pect of a candidate’s language proficiency. In Part 1, candidates respond to eight questions, only six of w...

    work page 2025

  4. [4]

    tised) [23], in a zero-shot setting

    Experimental setup Qwen 2.5 72B + NLA (Q+NLA): Our proposed NLA ap- proach utilises an open-source LLM, Qwen 2.5 72B (4-bit quan- 1Our previous work [28] also evaluated Qwen2Audio in a zero-shot setting, though with limited success. tised) [23], in a zero-shot setting. As shown in Figure 1, ASR transcriptions of learner responses generated with Ope- nAI W...

    work page

  5. [5]

    Train Type PCC SRC Dev Eval Dev Eval BERT S&I Spont

    Experimental results and analysis Table 3: Results in terms of PCC and SRC on S&I dev and eval. Train Type PCC SRC Dev Eval Dev Eval BERT S&I Spont. 0.753 0.727 0.764 0.728 Q+NLA - - 0.806 0.761 0.812 0.755 Q2Audio LNG Read 0.740 0.750 0.733 0.766 Q2Audio S&I Spont. 0.833 0.821 0.837 0.824 Table 3 reports the overall results considering all the grad- ing ...

    work page

  6. [6]

    Our results demon- strate that this approach, relying exclusively on textual informa- tion in a zero-shot setting, achieves competitive performance

    Conclusions and future work In this paper, we presented our initial experiments on L2 speaking assessment using LLMs in combination with natu- ral language-based analytic descriptors. Our results demon- strate that this approach, relying exclusively on textual informa- tion in a zero-shot setting, achieves competitive performance. While it does not outper...

    work page

  7. [7]

    Howson, The English effect

    P. Howson, The English effect. London: British Council, 2013

    work page 2013

  8. [8]

    Contrasting Automated and Human Scoring of Es- says,

    M. Zhang, “Contrasting Automated and Human Scoring of Es- says,” R&D Connections, no. 21, pp. 1–11, 2013

    work page 2013

Show all 42 references
  1. [9]

    Fully automated speaking assessment: changes to pro- ficiency testing and the role of pronunciation,

    T. Isaacs, “Fully automated speaking assessment: changes to pro- ficiency testing and the role of pronunciation,” in The Routledge handbook of contemporary English pronunciation, O. Kang, R. I. Thomson, and J. Murphy, Eds. London; New York: Routledge, 2017, pp. 570–582

    2017

  2. [10]

    Limitations of the Common European Framework for developing comparable examinations and tests,

    C. J. Weir, “Limitations of the Common European Framework for developing comparable examinations and tests,” Language Test- ing, vol. 3, no. 22, pp. 281–300, 2005

    2005

  3. [11]

    Strasbourg: Council of Europe, 2020

    Council of Europe, Common European Framework of Reference for Languages: Learning, Teaching, Assessment - Companion volume. Strasbourg: Council of Europe, 2020

    2020

  4. [12]

    Facets of Speaking Proficiency,

    N. H. De Jong, M. P. Steinel, A. F. Florijn, R. Schoonen, and J. H. Hulstijn, “Facets of Speaking Proficiency,”Studies in Second Language Acquisition, vol. 34, no. 1, p. 5–34, 2012

    2012

  5. [13]

    Automat- ically assessing the oral proficiency of proficient L2 speakers

    P. M ¨uller, F. De Wet, C. Van Der Walt, and T. Niesler, “Automat- ically assessing the oral proficiency of proficient L2 speakers.” in Proc. Workshop on Speech and Language Technology for Educa- tion (SLaTE), 2009, pp. 29–32

    2009

  6. [14]

    Applications of text analysis tools for spoken response grading

    S. A. Crossley and D. McNamara, “Applications of text analysis tools for spoken response grading.” Language Learning & Tech- nology, vol. 17, no. 2, pp. 171–192, 2013

    2013

  7. [15]

    Towards automatic as- sessment of spontaneous spoken English,

    Y . Wang, M. J. F. Gales, K. M. Knill, K. Kyriakopoulos, A. Ma- linin, R. C. van Dalen, and M. Rashid, “Towards automatic as- sessment of spontaneous spoken English,” Speech Communica- tion, vol. 104, pp. 47–56, 2018

    2018

  8. [16]

    Dolphin: a spoken language proficiency assessment system for elementary education,

    Z. Liu, G. Xu, T. Liu, W. Fu, Y . Qi, W. Ding, Y . Song, C. Guo, C. Kong, S. Yanget al., “Dolphin: a spoken language proficiency assessment system for elementary education,” in Proc. The Web Conference 2020, 2020, pp. 2641–2647

    2020

  9. [17]

    In- vestigating the Effect of Auxiliary Objectives for the Automated Grading of Learner English Speech Transcriptions,

    H. Craighead, A. Caines, P. Buttery, and H. Yannakoudakis, “In- vestigating the Effect of Auxiliary Objectives for the Automated Grading of Learner English Speech Transcriptions,” in Proceed- ings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, 2...

    2020

  10. [18]

    Universal Adversarial Attacks on Spoken Language Assessment Systems,

    V . Raina, M. J. Gales, and K. M. Knill, “Universal Adversarial Attacks on Spoken Language Assessment Systems,” in Proc. In- terspeech 2020, 2020, pp. 3855–3859

    2020

  11. [19]

    Automated Scoring of Spontaneous Speech from Young Learners of English Using Transformers,

    X. Wang, K. Evanini, Y . Qian, and M. Mulholland, “Automated Scoring of Spontaneous Speech from Young Learners of English Using Transformers,” in2021 IEEE Spoken Language Technology Workshop (SLT), 2021, pp. 705–712

    2021

  12. [20]

    Proficiency assessment of L2 spo- ken English using wav2vec 2.0,

    S. Bann `o and M. Matassoni, “Proficiency assessment of L2 spo- ken English using wav2vec 2.0,” in 2022 IEEE Spoken Language Technology Workshop (SLT), 2023, pp. 1088–1095

    2022

  13. [21]

    Assessment of L2 Oral Proficiency Using Self-Supervised Speech Representation Learning,

    S. Bann `o, K. M. Knill, M. Matassoni, V . Raina, and M. Gales, “Assessment of L2 Oral Proficiency Using Self-Supervised Speech Representation Learning,” in Proc. 9th Workshop on Speech and Language Technology in Education (SLaTE) , 2023, pp. 126–130

    2023

  14. [22]

    Automatic Assessment of Conversational Speaking Tests,

    S. W. McKnight, A. Civelekoglu, M. Gales, S. Bann `o, A. Liusie, and K. M. Knill, “Automatic Assessment of Conversational Speaking Tests,” in9th Workshop on Speech and Language Tech- nology in Education (SLaTE), 2023, pp. 99–103

    2023

  15. [23]

    Rating nonnative writing: The trouble with holistic scoring,

    L. Hamp-Lyons, “Rating nonnative writing: The trouble with holistic scoring,” TESOL Quarterly, vol. 29, no. 4, pp. 759–762,

  16. [24]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi...

    2019

  17. [25]

    Toward Automated Multi-trait Scoring of Essays: Investigating Links among Holistic, Analytic, and Text Feature Scores,

    Y .-W. Lee, C. Gentile, and R. Kantor, “Toward Automated Multi-trait Scoring of Essays: Investigating Links among Holistic, Analytic, and Text Feature Scores,” Applied Linguistics, vol. 31, no. 3, pp. 391–417, 11 2009. [Online]. Available: https://doi.org/10.1093/applin/amp040

    2009 doi

  18. [26]

    Weight-Based Classification of Raters and Rater Cognition in an EFL Speaking Test,

    H. Cai, “Weight-Based Classification of Raters and Rater Cognition in an EFL Speaking Test,” Language Assessment Quarterly, vol. 12, no. 3, pp. 262–282, 2015. [Online]. Available: https://doi.org/10.1080/15434303.2015.1053134

    2015

  19. [27]

    Douglas and J

    D. Douglas and J. Smith, Theoretical underpinnings of the Test of Spoken English revision project. Educational Testing Service Princeton, NJ, 1997

    1997

  20. [28]

    Speak & Improve Corpus 2025: an L2 English Speech Corpus for Language Assessment and Feedback,

    K. Knill, D. Nicholls, M. J. F. Gales, M. Qian, and P. Stroinski, “Speak & Improve Corpus 2025: an L2 English Speech Corpus for Language Assessment and Feedback,” 2024. [Online]. Available: https://arxiv.org/abs/2412.11986

    2025 arXiv

  21. [29]

    Emergent Abilities of Large Language Models,

    J. Wei, Y . Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus, “Emergent Abilities of Large Language Models,”Transactions on Machine Learning Research, 2022, surv...

    2022

  22. [30]

    Qwen2.5 Technical Report,

    A. Yang et al. , “Qwen2.5 Technical Report,” arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  23. [31]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

    2023

  24. [32]

    Qwen2-Audio Technical Report,

    Y . Chu et al., “Qwen2-Audio Technical Report,” 2024. [Online]. Available: https://arxiv.org/abs/2407.10759

    2024 arXiv

  25. [33]

    Rating short L2 essays on the CEFR scale with GPT-4,

    K. P. Yancey, G. Laflair, A. Verardi, and J. Burstein, “Rating short L2 essays on the CEFR scale with GPT-4,” in Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023) , E. Kochmar, J. Burstein, A. Horbach, R. Laarmann-Quante...

    2023

  26. [34]

    Distribution-free multiple comparisons,

    P. B. Nemenyi, “Distribution-free multiple comparisons,” Ph.D. dissertation, Princeton University, 1963

    1963

  27. [35]

    Can GPT-4 do L2 analytic assessment?

    S. Bann `o, H. K. Vydana, K. Knill, and M. Gales, “Can GPT-4 do L2 analytic assessment?” in Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024) . Mexico City, Mexico: Association for Computational Linguistics, Jun. 2024, p...

    2024

  28. [36]

    As- sessment of L2 Oral Proficiency Using Speech Large Language Models,

    R. Ma, M. Qian, S. Tang, S. Banno, K. Knill, and M. Gales, “As- sessment of L2 Oral Proficiency Using Speech Large Language Models,” to appear in Proc. Interspeech 2025

    2025

  29. [37]

    Ludlow, Official Quick Guide to Linguaskill

    K. Ludlow, Official Quick Guide to Linguaskill . Cambridge: Cambridge University Press, 2020

    2020

  30. [38]

    Speak & Improve Challenge 2025: Tasks and Baseline Systems,

    M. Qian, K. Knill, S. Banno, S. Tang, P. Karanasou, M. J. F. Gales, and D. Nicholls, “Speak & Improve Challenge 2025: Tasks and Baseline Systems,” 2024. [Online]. Available: https://arxiv.org/abs/2412.11985

    2025 arXiv

  31. [40]

    LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models,

    A. Liusie, P. Manakul, and M. Gales, “LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long ...

    2024

  32. [41]

    The use of ranks to avoid the assumption of nor- mality implicit in the analysis of variance,

    M. Friedman, “The use of ranks to avoid the assumption of nor- mality implicit in the analysis of variance,” Journal of the Ameri- can Statistical Association, vol. 32, no. 200, pp. 675–701, 1937

    1937

  33. [584]

    Available: https://aclanthology.org/2023.bea-1.49

    [Online]. Available: https://aclanthology.org/2023.bea-1.49

    2023

  34. [1995]

    Available: http://www.jstor.org/stable/3588173

    [Online]. Available: http://www.jstor.org/stable/3588173

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.