Assessment of L2 Oral Proficiency using Speech Large Language Models

Kate M. Knill; Mark J.F. Gales; Mengjie Qian; Rao Ma; Siyuan Tang; Stefano Bann\`o

REVIEW 3 major objections 5 minor 38 references

Assessment of L2 Oral Proficiency using Speech Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A speech large language model trained with a fair-average loss becomes the best automatic grader of L2 English oral proficiency, reaching a Pearson correlation of 0.954 on a general English test set and 0.938 on a business English test set.

desk verdict This is the first solid application of speech LLMs to holistic L2 oral scoring, with consistent gains, but the missing candidate-level split description leaves the headline SOTA claim unproven. read the letter →

arxiv 2505.21148 v1 pith:4SHJJ6LH submitted 2025-05-27 cs.CL cs.SDeess.AS

Rao Ma , Mengjie Qian , Siyuan Tang , Stefano Bannò , Kate M. Knill , Mark J.F. Gales This is my paper

classification cs.CLcs.SDeess.AS

keywords spokenlanguageassessmentspeechlargemodelsL2oralproficiencyfairaveragelossQwen2-AudioLoRAadaptationcross-taskgeneralizationCEFRscoring

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish that speech large language models, which take audio as input and generate text, can grade the holistic speaking proficiency of second-language English learners as accurately as or better than established automatic graders. It trains the Qwen2Audio model with three targets — classification, classification with a fair-average loss, and regression — and evaluates on the Linguaskill and Speak & Improve corpora. The best system, trained with the fair-average loss and decoded with soft scoring, reaches Pearson correlations of 0.954 on general English and 0.938 on business English, outperforming a BERT-based cascade and a wav2vec2 end-to-end grader. If this result is right, speech LLMs are the new state of the art for this task and remove the need for a separate automatic speech recognition step.

What carries the argument

The central object is a grader built from Qwen2Audio, a pre-trained audio-language model, adapted with LoRA adapters of rank 16 on both encoder and decoder, adding 10 million parameters. The load-bearing mechanism is the fair-average loss: the model's logits over six letter grades are passed through a softmax, each grade's probability is weighted by its numeric score (1 to 6), and the weighted average is trained against the human reference score with mean squared error. This preserves intermediate half-point scores that ordinary classification rounding would discard, and the same weighted average is used at decoding time as the soft decoding strategy.

What would settle it

Retrain the Qwen2Audio-fa grader on a speaker-disjoint split of Linguaskill and Speak & Improve, ensuring no candidate appears in both training and test sets, and check whether the PCC on LinGen remains near 0.954 or drops substantially.

Watch

Extended reading notes

Core claim

The paper claims that a speech LLM can serve directly as a holistic L2 oral proficiency grader, and that its predictions align with human scores better than prior systems do. Using Qwen2Audio-7B-Instruct with LoRA adapters, the authors frame grading as a classification task over letter grades mapped to numeric scores, and train with a fair-average loss that computes the softmax-weighted score and minimizes squared error against the reference. On submission-level evaluation across all five test parts of Linguaskill, this grader achieves RMSE of 0.323 and PCC of 0.954 on LinGen and RMSE of 0.356 and PCC of 0.938 on LinBus, beating the wav2vec2 attention-pooling grader and the BERT cascade. The paper further reports that the trained grader generalises across test parts and across corpora, including a low-resource setting where training uses only the small Speak & Improve training set.

Load-bearing premise

The results depend on the training and test splits being speaker-disjoint; the paper splits by test submission and does not state whether any candidate contributes to both sides, which could allow the model to memorise voices rather than learn proficiency.

Editorial extensions

If this is right

Automatic L2 speaking assessment can be built directly from audio, without an ASR module, avoiding the propagation of transcription errors.
The Qwen2Audio-fa grader becomes the strongest single system on Linguaskill-style data, with PCC gains over the BERT cascade of about 0.01 on both LinGen and LinBus.
A grader trained only on read-aloud sentences transfers to spontaneous and short-answer parts of the test, so read-aloud data are a viable and easier-to-collect starting point.
In the low-resource S&I setting, the speech LLM reaches PCC 0.833 versus 0.753 for the BERT baseline, showing an advantage when training data are limited.
Models trained on one corpus transfer to the other in both directions, indicating robustness to test format and topic domain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next step is to apply the same fair-average training to analytic scoring dimensions, since the model already encodes acoustic and linguistic cues needed for separate pronunciation, fluency, grammar, and vocabulary scores.
If the speaker-leakage concern is addressed, the zero-shot correlations near 0.64 suggest an emergent proficiency-judging ability in speech LLMs; probing prompt variations or adding few-shot demonstrations could make zero-shot grading practical for new tests.
The fair-average loss is a generic recipe for tasks with half-step or ordinal human labels, so it could transfer to other rating domains such as essay scoring or medical image grading.
A direct validation would compare the model's agreement with human inter-rater agreement on the same submissions; if the model reaches human-level consistency, it could serve as a second rater in operational assessment.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, the useful headline: this is the first paper to apply a speech LLM to holistic L2 oral proficiency scoring, and the results are consistent enough that it deserves a serious referee. The authors show Qwen2Audio with LoRA tuning outperforms the BERT cascade and wav2vec2 baselines on both general and business Linguaskill sets, and the cross-part and cross-task generalization results are genuinely interesting.

What is actually new: the application itself, plus a small training trick they call fair average loss, which is basically a soft-label regression on class probabilities. That is not a big technical leap, but it is sensible and works better than hard classification or a linear regression head. The zero-shot result (PCC ~0.64) and the read-aloud-only grader transferring to other parts (0.911 overall PCC) are the kind of empirical findings that could be useful beyond this specific dataset.

The methodology is generally clean, and the baselines are reasonable. The S&I results on a public corpus help, and the zero-shot numbers are independent of training, so they should be taken as reliable evidence of speech LLM capability.

The soft spots are real but not fatal. The biggest one, which the stress-test note correctly flags, is that the paper never states whether the Linguaskill train/dev/test splits are by candidate. Linguaskill has five parts per candidate; if the split is at submission level, the same speaker's responses appear in both train and test. With a 7B model, that could inflate the matched and cross-part results. The authors need to clarify this. It is a missing detail, not proof of leakage, but it blocks the SOTA claim.

Second, all results come from a single training run with no error bars or significance tests. The headline gain over BERT on LinGen is 0.012 PCC, which could easily be within run-to-run noise. Third, for S&I, the dev set is used both for calibration and for final reporting, which is a minor protocol weakness. The "all previous competitive baselines" phrasing is a bit strong, but the baselines they compare against are the standard ones from the literature, so I would call it a wording issue, not a fatal one.

Bottom line: I largely agree with the reader's conditional verdict. The paper is a solid, competent empirical study that advances the application of speech LLMs to a new assessment task. It deserves peer review, with the split protocol and significance testing as the key conditions.

Referee Report

3 major / 5 minor

Summary. The paper investigates whether speech large language models can serve as holistic L2 oral proficiency graders. Using Qwen2-Audio-7B-Instruct, it compares three training schemes—classification with cross-entropy loss, classification with a proposed 'fair average' loss, and regression—combined with hard and soft decoding strategies. Experiments are conducted on the private Linguaskill corpus and the public Speak & Improve Corpus 2025. The authors report that the LoRA-adapted Qwen2Audio grader with fair-average loss and soft decoding outperforms BERT-cascade and wav2vec2 end-to-end baselines on Linguaskill LinGen and LinBus (PCC 0.954 and 0.938, respectively), and that it generalizes across test parts and across tasks to S&I. The paper also reports zero-shot results showing moderate correlation without task-specific training.

Significance. If the results hold, this is a meaningful advance: a single multimodal LLM grader that consumes audio directly can outperform cascaded ASR-plus-BERT systems and self-supervised speech encoders on holistic L2 scoring, with promising generalization from read-aloud training to spontaneous speech. The proposed fair-average loss is a simple and sensible way to handle half-point human scores, and the paper includes a useful zero-shot comparison. However, the strength of the empirical claim depends on two currently unverified assumptions: that train/test splits are disjoint at the candidate level, and that the reported improvements over strong baselines are statistically reliable given the small PCC differences and the absence of error bars. Both issues are verifiable and should be addressed before the state-of-the-art claim is accepted.

major comments (3)

[Section 3.1, Table 1] The paper reports train/dev/test splits by number of test submissions and hours, but never states whether the splits are disjoint at the level of individual candidates. Linguaskill has five scored parts per candidate, so if a candidate's responses appear in both training and test partitions, the Qwen2Audio model—with its large capacity and LoRA adaptation—could memorize speaker-identifying acoustic and proficiency cues. This would inflate the matched results in Table 3 and would be especially damaging to the cross-part generalization claim in Table 4, where a grader trained only on Part 2 read-aloud responses reaches an overall PCC of 0.911 on LinGen; it could also inflate the cross-task numbers in Table 5. Please report the number of unique candidates per split, confirm that no candidate appears in more than one split, and if overlap exists, re-run the evaluation on candidate-disjoint splits.
[Section 3.3, Table 3] The headline improvements over the BERT baseline are small: on LinGen the PCC difference is 0.954 vs. 0.942 (Δ = 0.012) and on LinBus it is 0.938 vs. 0.928 (Δ = 0.010). All numbers come from a single training run, with no multiple seeds, error bars, confidence intervals, or significance tests. Given that the paper's central claim is that speech LLMs 'outperform all previous competitive baselines,' the authors should report variance across at least a few training runs and provide a significance test for the metric differences (for example, a bootstrap or permutation test over test responses). Without this, the superiority over BERT on LinGen/LinBus is not statistically established.
[Section 3.1, Table 5] For the S&I corpus, the development set is used to extract linear calibration factors, and the same S&I dev set also serves as the test set in Table 5. Reporting calibrated metrics on the calibration set can inflate performance; although linear calibration primarily affects RMSE, it can still shift PCC/SRC slightly, and using the same set for calibration and final reporting is not standard practice. Please clarify whether a separate S&I test set exists, and if not, report uncalibrated metrics on a held-out portion of the dev set or use cross-validation so that the numbers in the S&I column are not optimistically biased.

minor comments (5)

[Section 2.1.1, Eq. (1)] The cross-entropy loss as written is ∑_c y_{i,c} log(ŵ_{i,c}), but y_{i,c} is the predicted probability and ŵ_{i,c} is the one-hot reference label. The correct form is −∑_c ŵ_{i,c} log y_{i,c}, with a minus sign. Please fix the notation.
[Section 3.4] The text says 'Table 6 evaluates the cross-part performance of the trained graders,' but the table is labeled Table 4. Please correct the cross-reference.
[Figure 1 caption] The caption says 'Audio2Qwen graders,' but the model is referred to as Qwen2Audio throughout the paper. Please align the naming.
[Table 2] The regression row is labeled 'reg-' with a trailing hyphen; this should be 'Reg' for consistency with the other rows.
[Section 2.3] The prompt omits the actual text for Options B through E, showing ellipses. For reproducibility, please include the full prompt in an appendix or supplementary material.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the grader is trained on labeled data and evaluated on held-out test sets; self-citations are comparison baselines only.

full rationale

The paper's derivation is a standard supervised learning pipeline: Qwen2Audio is adapted with LoRA using CE, FA, and Reg losses (Eqs. 1-3) on Linguaskill and S&I training data, then evaluated on LinGen, LinBus, and the S&I dev set. The target scores are human reference labels, and no predicted quantity is fed back into the training objective. The FA loss is not defined in terms of the evaluation metric; it is an MSE between the reference score and the softmax-weighted class scores, and soft decoding computes the same expectation, so the fair-average naming is consistent but the held-out correlation is not forced by construction. Calibration on the dev set is an affine transform, which can reduce RMSE but cannot create PCC or SRC gains, so the correlation claims are not a fitted-input-called-prediction artifact. The cited prior systems [18, 19, 31] are external baseline comparators rather than load-bearing assumptions; no uniqueness theorem or ansatz is imported from same-author work. The unresolved candidate-split question is a potential data-leakage concern, not circular reasoning. No specific circular step can be quoted, so the appropriate score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the pre-trained capabilities of Qwen2-Audio, the reliability of human reference scores, the appropriateness of the CEFR prompt, and the assumption that the data splits isolate speakers. No new physical or abstract entities are postulated; the fair average loss is a training objective, not an entity.

free parameters (6)

Linear calibration scale and bias = not reported (fitted on dev set)
Fitted on the development set to map model outputs to reference scores; affects reported RMSE (less so PCC/SRC, which are rank or affine-invariant).
LoRA rank = 16
Rank of LoRA adapters inserted into encoder and decoder layers (Section 3.2); controls model capacity for adaptation.
Training epochs = 2
Number of epochs on the training set; selected with dev-set monitoring.
Learning rate = 1e-4
Peak learning rate with cosine scheduling and weight decay (Section 3.2).
Batch size = 64 (Linguaskill), 8 (S&I)
Per-configuration batch sizes used for training (Section 3.2).
Audio chunking for parts 3/4 = two 30-second chunks
Since Qwen2Audio accepts only <30s audio, long responses are split into two chunks and scores are averaged (Section 3.2).

assumptions (5)

domain assumption Qwen2-Audio's pre-trained audio understanding transfers to L2 proficiency assessment
The method assumes that representations learned for general audio tasks carry enough signal about pronunciation, fluency, and content for holistic grading; stated in the abstract and Section 3.4.
domain assumption Human reference scores on the 1-6 scale are reliable ground truth
Training and evaluation treat annotator scores as the target; inconsistencies in human labelling are acknowledged in the introduction but not modeled.
domain assumption CEFR descriptors used in the prompt are suitable grade anchors
The prompt lists options based on CEFR descriptors (Section 2.3); the mapping of descriptors to numeric levels is taken as given.
domain assumption Linguaskill train/dev/test splits are by candidate with no speaker overlap
The data setup (Section 3.1, Table 1) does not state the split criterion; the generalization results assume no same-candidate leakage across splits.
domain assumption Development-set calibration generalizes to test sets
The paper reports calibrated results using linear factors from the dev set (Section 3.1), assuming the dev distribution matches the test distribution.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Assessment of L2 Oral Proficiency using Speech Large Language Models." pith.science (2026). https://pith.science/paper/4SHJJ6LH

@misc{pith2026250521148,
  author       = {Pith},
  title        = {Pith review of: Assessment of L2 Oral Proficiency using Speech Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4SHJJ6LH}},
  note         = {Machine review of arXiv:2505.21148}
}

read the original abstract

The growing population of L2 English speakers has increased the demand for developing automatic graders for spoken language assessment (SLA). Historically, statistical models, text encoders, and self-supervised speech models have been utilised for this task. However, cascaded systems suffer from the loss of information, while E2E graders also have limitations. With the recent advancements of multi-modal large language models (LLMs), we aim to explore their potential as L2 oral proficiency graders and overcome these issues. In this work, we compare various training strategies using regression and classification targets. Our results show that speech LLMs outperform all previous competitive baselines, achieving superior performance on two datasets. Furthermore, the trained grader demonstrates strong generalisation capabilities in the cross-part or cross-task evaluation, facilitated by the audio understanding knowledge acquired during LLM pre-training.

Figures

Figures reproduced from arXiv: 2505.21148 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 30 canonical work pages

[1]

Assessment of L2 Oral Proficiency using Speech Large Language Models

Introduction With the growing number of second language (L2) English learners worldwide, the demand for building automated systems capable of assessing their spoken language proficiency has been steadily increasing. These systems play a crucial role in scoring the candidates’ responses and providing feedback in language learning. Compared to human annotat...

work page Pith review arXiv 2025
[2]

A”to“F”) based on the candidate’s speech responses. Here, a function scoreis used to map each text-based grade into its numeric class (e.g. label“A

Methodology For the spoken language assessment datasets studied in this pa- per, reference scores are annotated by human graders on a scale from 1 to 6 for each audio response, with specific instructions provided for each class level. Half-point scores (e.g., 3.5) are allowed for intermediate ratings. Our goal is to develop auto graders that produce score...

work page
[3]

Experiments 3.1. Data Setup This paper uses two datasets for building and evaluating spo- ken language assessment systems: a private dataset, Lingual- skill [29]; and a public corpus, Speak & Improve Corpus 2025 (S&I) [30]. Each test from Linguaskill and S&I contains five parts focusing on different aspects of the candidate’s language proficiency: For Par...

work page 2025
[4]

Vari- ous training and decoding schemes are compared on Linguaskill and S&I datasets

Conclusions In this paper, we examine the effectiveness of building spo- ken language assessment systems using speech LLMs. Vari- ous training and decoding schemes are compared on Linguaskill and S&I datasets. The zero-shot experiments show the potential of leveraging speech LLMs for L2 oral scoring. With further adaptation, the Qwen2Audio grader trained ...

work page
[5]

Automatic evaluation and training in English pronuncia- tion,

J. Bernstein, M. Cohen, H. Murveit, D. Rtischev, and M. Wein- traub, “Automatic evaluation and training in English pronuncia- tion,” inFirst International Conference on Spoken Language Pro- cessing, 1990

work page 1990
[6]

Automatic evaluation of Dutch pronunciation by using speech recognition technology,

C. Cucchiarini, H. Strik, and L. Boves, “Automatic evaluation of Dutch pronunciation by using speech recognition technology,” in 1997 IEEE Workshop on Automatic Speech Recognition and Un- derstanding Proceedings, 1997, pp. 622–629

work page 1997
[7]

The SRI EduSpeak™ system: Recog- nition and pronunciation scoring for language learning,

H. Franco, V . Abrash, K. Precoda, H. Bratt, R. Rao, J. Butzberger, R. Rossier, and F. Cesari, “The SRI EduSpeak™ system: Recog- nition and pronunciation scoring for language learning,” inPro- ceedings of InSTILL, 2000, pp. 123–128

work page 2000
[8]

Speechrater: A construct- driven approach to scoring spontaneous non-native speech,

K. Zechner, D. Higgins, and X. Xi, “Speechrater: A construct- driven approach to scoring spontaneous non-native speech,” in Proc. SLaTE, 2007, pp. 128–131

work page 2007

Show all 38 references

[9]

The use of DBN-HMMs for mispronunciation detection and diagnosis in L2 English to sup- port computer-aided pronunciation training,

X. Qian, H. Meng, and F. K. Soong, “The use of DBN-HMMs for mispronunciation detection and diagnosis in L2 English to sup- port computer-aided pronunciation training,” inProc. Interspeech, 2012

2012
[10]

Improvements to an automated con- tent scoring system for Spoken CALL responses: the ETS sub- mission to the Second Spoken CALL Shared Task

K. Evanini, M. Mulholland, R. Ubale, Y . Qian, R. A. Pugh, V . Ra- manarayanan, and A. Cahill, “Improvements to an automated con- tent scoring system for Spoken CALL responses: the ETS sub- mission to the Second Spoken CALL Shared Task.” inProc. In- terspeech, 2018, pp. 2379–2383

2018
[11]

The University of Birm- ingham 2019 Spoken CALL Shared Task Systems: Exploring the importance of word order in text processing,

M. Qian, P. Jan ˇcoviˇc, and M. Russell, “The University of Birm- ingham 2019 Spoken CALL Shared Task Systems: Exploring the importance of word order in text processing,” in8th ISCA Work- shop on Speech and Language Technology in Education (SLaTE 2019), 2019, pp. 11–15

2019
[12]

BERT: Pre- training of deep bidirectional transformers for language under- standing,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language under- standing,” inProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Techno...

2019
[13]

Universal adversarial attacks on spoken language assessment systems,

V . Raina, M. J. F. Gales, and K. M. Knill, “Universal adversarial attacks on spoken language assessment systems,” inProc. Inter- speech, 2020, pp. 3855–3859

2020
[14]

Automated scoring of spontaneous speech from young learners of English using transformers,

X. Wang, K. Evanini, Y . Qian, and M. Mulholland, “Automated scoring of spontaneous speech from young learners of English using transformers,” in2021 IEEE Spoken Language Technology Workshop (SLT), 2021, pp. 705–712

2021
[15]

wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,

A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,” inProceedings of the 34th Conference on Neural Informa- tion Processing Systems (NeurIPS 2020), 2020, pp. 1–12

2020
[16]

HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,

W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhut- dinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 29, pp. 3451–3460, 2021

2021
[17]

A Study on Fine- Tuning wav2vec2.0 Model for the Task of Mispronunciation De- tection and Diagnosis,

L. Peng, K. Fu, B. Lin, D. Ke, and J. Zhan, “A Study on Fine- Tuning wav2vec2.0 Model for the Task of Mispronunciation De- tection and Diagnosis,” inProc. Interspeech, 2021, pp. 4448– 4452

2021
[18]

Transformer based end-to-end mispronunciation detection and diagnosis,

M. Wu, K. Li, W.-K. Leung, and H. Meng, “Transformer based end-to-end mispronunciation detection and diagnosis,” inProc. Interspeech, 2021, pp. 3954–3958

2021
[19]

Explore wav2vec 2.0 for Mispronunciation Detection,

X. Xu, Y . Kang, S. Cao, B. Lin, and L. Ma, “Explore wav2vec 2.0 for Mispronunciation Detection,” inProc. Interspeech, 2021, pp. 4428–4432

2021
[20]

Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learn- ing,

E. Kim, J.-J. Jeon, H. Seo, and H. Kim, “Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learn- ing,” inProc. Interspeech, 2022, pp. 1411–1415

2022
[21]

Proficiency assessment of L2 spo- ken English using wav2vec 2.0,

S. Bann `o and M. Matassoni, “Proficiency assessment of L2 spo- ken English using wav2vec 2.0,” in2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 1088–1095

2023
[22]

Assessment of L2 Oral Proficiency Using Self-Supervised Speech Representation Learning,

S. Bann `o, K. M. Knill, M. Matassoni, V . Raina, and M. Gales, “Assessment of L2 Oral Proficiency Using Self-Supervised Speech Representation Learning,” inProc. 9th Workshop on Speech and Language Technology in Education (SLaTE), 2023, pp. 126–130

2023
[23]

Automatic assessment of conversational speak- ing tests,

S. W. McKnight, A. Civelekoglu, M. Gales, S. Bann `o, A. Liusie, and K. M. Knill, “Automatic assessment of conversational speak- ing tests,” in9th Workshop on Speech and Language Technology in Education (SLaTE), 2023, pp. 99–103

2023
[24]

SALMONN: Towards Generic Hearing Abilities for Large Language Models,

C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, M. Zejun, and C. Zhang, “SALMONN: Towards Generic Hearing Abilities for Large Language Models,” inThe Twelfth International Con- ference on Learning Representations
[25]

Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,

Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,”arXiv preprint arXiv:2311.07919, 2023

2023 arXiv
[26]

Qwen2-audio technical report,

Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Linet al., “Qwen2-audio technical report,”arXiv preprint arXiv:2407.10759, 2024

2024 arXiv
[27]

Exploring the potential of using an ai language model for automated essay scoring,

A. Mizumoto and M. Eguchi, “Exploring the potential of using an ai language model for automated essay scoring,”Research Meth- ods in Applied Linguistics, vol. 2, no. 2, p. 100050, 2023

2023
[28]

Rating short L2 essays on the CEFR scale with GPT-4,

K. P. Yancey, G. Laflair, A. Verardi, and J. Burstein, “Rating short L2 essays on the CEFR scale with GPT-4,” inProceedings of the 18th Workshop on Innovative Use of NLP for Building Educa- tional Applications (BEA 2023). Toronto, Canada: Association for Computational Linguist...

2023
[29]

Can GPT-4 do L2 analytic assessment?

S. Bann `o, H. K. Vydana, K. Knill, and M. Gales, “Can GPT-4 do L2 analytic assessment?” inProceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024). Mexico City, Mexico: Association for Computa- tional Linguistics, Jun. 2024, p...

2024
[30]

Pronunciation Assess- ment with Multi-modal Large Language Models,

K. Fu, L. Peng, N. Yang, and S. Zhou, “Pronunciation Assess- ment with Multi-modal Large Language Models,”arXiv preprint arXiv:2407.09209, 2024

2024 arXiv
[31]

Emergent Abilities of Large Language Models,

J. Wei, Y . Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzleret al., “Emergent Abilities of Large Language Models,”Transactions on Machine Learning Research, 2022

2022
[32]

of Europe

C. of Europe. Council for Cultural Co-operation. Education Com- mittee. Modern Languages Division,Common European frame- work of reference for languages: Learning, teaching, assessment. Cambridge University Press, 2001

2001
[33]

Ludlow,Official Quick Guide to Linguaskill

K. Ludlow,Official Quick Guide to Linguaskill. Cambridge Uni- versity Press, 2020

2020
[34]

The Speak & Improve Corpus 2025: an L2 English Speech Corpus for Language Assessment and Feedback,

K. Knill, D. Nicholls, M. J. Gales, M. Qian, and P. Stroinski, “The Speak & Improve Corpus 2025: an L2 English Speech Corpus for Language Assessment and Feedback,” 2025. [Online]. Available: https://doi.org/10.17863/CAM.114333

2025 doi
[35]

Speak & Improve Challenge 2025: Tasks and Baseline Systems,

M. Qian, K. Knill, S. Banno, S. Tang, P. Karanasou, M. J. Gales, and D. Nicholls, “Speak & Improve Challenge 2025: Tasks and Baseline Systems,”arXiv preprint arXiv:2412.11985, 2024

2025 arXiv
[36]

Universal adversarial attacks on spoken language assessment systems,

V . Raina, M. J. Gales, and K. M. Knill, “Universal adversarial attacks on spoken language assessment systems,” inProc. Inter- speech, 2020, pp. 3855–3859

2020
[37]

Robust speech recognition via large-scale weak supervision,

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

2023
[38]

Mitigating Word Bias in Zero-shot Prompt-based Classifiers,

A. Liusie, P. Manakul, and M. Gales, “Mitigating Word Bias in Zero-shot Prompt-based Classifiers,” inFindings of the Associa- tion for Computational Linguistics: IJCNLP-AACL 2023 (Find- ings), 2023, pp. 327–335

2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Assessment of L2 Oral Proficiency using Speech Large Language Models

Introduction With the growing number of second language (L2) English learners worldwide, the demand for building automated systems capable of assessing their spoken language proficiency has been steadily increasing. These systems play a crucial role in scoring the candidates’ responses and providing feedback in language learning. Compared to human annotat...

work page Pith review arXiv 2025

[2] [2]

A”to“F”) based on the candidate’s speech responses. Here, a function scoreis used to map each text-based grade into its numeric class (e.g. label“A

Methodology For the spoken language assessment datasets studied in this pa- per, reference scores are annotated by human graders on a scale from 1 to 6 for each audio response, with specific instructions provided for each class level. Half-point scores (e.g., 3.5) are allowed for intermediate ratings. Our goal is to develop auto graders that produce score...

work page

[3] [3]

Experiments 3.1. Data Setup This paper uses two datasets for building and evaluating spo- ken language assessment systems: a private dataset, Lingual- skill [29]; and a public corpus, Speak & Improve Corpus 2025 (S&I) [30]. Each test from Linguaskill and S&I contains five parts focusing on different aspects of the candidate’s language proficiency: For Par...

work page 2025

[4] [4]

Vari- ous training and decoding schemes are compared on Linguaskill and S&I datasets

Conclusions In this paper, we examine the effectiveness of building spo- ken language assessment systems using speech LLMs. Vari- ous training and decoding schemes are compared on Linguaskill and S&I datasets. The zero-shot experiments show the potential of leveraging speech LLMs for L2 oral scoring. With further adaptation, the Qwen2Audio grader trained ...

work page

[5] [5]

Automatic evaluation and training in English pronuncia- tion,

J. Bernstein, M. Cohen, H. Murveit, D. Rtischev, and M. Wein- traub, “Automatic evaluation and training in English pronuncia- tion,” inFirst International Conference on Spoken Language Pro- cessing, 1990

work page 1990

[6] [6]

Automatic evaluation of Dutch pronunciation by using speech recognition technology,

C. Cucchiarini, H. Strik, and L. Boves, “Automatic evaluation of Dutch pronunciation by using speech recognition technology,” in 1997 IEEE Workshop on Automatic Speech Recognition and Un- derstanding Proceedings, 1997, pp. 622–629

work page 1997

[7] [7]

The SRI EduSpeak™ system: Recog- nition and pronunciation scoring for language learning,

H. Franco, V . Abrash, K. Precoda, H. Bratt, R. Rao, J. Butzberger, R. Rossier, and F. Cesari, “The SRI EduSpeak™ system: Recog- nition and pronunciation scoring for language learning,” inPro- ceedings of InSTILL, 2000, pp. 123–128

work page 2000

[8] [8]

Speechrater: A construct- driven approach to scoring spontaneous non-native speech,

K. Zechner, D. Higgins, and X. Xi, “Speechrater: A construct- driven approach to scoring spontaneous non-native speech,” in Proc. SLaTE, 2007, pp. 128–131

work page 2007