REVIEW 2 major objections 5 minor 29 references
The NTNU System at the S&I Challenge 2025 SLA Open Track
T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper's central claim is that score-conditioned fusion of a wav2vec 2.0 acoustic grader and a Phi-4 multimodal grader reaches RMSE 0.375 on the challenge evaluation set, beating each single grader and the official baseline.
desk verdict A clean, honest challenge system paper whose fusion gain over its best single grader is real enough for the leaderboard but not yet shown to be statistically reliable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the score-conditioned fusion identity: each grader's predicted score is placed into one of eight CEFR-aligned intervals, and each interval gets its own interpolation weight, chosen on the development set to minimize RMSE. The fusion works because the wav2vec 2.0 grader is strong on pronunciation, fluency, and prosody, while the task-specific Phi-4 multimodal grader carries semantic and syntactic reasoning; their errors only partially overlap, so a weighted average reduces overall error. Task-specific models are trained separately for each assessment part, and the overall proficiency score is the average of part scores.
What would settle it
Take the trained wav2vec grader, the trained Phi-4 task-specific graders, and the fixed per-interval weights, run them on a fresh held-out set drawn from a different task, prompt set, or learner population, and compare fused RMSE to the best single grader; if fusion no longer beats the single grader, the claim of general complementary benefit fails.
Extended reading notes
Core claim
The central claim is that score-conditioned fusion of two graders, denoted W2V + Phi4-STG, improves spoken language assessment accuracy beyond either component alone. A wav2vec 2.0 grader estimates proficiency from raw audio using attention pooling and prototype similarity, while a task-specific Phi-4 multimodal grader consumes raw audio, the ASR transcript, the question, and the task description. Their predictions are combined as $\hat{y} = (1-w_k)\hat{y}_{w2v} + w_k \hat{y}_{mllm}$, where the weight $w_k$ is chosen per CEFR-aligned score interval by grid search on the development set. On the evaluation set this fused system gives RMSE 0.375, PCC 0.820, and SRC 0.827, whereas the two single graders give RMSE 0.394 and 0.389. The result ranks second on the official leaderboard, behind a system at 0.364 and ahead of the third-place system at 0.384.
Load-bearing premise
The eight fusion weights were tuned on the development set and then held fixed, so the reported advantage assumes the evaluation responses come from the same distribution; a distribution shift could erase the 0.375 versus 0.389 gain.
Editorial extensions
If this is right
- If the RMSE 0.375 result holds, score-level fusion is a practical way to combine a lightweight acoustic model with a large multimodal model without jointly fine-tuning them.
- Task-specific per-part models outperform one shared cross-task model in this setup, suggesting that assessment parts need specialized graders rather than a single generalist.
- Acoustic input remains essential: variants that use only transcription are worse, so raw speech carries irreplaceable signal that text alone cannot supply.
- The per-interval weights show that the optimal mixing ratio depends on the predicted score band, so low-scoring and high-scoring responses may favor different graders.
- The fused system's gains over the official BERT baseline (0.375 vs 0.445) are large enough to matter for real CALL systems that give learners fine-grained feedback.
Reading between the lines
- A natural extension the paper does not test is whether the per-interval weights transfer across languages, prompts, or learner populations; a small calibration set could reveal how much of the 0.014 RMSE gain over the single MLLM is stable versus dataset-specific.
- Because fusion does not require retraining either grader, one could replace the Phi-4 grader with a cheaper semantic model and measure how much benefit remains, isolating the source of complementarity.
- The grid search over eight fixed intervals could be replaced by a smooth learned weighting function, which might improve predictions near interval boundaries.
- The same two-grader pattern may generalize to other holistic assessments, such as pronunciation scoring or oral exams, where one model handles delivery and another handles content.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes the NTNU system submitted to the Speak & Improve Challenge 2025 SLA Open Track. The system combines two graders for L2 spoken language assessment: a wav2vec 2.0-based speech grader that processes raw audio, and a Phi-4-multimodal LLM grader (task-specific variants, Phi4-STG, and cross-task variants, Phi4-CTG) that uses raw speech, ASR transcripts, question prompts, and task descriptions. The two graders are combined by a score fusion in Eq. (4), where per-CEFR-interval weights are selected on the development set by grid search to minimize RMSE. The final system (W2V + Phi4-STG) is reported to achieve RMSE 0.375 on the official evaluation set, ranking second in the challenge, compared with 0.394 for W2V alone, 0.389 for Phi4-STG alone, and 0.444/0.445 for the official baseline. The paper also presents comparisons among BERT, W2V, Phi4-CTG variants, and the fused systems.
Significance. If the reported result is reliable, the paper makes a useful empirical contribution to automated L2 speaking assessment by demonstrating that a score-level fusion of an acoustic self-supervised grader and a multimodal LLM grader can outperform either single grader on an official, held-out challenge evaluation set. The experimental setup is clearly described and follows the official data splits and evaluation metrics, which is a strength. The paper also compares several model configurations (task-specific vs. cross-task MLLMs, audio-only vs. transcript-only vs. combined inputs), which gives the reader a reasonable picture of the design space. These concrete comparisons, together with the second-place leaderboard result, are meaningful evidence for the viability of the proposed approach. The main weaknesses are the absence of uncertainty quantification and the incomplete per-part analysis of the fused system, which limit how strongly the general claim of fusion benefit can be stated.
major comments (2)
- [§2.4, Eq. (4)] The central claim that score fusion improves over both single graders rests on point estimates from a single official evaluation set and on eight interval-specific weights selected on the development set. The reported gains are small (RMSE 0.375 vs. 0.389 for Phi4-STG and 0.394 for W2V), and the paper reports no confidence intervals, bootstrap estimates, or significance tests; it also does not state the number of evaluation responses. Because the weights in Eq. (4) are tuned to minimize dev-set RMSE, the evaluation result is not independent of model-selection noise, and it is unclear whether the observed 0.014–0.019 RMSE improvement is distinguishable from selection effects. Please report the development-set RMSE before and after fusion, the selected weight values, and an uncertainty estimate (e.g., bootstrap over evaluation responses), or explicitly temper the conclusion to a leaderboard result rather than a general claim about fusion benefit.
- [Figure 3, Table 1] The paper's interpretative claim that fusion 'substantiates the value of combining' W2V and Phi4-STG would be much better supported by a per-part breakdown of the fused system. Figure 3 reports per-part RMSE for BERT, W2V, Phi4-STG, and Phi4-CTG, but no such breakdown is given for W2V + Phi4-STG. Without this, it is impossible to know whether the overall improvement is consistent across Parts 1, 3, 4, and 5 or driven by a single part, and the fusion weights in Eq. (4) are not reported or analyzed for stability across parts. Please add the per-part RMSE for the fused system and discuss whether the fusion advantage is systematic or localized.
minor comments (5)
- [Abstract and Section 1 vs. Tables 1–2] The official baseline RMSE is reported inconsistently: the abstract and Section 1 state 0.444, Table 2 lists 0.440, and Table 1 lists the replicated BERT baseline as 0.445. Please reconcile these numbers and clarify which value is the official one.
- [Section 4] The sentence 'The standalone wav2vec 2.0 (W2V) model outperform this baseline' should use the singular verb 'outperforms'.
- [Figure 3] The y-axis of Figure 3 starts at 0, which compresses the RMSE differences among models (all values are between roughly 0.3 and 0.6); starting the axis near 0.3 would make the per-part comparisons much easier to read.
- [Section 2.3] The description of speaker context for Parts 1 and 5 says that 'all of a speaker's audio samples' are processed collectively, but no implementation details are given for how multiple audio samples are fed into the model or how the scores are aggregated. Please specify this for reproducibility.
- [Section 2.4] The term 'score-conditioned fusion' is used for weights that depend only on the discretized MLLM score interval. Consider defining this term explicitly at first use, since it could be confused with conditioning on the reference score or on the input speech.
Circularity Check
No significant circularity: the fusion result is an out-of-sample evaluation of dev-tuned weights.
full rationale
The paper's central claim is an empirical result measured on the official Speak & Improve Challenge 2025 evaluation set. The fusion weights w_k in Eq. (4) are selected on the development set by grid search to minimize RMSE and then fixed for evaluation, as stated in Section 2.4: "For each interval, the optimal interpolation weight w_k is selected via grid search on the development set, with the objective of minimizing the root mean square error (RMSE) between the fused prediction and the reference score. The weights were then fixed for evaluation." Thus the reported evaluation RMSE of 0.375 is not a fitted quantity; the evaluation labels are not used to choose the fusion weights or the component model parameters. The component graders are trained on the official training split with model selection on the development set, and the final evaluation is on a held-out set. This is standard supervised learning, not circular. The only overlapping-author citation is [15], used for the prototypical embedding module in the W2V grader. That is a methodological reference to a published architectural component and is not used to justify the central fusion claim or to exclude alternative approaches. Concerns about the statistical reliability of the small fusion gain, possible dev/test distribution shift, and leaderboard selection are validity or robustness issues, not circularity, and do not change this verdict.
Assumptions & free parameters
free parameters (1)
- Fusion weights w_k per CEFR interval =
not reported (8 values)
assumptions (3)
- domain assumption The official data splits as provided by the challenge are used, and the evaluation set is unseen during training.
- domain assumption The overall proficiency score is the average of the four part scores (Eq. 5).
- domain assumption The pretrained wav2vec 2.0 and Phi-4 models provide useful representations for SLA without task-specific architectural changes.
Cite this review
Pith. "Pith review of The NTNU System at the S&I Challenge 2025 SLA Open Track." pith.science (2026). https://pith.science/paper/C4OYTHPB
@misc{pith2026250605121,
author = {Pith},
title = {Pith review of: The NTNU System at the S&I Challenge 2025 SLA Open Track},
year = {2026},
howpublished = {\url{https://pith.science/paper/C4OYTHPB}},
note = {Machine review of arXiv:2506.05121}
}
read the original abstract
A recent line of research on spoken language assessment (SLA) employs neural models such as BERT and wav2vec 2.0 (W2V) to evaluate speaking proficiency across linguistic and acoustic modalities. Although both models effectively capture features relevant to oral competence, each exhibits modality-specific limitations. BERT-based methods rely on ASR transcripts, which often fail to capture prosodic and phonetic cues for SLA. In contrast, W2V-based methods excel at modeling acoustic features but lack semantic interpretability. To overcome these limitations, we propose a system that integrates W2V with Phi-4 multimodal large language model (MLLM) through a score fusion strategy. The proposed system achieves a root mean square error (RMSE) of 0.375 on the official test set of the Speak & Improve Challenge 2025, securing second place in the competition. For comparison, the RMSEs of the top-ranked, third-ranked, and official baseline systems are 0.364, 0.384, and 0.444, respectively.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction With the rapid advancement of computer technology and the increasing global population of second language (L2) learn- ers, spoken language assessment (SLA) has garnered significant attention, particularly within the domain of computer-assisted language learning (CALL). SLA systems are designed to pro- vide timely and informative feedback on l...
work page 2025
-
[2]
The NTNU System at the S&I Challenge 2025 SLA Open Track
Method Spoken language assessment (SLA) has traditionally capital- ized on text-based models; iconic ones include those built on BERT, which process transcripts generated by ASR systems. These models are effective at evaluating content-related aspect arXiv:2506.05121v2 [cs.CL] 11 Sep 2025 ASR Transcript 𝑅! Embed. 𝐱 𝜎(.) Speech 𝑅" BERT Encoder B1B2C Pooler...
work page Pith review arXiv 2025
-
[3]
Experiments 3.1. Dataset and evaluation metrics The Speak & Improve (S&I) Corpus 2025 [13] [21] contains 315 hours of open-ended spoken responses from L2 English learners with CEFR levels ranging from A2 to C1+. This cor- pus includes four distinct task types: Interview (Part 1), Opin- ion (Part 3), Presentation (Part 4), and Communication Activity (Part ...
work page 2025
-
[4]
Results & Discussion The experimental results are presented in Table 1, which pro- vide a comprehensive evaluation of multiple system config- urations for spoken language assessment (SLA). The official BERT-based baseline achieves an RMSE of 0.445, serving as a reference point for subsequent comparisons. The standalone wav2vec 2.0 (W2V) model outperform t...
-
[5]
Conclusion and Future Work This work has presented a multimodal large language model (MLLM) system tailored for spoken language assessment on L2 learners, developed in the context of the Speak & Improve Challenge 2025. By integrating a wav2vec 2.0-based speech grader with task-specific MLLMs through a score-level fusion mechanism, the proposed system achi...
work page 2025
-
[6]
Proficiency assessment of l2 spo- ken english using wav2vec 2.0,
S. Bann `o and M. Matassoni, “Proficiency assessment of l2 spo- ken english using wav2vec 2.0,” in2022 IEEE Spoken Language Technology Workshop (SLT), 2023, pp. 1088–1095
work page 2023
-
[7]
BERT: Pre-training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” inProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, ...
work page 2019
-
[8]
Automated scoring of spontaneous speech from young learners of english us- ing transformers,
X. Wang, K. Evanini, Y . Qian, and M. Mulholland, “Automated scoring of spontaneous speech from young learners of english us- ing transformers,” in2021 IEEE Spoken Language Technology Workshop (SLT), 2021, pp. 705–712
work page 2021
Show all 29 references
-
[9]
L2 proficiency assessment using self- supervised speech representations,
S. Bann `o, K. M. Knill, M. Matassoni, V . Raina, and M. J. F. Gales, “L2 proficiency assessment using self- supervised speech representations,” 2022. [Online]. Available: https://arxiv.org/abs/2211.08849
2022 arXiv
-
[10]
Universal adversarial attacks on spoken language assessment systems,
V . Raina, M. J. Gales, and K. M. Knill, “Universal adversarial attacks on spoken language assessment systems,” inInterspeech 2020, 2020, pp. 3855–3859
2020
-
[11]
wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,
A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,”Advances in Neural Information Processing Systems, vol. 33, pp. 12 449–12 460, 2020
2020
-
[12]
Lces: Zero-shot automated essay scoring via pairwise comparisons using large language models,
T. Shibata and Y . Miyamura, “Lces: Zero-shot automated essay scoring via pairwise comparisons using large language models,”
-
[13]
Automatic pronunciation assessment using self-supervised speech representation learning,
E. Kim, J.-J. Jeon, H. Seo, and H. Kim, “Automatic pronunciation assessment using self-supervised speech representation learning,” inInterspeech 2022, 2022, pp. 1411–1415
2022
-
[14]
Assessment of l2 oral proficiency using self-supervised speech representation learning,
S. Bann `o, K. M. Knill, M. Matassoni, V . Raina, and M. Gales, “Assessment of l2 oral proficiency using self-supervised speech representation learning,” in9th Workshop on Speech and Lan- guage Technology in Education (SLaTE), 2023, pp. 126–130
2023
-
[15]
SALMONN: Towards generic hearing abilities for large language models,
C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. MA, and C. Zhang, “SALMONN: Towards generic hearing abilities for large language models,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=14rn7HpKVk
2024
-
[16]
WavLLM: Towards robust and adaptive speech large language model,
S. Hu, L. Zhou, S. Liu, S. Chen, L. Meng, H. Hao, J. Pan, X. Liu, J. Li, S. Sivasankaran, L. Liu, and F. Wei, “WavLLM: Towards robust and adaptive speech large language model,” inFindings of the Association for Computational Linguistics: EMNLP 2024, Y . Al-Onaizan, M. Bansal, ...
2024
-
[17]
Can large language models automatically score proficiency of written essays?
W. A. Mansour, S. Albatarni, S. Eltanbouly, and T. Elsayed, “Can large language models automatically score proficiency of written essays?” inProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024...
2024
-
[18]
GQA: Training generalized multi- query transformer models from multi-head checkpoints,
J. Ainslie, J. Lee-Thorp, M. de Jong, Y . Zemlyanskiy, F. Lebron, and S. Sanghai, “GQA: Training generalized multi- query transformer models from multi-head checkpoints,” inThe 2023 Conference on Empirical Methods in Natural Language Processing, 2023. [Online]. Available: http...
2023
-
[19]
Available: https://arxiv.org/abs/2505.08498
[Online]. Available: https://arxiv.org/abs/2505.08498
-
[20]
Speak & improve challenge 2025: Tasks and baseline systems,
M. Qian, K. Knill, S. Banno, S. Tang, P. Karanasou, M. J. F. Gales, and D. Nicholls, “Speak & improve challenge 2025: Tasks and baseline systems,” 2024. [Online]. Available: https://arxiv.org/abs/2412.11985
2025 arXiv
-
[21]
Self-attention encoding and pooling for speaker recognition,
P. Safari, M. India, and J. Hernando, “Self-attention encoding and pooling for speaker recognition,” inInterspeech 2020, 2020, pp. 941–945
2020
-
[22]
An effective automated speaking assessment approach to mitigating data scarcity and imbalanced distribution,
T.-H. Lo, F.-A. Chao, T.-i. Wu, Y .-T. Sung, and B. Chen, “An effective automated speaking assessment approach to mitigating data scarcity and imbalanced distribution,” inFindings of the Association for Computational Linguistics: NAACL 2024, K. Duh, H. Gomez, and S. Bethard, E...
2024
-
[23]
of Europe
C. of Europe. Council for Cultural Co-operation. Education Com- mittee. Modern Languages Division,Common European frame- work of reference for languages: Learning, teaching, assessment. Cambridge University Press, 2001
2001
-
[24]
Atten- tion is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Atten- tion is all you need,” inAdvances in Neural Information Processing Systems, I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garn...
2017
-
[26]
LongRoPE: extending LLM context window be- yond 2 million tokens,
Y . Ding, L. L. Zhang, C. Zhang, Y . Xu, N. Shang, J. Xu, F. Yang, and M. Yang, “LongRoPE: extending LLM context window be- yond 2 million tokens,” inProceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024
2024
-
[27]
LoRA: Low-rank adaptation of large language models,
E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations, 2022. [Online]. Available: https: //openreview.net/forum?id=nZeVKeeFYf9
2022
-
[28]
The Speak & Improve Corpus 2025: an L2 English Speech Corpus for Language Assessment and Feedback,
K. Knill, D. Nicholls, M. J. Gales, M. Qian, and P. Stroinski, “The Speak & Improve Corpus 2025: an L2 English Speech Corpus for Language Assessment and Feedback,” 2025. [Online]. Available: https://doi.org/10.17863/CAM.114333
2025 doi
-
[29]
Flashattention-2: Faster attention with better parallelism and work partitioning,
T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https: //openreview.net/forum?id=mZn2Xyh9Ec
2024
-
[2025]
Notably, it outperforms both its individual components and alternative fusion approaches, such as W2V + Phi4-CTG (RMSE 0.377). These results substantiate the value of combin- ing the fine-grained acoustic modeling of W2V with the contex- tual reasoning capabilities of task-spe...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.