REVIEW 2 major objections 5 minor 1 cited by
Non-native Children's Automatic Speech Assessment Challenge (NOCASA)
T0 review · 2 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read NOCASA releases a public benchmark for scoring children's Norwegian L2 pronunciation, with a best baseline UAR of 36.37%.
desk verdict Useful public benchmark for a missing task, but the headline performance ranking rests on 8 test speakers and utterance-level CIs, so treat the performance comparisons cautiously. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the paired corpus-plus-baseline setup. TeflonNorL2 provides 205 words with orthographic transcriptions and expert 1–5 ratings; the multi-task wav2vec 2.0 baseline carries the argument by keeping the original CTC/ASR head and adding a rating head on the last Transformer layer, using target words as ASR references only for high-scoring (4–5 star) utterances. The SVM/ComParE 16 baseline supplies an interpretable contrast. Together they define the evaluation protocol: UAR as primary metric, with accuracy, MAE, 95% bootstrap confidence intervals, and optional latency and explainability analyses.
What would settle it
A reader could refute the benchmark's usefulness by re-splitting the released data into a new held-out set that includes more speakers from underrepresented language backgrounds and checking whether the wav2vec 2.0 baseline still beats the SVM baseline by the same margin, or whether the ranking of submitted systems is preserved.
Extended reading notes
Core claim
The paper's central claim is that a usable public benchmark now exists for Norwegian L2 children's single-word pronunciation assessment, anchored by the TeflonNorL2 corpus and two reproducible baselines. On the released test set, the best official baseline, a multi-task wav2vec 2.0 model built on a Norwegian ASR base, achieves 36.37% UAR (95% CI 34.30–38.81), 54.45% accuracy, and 0.55 MAE, outperforming the SVM baseline (22.14% UAR). The paper also observes that the wav2vec model's latent clusters separate pronunciation levels only partially after fine-tuning, and that the rarest category (1 star) is never predicted; these findings are framed as directions for participants. The two top submitted systems, using regression with score calibration and goodness-of-pronunciation with ordinal loss, both outperform the baselines, supporting the benchmark's ability to distinguish stronger methods.
Load-bearing premise
The load-bearing premise is that the released test set, with only 8 speakers and only the most common first-language backgrounds, is representative enough to estimate system quality and rank competing methods for the broader population of Norwegian L2 children.
Editorial extensions
If this is right
- Any future system can be compared against two released baselines on the same fixed train/test split, making results across teams directly comparable.
- The multi-task wav2vec 2.0 design shows that keeping an ASR objective alongside the rating head is a viable way to use expected-word information under limited data.
- The reported performance leaves substantial room for improvement (36.37% UAR versus 20% chance-level), so the benchmark is positioned to track progress rather than declare the task solved.
- The top two submitted systems, using score calibration and goodness-of-pronunciation with ordinal loss, outperform both baselines, suggesting the benchmark can distinguish stronger methods.
Reading between the lines
- The paper's release of speaker characteristics only after the challenge means early users cannot control for speaker identity; a natural next step would be a speaker-disjoint cross-validation study on the training set to estimate how much the 8-speaker test set drives the ranking.
- Because ASR references are used only for 4–5 star recordings, the baseline may implicitly learn that high scores correlate with word-recognizability; extending this to all scores, or using a separate ASR pass, is a testable variant the paper leaves open.
- The low UAR on the 1-star category suggests a ceiling effect from class imbalance; techniques like class-weighted losses or synthetic augmentation for the rarest category could be evaluated directly against the released baselines.
- An implication for real-world deployment: the measured 30–50 ms per 3-second clip on a single GPU suggests the wav2vec 2.0 baseline is fast enough for real-time feedback, but whether that holds on edge devices used in a mobile app is not covered by the paper.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents NOCASA, a data competition for automatic pronunciation assessment of Norwegian L2 children's single-word speech. It releases the TeflonNorL2 corpus (7,857 training and 1,460 test utterances after cleaning), with expert ratings on a 1-5 star scale, and describes two official baselines: an SVM trained on ComParE 2016 features and a multi-task wav2vec 2.0 model. The multi-task wav2vec 2.0 model achieves a UAR of 36.37% on the held-out test set, and two participant systems are reported to surpass this baseline. The paper also discusses evaluation metrics, model robustness to random seeds, inference speed, and model interpretability.
Significance. If the resource works as claimed, it fills a clear gap: there is no previous public L2 children's pronunciation corpus for Norwegian. The paper's strengths include the public release of pseudo-anonymized data, fixed training/test splits, a code repository for the baselines, held-out test labels that were not used in model development, multiple evaluation metrics, bootstrapped confidence intervals, and explicit robustness checks across random seeds. The challenge format with a limited number of test trials and peer-reviewed participant papers is appropriate for benchmarking. The main weakness is that the test set is small and narrow, so the quantitative ranking evidence is less strong than the prose suggests; nevertheless, the corpus and baselines are a valuable community resource even if the performance comparisons need to be treated with caution.
major comments (2)
- [Section 4, Table 2] The 95% confidence intervals in Table 2 are computed by resampling utterances, but the 1,460 test utterances come from only 8 speakers. Because repeated utterances by the same child are strongly correlated, the effective sample size for speaker-level generalization is close to 8, not 1,460, so the reported intervals understate uncertainty. This matters for the Section 7 statements that Team 1 'managed to outperform both baselines' and Team 2 was 'the winning solution': Team 1's CI (33.13-53.80) already overlaps the MT w2v2 CI (34.30-38.81), and speaker-cluster resampling would likely widen all intervals further. Please recompute the intervals with speaker-level clustering (for example, bootstrapping over speakers) or add an explicit caveat that the ranking is based on point estimates and may not be statistically reliable.
- [Section 4, Abstract] The abstract states that TeflonNorL2 contains 10,334 recordings from 44 speakers, but Section 4 reports that after removing score-0 recordings and duplicate word attempts the final released data contain 7,857 training and 1,460 test audio files. Please reconcile these numbers and state clearly which quantity corresponds to the released corpus, since readers may otherwise be confused about the actual size of the benchmark.
minor comments (5)
- [Section 4] The test set contains only 8 speakers and only the most common L1 backgrounds, so the absolute UAR values and system rankings may not generalize to the full population of Norwegian L2 children; please state this limitation explicitly in the conclusions and avoid implying population-level conclusions.
- [Section 5] The SVM baseline is described as using 'default parameters', but no kernel or cost value is given; please provide the concrete hyperparameters or explicitly refer to the repository line numbers for full reproducibility.
- [Section 6] The observation that the 1-star class is never predicted is acknowledged, but reporting the per-class recall or a confusion matrix for the baselines would make the imbalance problem more concrete for participants.
- [Section 4] No inter-annotator agreement statistics are provided for the human ratings; a short summary or a reference to the specific section of [9] containing such statistics would help readers assess label noise.
- [Table 2] The confidence intervals for Team 1 and Team 2 are very wide; consider also reporting the number of test utterances per rating class or per speaker to make the source of this variability transparent.
Circularity Check
No circularity: NOCASA baselines are trained on a public training split and evaluated on a held-out test set whose labels are not used in development, so the central UAR claim is an independent empirical result.
full rationale
The paper makes no first-principles derivation; it releases a corpus and reports measured baseline performance. The central empirical claim, that the multi-task wav2vec 2.0 model achieves 36.37% UAR, comes from training on the provided training split (with a 10% validation subset used only for checkpoint selection) and evaluating on the held-out test split whose labels are not used in model development. No quantity in the paper is defined in terms of another fitted quantity: the SVM and wav2vec2 baselines are conventional methods, and reference [13], a prior work by some of the authors, is cited only as the architecture recipe, not as evidence that the baseline must perform at a given level. The corpus release and benchmark framing do not rename or redefine the target metric; UAR, ACC, and MAE are standard external metrics applied to unseen data. The statistical weaknesses noted elsewhere (8-speaker test set, utterance-level bootstrap confidence intervals) affect confidence and generalizability but are not circularity: they concern estimation error, not definitional equivalence. Accordingly, no circular step can be quoted, and the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- wav2vec2 learning rate =
2e-4
- number of epochs =
20
- batch size =
10
- SVM class weights =
not specified; class-weighting technique used
assumptions (4)
- domain assumption Human expert ratings on the 1 to 5 star scale are valid and sufficiently reliable ground truth for pronunciation accuracy.
- domain assumption The 8-speaker held-out test set, containing only the most common L1 backgrounds, is representative enough to rank systems by UAR.
- domain assumption The Norwegian ASR wav2vec2 model (NbAiLab nb-wav2vec2-300m-bokmaal) provides a suitable feature initialization for the pronunciation-rating head.
- standard math Bootstrapped confidence intervals are appropriate for quantifying uncertainty on this test set.
Cite this review
Pith. "Pith review of Non-native Children's Automatic Speech Assessment Challenge (NOCASA)." pith.science (2026). https://pith.science/paper/N3FM2STP
@misc{pith2026250420678,
author = {Pith},
title = {Pith review of: Non-native Children's Automatic Speech Assessment Challenge (NOCASA)},
year = {2026},
howpublished = {\url{https://pith.science/paper/N3FM2STP}},
note = {Machine review of arXiv:2504.20678}
}
read the original abstract
This paper presents the "Non-native Children's Automatic Speech Assessment" (NOCASA) - a data competition part of the IEEE MLSP 2025 conference. NOCASA challenges participants to develop new systems that can assess single-word pronunciations of young second language (L2) learners as part of a gamified pronunciation training app. To achieve this, several issues must be addressed, most notably the limited nature of available training data and the highly unbalanced distribution among the pronunciation level categories. To expedite the development, we provide a pseudo-anonymized training data (TeflonNorL2), containing 10,334 recordings from 44 speakers attempting to pronounce 205 distinct Norwegian words, human-rated on a 1 to 5 scale (number of stars that should be given in the game). In addition to the data, two already trained systems are released as official baselines: an SVM classifier trained on the ComParE_16 acoustic feature set and a multi-task wav2vec 2.0 model. The latter achieves the best performance on the challenge test set, with an unweighted average recall (UAR) of 36.37%.
Forward citations
Cited by 1 Pith paper
-
Comparison of End-to-end Speech Assessment Models for the NOCASA 2025 Challenge
The best model, GOP-CTC-AF-E2E, combines alignment-free CTC goodness-of-pronunciation features with a transformer and end-to-end fine-tuning, achieving the top leaderboard scores in the NOCASA 2025 challenge.
Reference graph
Works this paper leans on
-
[1]
Developing an ai-assisted low-resource spoken language learning app for children,
Y . Getman, N. Phan, R. Al-Ghezi, et al., “Developing an ai-assisted low-resource spoken language learning app for children,” IEEE Access, vol. 11, pp. 86 025– 86 037, 2023. DOI: 10 . 1109 / ACCESS . 2023 . 3304274
work page 2023
-
[2]
Auto- matic pronunciation assessment - a review,
Y . El Kheir, A. Ali, and S. A. Chowdhury, “Auto- matic pronunciation assessment - a review,” in Find- ings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali, Eds., Singapore: Association for Computational Linguistics, Dec. 2023, pp. 8304–8324. DOI: 10 . 18653 / v1 / 2023 . findings - emnlp . 557. [Online]. Ava...
work page 2023
-
[3]
J. M. Maxine Eskenazi and D. Graff, “The CMU Kids Corpus LDC97S63,” Philadelphia: Linguistic Data Consortium, 1997. DOI: 10.35111/b4v0-ff65
-
[4]
CSLU: Kids‘ Speech Version 1.1, LDC2007S18,
J.-P. H. Khaldoun Shobaki and R. Cole, “CSLU: Kids‘ Speech Version 1.1, LDC2007S18,” Philadelphia: Lin- guistic Data Consortium, 1997. DOI: 10 . 35111 / q5tn-8096
work page 1997
-
[5]
MyST Children’s Conversational Speech LDC2021S05,
R. C. Sameer Pradhan and W. Ward, “MyST Children’s Conversational Speech LDC2021S05,” Philadelphia: Linguistic Data Consortium, 2021. DOI: 10.35111/ cyxy-p432
work page 2021
-
[6]
Jasmin-spraakcorpus (version 1.0),
O. v. H. C. Cucchiarini H. V . Hamme and F. Smits, “Jasmin-spraakcorpus (version 1.0),” Available at the Dutch Language Institute: http://hdl.handle.net/10032/tm- a2-j7, 2008
work page 2008
-
[7]
The slt 2021 children speech recognition challenge: Open datasets, rules and baselines,
F. Y . et al., “The slt 2021 children speech recognition challenge: Open datasets, rules and baselines,” inProc. IEEE Spoken Lang. Tech. Workshop (SLT) , described in Proc. IEEE Spoken Lang. Tech. Workshop (SLT) 2021, Shenzen, China, 2021
work page 2021
-
[8]
kidsTALC: A Corpus of 3- to 11-year-old German Children’s Con- nected Natural Speech,
L. Rumberg, C. Gebauer, H. Ehlert, et al., “kidsTALC: A Corpus of 3- to 11-year-old German Children’s Con- nected Natural Speech,” in Proc Interspeech , 2022, pp. 5160–5164. DOI: 10.21437/interspeech. 2022-330
Show all 19 references
-
[9]
Collecting linguistic resources for assessing children‘s pronunciation of Nordic languages,
A. M. H. Olstad, A. Smolander, S. Str ¨ombergsson, et al. , “Collecting linguistic resources for assessing children‘s pronunciation of Nordic languages,” in Pro- ceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation...
2024
-
[10]
Liblinear: A library for large linear classi- fication,
R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin, “Liblinear: A library for large linear classi- fication,” J. Mach. Learn. Res., vol. 9, pp. 1871–1874, Jun. 2008, ISSN : 1532-4435
2008
-
[11]
The inter- speech 2016 computational paralinguistics challenge: Deception, sincerity & native language,
B. Schuller, S. Steidl, A. Batliner, et al., “The inter- speech 2016 computational paralinguistics challenge: Deception, sincerity & native language,” inInterspeech 2016, 2016, pp. 2001–2005. DOI: 10.21437/Inte rspeech.2016-129
2016 doi
-
[12]
Logistic regression in rare events data,
G. King and L. Zeng, “Logistic regression in rare events data,” Political analysis, vol. 9, no. 2, pp. 137– 163, 2001
2001
-
[13]
Multi-task wav2vec2 serving as a pronunciation train- ing system for children,
Y . Getman, R. Al-Ghezi, T. Grosz, and M. Kurimo, “Multi-task wav2vec2 serving as a pronunciation train- ing system for children,” in 9th Workshop on Speech and Language Technology in Education (SLaTE) , 2023, pp. 36–40. DOI: 10.21437/SLaTE.2023-8
2023 doi
-
[14]
Boosting Norwegian automatic speech recog- nition,
J. De La Rosa, R.-A. Braaten, P. Kummervold, and F. Wetjen, “Boosting Norwegian automatic speech recog- nition,” in Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), T´orshavn, Faroe Islands: University of Tartu Library, May 2023, pp. 555–564. ...
2023
-
[15]
Ferrer and P
L. Ferrer and P. Riera, Confidence Intervals for evalua- tion in machine learning. [Online]. Available:https: //github.com/luferrer/ConfidenceInte rvals
-
[16]
Uniform manifold ap- proximation and projection,
J. Healy and L. McInnes, “Uniform manifold ap- proximation and projection,” Nature Reviews Methods Primers, vol. 4, no. 1, p. 82, 2024
2024
-
[17]
On calibration of speech classification models: Insights from energy-based model investigations,
Y . Hao, C. Hu, Y . Gao, S. Zhang, and J. Feng, “On calibration of speech classification models: Insights from energy-based model investigations,” in Inter- speech 2024, 2024, pp. 3175–3179. DOI: 10.21437/ Interspeech.2024-1643
2024
-
[18]
Automated pronunciation scoring of child l2 learners with score calibration for imbalanced distribu- tions,
T. Yuuki, “Automated pronunciation scoring of child l2 learners with score calibration for imbalanced distribu- tions,” in Proc. IEEE MLSP, 2025
2025
-
[19]
Comparison of end- to-end speech assessment models for the nocasa 2025 challenge,
A. ˇZavoronkov and T. Alum ¨ae, “Comparison of end- to-end speech assessment models for the nocasa 2025 challenge,” in Proc. IEEE MLSP, 2025
2025
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.