{"id":"fd774268-fd9d-4f0a-94ad-38979938198a","arxiv_id":"2509.03256","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The best model, GOP-CTC-AF-E2E, combines alignment-free CTC goodness-of-pronunciation features with a transformer and end-to-end fine-tuning, achieving the top leaderboard scores in the NOCASA 2025 challenge.","lead":"This paper compares three end-to-end neural models that automatically score how well children pronounce Norwegian words, and reports that a model using CTC-based pronunciation features, trained end-to-end, won the NOCASA 2025 challenge leaderboard. The result is a practical advance for computer-aided language learning for low-resource languages.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set size contradicts itself (1,930 vs 1,460 utterances), leaving the experimental basis of the top-leaderboard claim ambiguous.","rationale":"The reader's verdict is CONDITIONAL, and I agree with that outcome, but I locate the load-bearing concern differently. The reader singled out the pseudo-speaker clustering in §3.1 as the weakest assumption; while that is a legitimate internal-validity issue, it does not directly threaten the headline claim because the final model was selected on the official test leaderboard, not on the dev set, and any speaker leakage in the dev split would tend to make test performance conservative rather than optimistic. The dataset-size contradiction, by contrast, strikes at the reproducibility of the central empirical claim itself. The introduction and §3.1 give incompatible counts for both the training and test sets (10,334 vs 7,857 training; 1,930 vs 1,460 test). Since the paper's contribution is an empirical comparison on an official held-out test set, a reader cannot verify the experimental basis of the numbers without resolving this contradiction. The official leaderboard is external and provides some support for the claim, but the manuscript's internal inconsistency prevents a reader from knowing whether the reported 44.8% UAR / 0.505 MAE correspond to the official test set or to an unstated subset. This is a concrete, checkable flaw, and correcting it does not require any new experiments—only a comparison with the official challenge documentation and a clear statement of which utterances were used in each split. Therefore the verdict remains CONDITIONAL, with the condition being the resolution of this dataset-size discrepancy.","tokens_in":7583,"tokens_out":15490,"duration_ms":136855,"concrete_test":"Consult the official NOCASA 2025 challenge description (arXiv:2504.20678) and the evaluation server to establish the exact number of test utterances and labeled training utterances. Confirm whether the test set contains 1,930 or 1,460 utterances. If the official test set is 1,930, the Section 3.1 description is erroneous; the authors must clarify whether the 1,460 figure refers to a filtered subset and explain why the internal split only accounts for 7,857 of the 10,334 training utterances. If the official test set is 1,460, the introduction's 1,930 is wrong. In either case, the authors should correct the discrepancy and state which utterances were used for training, development, and the five leaderboard submissions, so the reported UAR/MAE can be traced to a single, well-defined test set.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central empirical claim rests on an internally inconsistent description of the challenge data. The introduction states a training corpus of 10,334 utterances and a test set of 1,930 utterances from 8 speakers; Section 3.1 states the provided dataset 'consists of 7857 labeled training utterances and 1460 unlabeled test utterances.' These figures differ by 2,477 training utterances and 470 test utterances. Section 3.1's internal speaker-level split yields 6,395 train + 1,462 dev = 7,857, matching the 7,857 figure but not the 10,334 figure. Because the headline result (UAR 44.8%, MAE 0.505) is a comparison against the official challenge baselines on the official test set, the reader cannot tell whether the reported numbers were computed on the same test set described by the organizers. If the official test set is 1,930, then Section 3.1 describes a different, smaller evaluation set, and the leaderboard comparison may rely on an unstated subset or a mislabeled dev set. If the official test set is 1,460, then the introduction overstates the test data by 470 utterances. Either way, the experimental setup is ambiguous, and the central empirical claim is not fully reproducible from the manuscript alone. This concern is independent of the pseudo-speaker clustering issue: even a perfect speaker split would not resolve the dataset-size contradiction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports the TalTech submission to the NOCASA 2025 pronunciation assessment challenge. It describes three end-to-end models: a Siamese encoder-decoder model (E2E-R), a prefix-tuned wav2vec2.0 classification model, and a novel GOP-CTC-AF-E2E model that makes alignment-free CTC-based goodness-of-pronunciation features end-to-end trainable. The paper also proposes a weighted ordinal cross-entropy loss aimed at optimizing UAR and MAE. The authors report development-set comparisons, an ablation study, and five official leaderboard submissions. Their primary system (submission #4) is claimed to achieve top leaderboard scores with UAR 44.8% and MAE 0.505, surpassing the challenge baselines (UAR 36.4%, MAE 0.55).","tokens_in":7863,"tokens_out":8995,"duration_ms":86386,"significance":"The strongest asset of the paper is that the headline result comes from the official NOCASA leaderboard rather than from a self-constructed test set, which is a meaningful form of external validation. The ablation showing that freezing the pretrained CTC model drops UAR from 44.0% to 32.6% is a clear, falsifiable demonstration that end-to-end fine-tuning is important. The idea of making alignment-free GOP features differentiable and trainable is a useful methodological contribution. I do not see a circularity issue: the GOP features are computed from a pretrained model, after which all parameters are fine-tuned on human scores; no already-fitted quantity is reused to certify the result. However, the manuscript is not currently reproducible as a data description because of the internal inconsistency in training/test sizes and the unvalidated pseudo-speaker splitting procedure.","major_comments":[{"comment":"The paper gives incompatible dataset sizes in two places. Section 1 states a training corpus of 10,334 utterances and a test set of 1,930 utterances from 8 speakers, while Section 3.1 states that the provided dataset consists of 7,857 labeled training utterances and 1,460 unlabeled test utterances. Since Table 2 reports official leaderboard scores, the reader cannot tell whether the leaderboard was evaluated on the same test set described in Section 3.1. If the official test set is 1,930 utterances, then Section 3.1 describes a different, smaller evaluation set and the leaderboard comparison is not reproducible from the manuscript alone. If the official test set is 1,460 utterances, the introduction is wrong by 470 utterances. Please reconcile these numbers and explicitly state which split was used for each submission.","section":"Section 1 vs Section 3.1"},{"comment":"All model selection decisions (checkpoint selection by dev UAR, interpolation weights, loss alpha, class weights) are made on an internal development set obtained by unsupervised speaker clustering into pseudo-speakers. The paper provides no verification that the pseudo-speaker clusters correspond to true speakers. If a true speaker appears in both the training and development clusters, the development-set metrics are optimistic and the selected hyperparameters may not transfer to the official test set. The performance drop of submission #5 from 50.4% UAR on the development set to 42.0% on the test set is consistent with development overfitting. Please report cluster-quality diagnostics where possible, or at minimum show sensitivity of the main conclusions to alternative split seeds and clustering parameters.","section":"Section 3.1"},{"comment":"The weighted ordinal cross-entropy loss as written is under-specified. The summation runs over all classes i, including i = y, where d(y,y)=0. The term for i=y is therefore 0 times -log(1-p_y), which is undefined in the limit p_y -> 1. If the intended loss is a distance-weighted complement loss over i != y, the equation should state that explicitly; if conventional ordinal cross-entropy was intended, the formula should be corrected. As written, the method section does not support the claim that the loss is a cross-entropy-based objective, and a reader cannot reproduce the training loss from the formula.","section":"Section 2.2, Eq. (1)"}],"minor_comments":[{"comment":"The table references are inconsistent: Section 3.3 says 'Table 3 lists results' for the development-set results, which is actually Table 1, and the submission table is referred to as 'Table 3.2' rather than Table 2. Section 3.4 refers to 'Table 3.4' for the ablation table, which should be Table 3.","section":"Section 3.3 and Section 3.4"},{"comment":"There is a typo in the numbered list: 'Predictons' should be 'Predictions'.","section":"Section 3.3"},{"comment":"The text says 'mean average error (MAE)', but the metric is described elsewhere as mean absolute error; please use the correct term consistently.","section":"Section 2.2"},{"comment":"The test set is called 'unlabeled' in Section 3.1, but Section 1 says word transcripts are provided for the test set. Please clarify that only pronunciation scores are withheld.","section":"Section 3.1"},{"comment":"The abstract and introduction state that the system achieved top leaderboard scores, but Table 2 only lists the authors' own submissions and challenge baselines. Please include the final leaderboard rank or a link to the official leaderboard so this claim is verifiable.","section":"Section 1"},{"comment":"The paper reports single evaluation numbers without confidence intervals or significance tests. Given the small number of official submissions allowed, a bootstrap over the development set or a statement about evaluation variance would help the reader assess whether the differences between submissions are meaningful.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The data-size contradiction and the unvalidated pseudo-speaker split are the main obstacles to acceptance. The leaderboard result may well be correct, but as written the manuscript does not allow a reader to map the reported scores to a clearly defined test set. Please ask the authors to correct the dataset description, state the exact official split used for each submission, and provide cluster-validation or robustness evidence before requesting a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing you should know: this paper reports a clear win on the NOCASA 2025 challenge — their GOP-CTC-AF-E2E model beats the official baselines by a wide margin (44.8% UAR vs 36.4% baseline, MAE 0.505 vs 0.55). The core novelty is making the alignment-free CTC GOP features end-to-end trainable, with letter embeddings and a transformer layer on top. The ablations show that end-to-end fine-tuning matters a lot (frozen CTC drops UAR from 44.0 to 32.6), and the weighted ordinal CE loss is a simple, reusable idea. The paper is well written and the negative result on dev-based interpolation (submission #5 overfitting) is honestly reported.\n\nNow the soft spots. The stress-test note is correct: the test-set size is internally inconsistent. The introduction says 1,930 test utterances from 8 speakers; Section 3.1 says 1,460. That’s a 470-utterance gap, not a typo-level error. Since the headline claim is about beating baselines on the official test set, the reader genuinely cannot tell which number is right or whether the evaluation was done on the same set described. This needs an explicit correction or the experimental basis is ambiguous. Second, there are no confidence intervals or significance tests. On a test set from only 8 speakers, a 8.4-point UAR gap is likely real, but with that few speakers the variance could be large. Third, the pseudo-speaker clustering for the dev split is unvalidated — speaker leakage between train and dev could bias checkpoint selection and hyperparameters, though the final leaderboard numbers come from the official server so this is a secondary issue. Finally, no code or data release, which limits reproducibility.\n\nThe citations look reasonable: they build on Cao et al.'s CTC GOP work and cite the challenge description, and the self-citation to Alumäe's MADASR system is legitimately relevant for the prefix-tuning variant.\n\nThis is a workshop-style paper (IEEE MLSP) and it does what a good challenge paper should: clear architecture comparison, honest ablations, and a strong official result. The central idea is likely correct, but the data inconsistency and missing error bars need addressing. I'd send it to peer review, but with the expectation that the authors clean up the data description and add uncertainty estimates before publication.\n\nReading group: worth a look if anyone in your group does speech assessment or uses ordinal losses. Otherwise, it's a fine skim.","headline":"A solid, incremental workshop paper with a real empirical result (top NOCASA leaderboard scores) but a significant data-description inconsistency that must be fixed before the leaderboard claim is credible.","tokens_in":8400,"tokens_out":2267,"would_cite":false,"duration_ms":22108,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An end-to-end trainable GOP-CTC model achieved the top leaderboard scores in the NOCASA 2025 pronunciation assessment challenge, with 44.8% UAR and 0.505 MAE on the official test set.","keywords":["pronunciation assessment","NOCASA 2025","goodness-of-pronunciation","CTC","end-to-end","Norwegian L2","ordinal loss","speech scoring"],"falsifier":"Run the same training pipeline using genuine speaker identities for the train/dev split and compare the selected models' UAR and MAE on the official test set; if the pseudo-speaker clustering had leaked speakers between train and dev, the primary result (44.8% UAR) would not replicate or the dev-based ranking would change.","tokens_in":7380,"feed_emoji":"🎙️","tokens_out":5121,"duration_ms":40735,"temperature":0.7,"pith_summary":"This paper compares three end-to-end systems for word-level pronunciation assessment of children learning Norwegian as a second language in the NOCASA 2025 challenge. Its central finding is that a model which converts alignment-free CTC-based goodness-of-pronunciation features into trainable end-to-end inputs, called GOP-CTC-AF-E2E, outperforms both the other architectures and the official baselines. On the challenge test set, the model reaches 44.8% unweighted average recall (UAR) and 0.505 mean absolute error (MAE), versus baseline scores of 36.4% UAR and 0.55 MAE. The results matter because reliable automatic scoring of L2 child speech could enable feedback at scale without expert human listeners.","feed_headline":"GOP-CTC model tops NOCASA speech-scoring leaderboard","feed_subtitle":"Alignment-free CTC pronunciation features, end-to-end trained, beat challenge baselines on both UAR and MAE.","key_machinery":"The central object is the set of alignment-free CTC GOP features for each canonical letter $c_i$: the log posterior of the whole canonical sequence (LPP), the vector of log posterior ratios for every possible substitution $\\mathrm{LPR}_{sub}(i,j)$, and the scalar log posterior ratio for deleting $c_i$, all computed from CTC emissions and marginalized with the forward algorithm. These values, concatenated with a learned embedding of the letter, form a token sequence fed into a single Transformer layer (the 'GOP Transformer'), max-pooled, and classified into scores 1 to 5. The other load-bearing ingredient is the weighted ordinal cross-entropy loss $\\mathcal{L}_{ordinal}(P,y)=\\sum_i w_y[-\\log(1-p_i)\\cdot d(y,i)^\\alpha]$, which penalizes errors by their ordinal distance and balances classes; with $\\alpha=0.5$ it improves UAR and MAE.","core_discovery":"On the paper's own terms, the discovery is that pronunciation quality can be scored from substitution and deletion likelihood ratios computed with CTC, without any forced alignment, and that the whole pipeline—feature extraction, letter embeddings, a small Transformer, and the classifier, together with the pretrained CTC backbone—can be fine-tuned jointly for the scoring task. The authors show that freezing the backbone collapses performance on the internal development set (UAR drops from 44.0% to 32.6%), while removing the Transformer or the letter embeddings costs only a few points, and swapping the weighted ordinal cross-entropy for plain cross-entropy also hurts UAR and MAE. The resulting system achieved the top leaderboard scores among all participants.","pith_inferences":["If the same recipe transfers, alignment-free CTC GOP features could be used for phoneme-level or utterance-level feedback and for other L2 languages with pretrained CTC models.","The dev-set split relies on pseudo-speaker clustering; without verification that clusters correspond to speakers, the reported dev-based model rankings may be optimistic, and official test scores could shift if selection were repeated with true speaker labels.","A testable extension is to run the same challenge pipeline using ground-truth speaker information when it is released, to quantify the effect of pseudo-labeling on checkpoint and hyperparameter selection.","The efficiency claim—CTC emissions computed once, then many marginalized likelihoods—suggests the approach can scale to real-time feedback apps, but the paper does not measure latency directly."],"forward_implications":["Making the CTC backbone trainable is essential: freezing it lowers UAR from 44.0% to 32.6% on the internal development set.","Weighted ordinal cross-entropy with $\\alpha=0.5$ gives the best UAR/MAE trade-off, while larger $\\alpha$ suppresses predictions of extreme scores 1 and 5.","Interpolating the three models improves development UAR (50.4%) but not test UAR (42.0% vs 44.8%), indicating that dev-based interpolation weights overfit.","Alignment-free GOP features remove the need for external forced-alignment tools, making the method applicable to languages and datasets without phoneme alignments."],"supporting_citations":[{"why":"Defines the NOCASA challenge, its data, and the two baselines that the best model outperforms.","marker":"[1]"},{"why":"Supplies the original alignment-free CTC GOP feature extraction that the winning model makes end-to-end trainable.","marker":"[2]"},{"why":"Provides the E2E-R architecture that is adapted and compared as one of the three systems.","marker":"[3]"},{"why":"Introduces prefix-tuning used by the second model to condition on prompted word identity.","marker":"[5]"}],"fun_headline_variants":["GOP-CTC without alignment wins NOCASA pronunciation scoring","Alignment-free CTC pronunciation scoring wins NOCASA challenge","CTC-based GOP without alignment takes top NOCASA spot","GOP-CTC tops NOCASA leaderboard with alignment-free scoring","Jointly fine-tuned CTC-GOP model beats all in NOCASA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the internal development set, created by spectral clustering of speaker embeddings into pseudo-speakers, accurately separates true speakers, since all model selection—checkpoints, class weights, loss scale $\\alpha$, and interpolation weights—is tuned against this set.","fun_headline_variants_meta":{"raw":{"variants":["GOP-CTC without alignment wins NOCASA pronunciation scoring","Alignment-free CTC pronunciation scoring wins NOCASA challenge","CTC-based GOP without alignment takes top NOCASA spot","GOP-CTC tops NOCASA leaderboard with alignment-free scoring","Jointly fine-tuned CTC-GOP model beats all in NOCASA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000873,"raw_usage":{"total_tokens":3713,"prompt_tokens":813,"completion_tokens":2900,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":2812}},"tokens_in":429,"tokens_out":2900,"duration_ms":18712,"temperature":1.0,"reasoning_tokens":2812,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:31:55.279572+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same training pipeline using genuine speaker identities for the train/dev split and compare the selected models' UAR and MAE on the official test set; if the pseudo-speaker clustering had leaked speakers between train and dev, the primary result (44.8% UAR) would not replicate or the dev-based ranking would change.","supporting_citations":[{"cited_title":"Such systems can be used to provide feedback in computer-aided language learning applications","cited_arxiv_id":null,"evidence_quote":"Defines the NOCASA challenge, its data, and the two baselines that the best model outperforms."},{"cited_title":"Comparison of End-to-end Speech Assessment Models for the NOCASA 2025 Challenge","cited_arxiv_id":"2509.03256","evidence_quote":"Supplies the original alignment-free CTC GOP feature extraction that the winning model makes end-to-end trainable."},{"cited_title":"GOP Transformer","cited_arxiv_id":null,"evidence_quote":"Provides the E2E-R architecture that is adapted and compared as one of the three systems."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces prefix-tuning used by the second model to condition on prompted word identity."}],"review_version":2}