{"id":"75739df9-acc8-4d7b-aa75-4a468686d635","arxiv_id":"2506.12067","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Logit-based GOP scores, especially maximum logit, can improve correlation with human pronunciation ratings, but do not consistently beat probability-based GOP in classification.","lead":"This paper tests whether raw logit values from a speech model, instead of softmax probabilities, give better scores for detecting mispronunciations in second-language English. Logit-based scores matched human judgments better on one dataset, but the improvement depends on the dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline MaxLogit-vs-human PCC is selected post hoc on the test set with no significance testing; confidence intervals or a held-out selection step are needed before claiming strongest alignment with human perception.","rationale":"The reader correctly flags the MPC simulated-error assumption and notes threshold optimization and significance testing as concerns, but their weakest_assumption is not the single most load-bearing point. The paper's strongest and most distinctive contribution is the perceptual-alignment result for GOPMaxLogit, not the dataset-dependent classification comparison, and that result is reported without any uncertainty quantification. The selection of MaxLogit as the best of four variants on the same test set where the correlations are computed introduces a winner's-curse risk that a bootstrap or held-out selection procedure would settle. The test-set threshold optimization is a separate validity threat to the classification claim, but it does not bear on the PCC result, so I focus the concrete test on the PCC. My read supports the reader's CONDITIONAL verdict: the paper is plausible and useful as a preliminary empirical comparison, but the headline correlation claim needs significance testing and proper validation before it can be accepted as established.","tokens_in":8466,"tokens_out":3719,"duration_ms":42164,"concrete_test":"Bootstrap SpeechOcean762 at the utterance level (or speaker level, to respect clustering) with at least 10,000 resamples; for each resample, compute PCC for GOPMaxLogit and GOPDNN, and report percentile 95% confidence intervals plus a paired bootstrap or Steiger test for the difference between the two correlations. Separately, to address the threshold issue, partition SpeechOcean762 into train/validation/test, select thresholds on the validation split only, and report test-set MCC/AUC for each GOP variant. If the MaxLogit-DNN PCC difference is not significant, or if MaxLogit no longer exceeds DNN after proper validation, the headline claim should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's most novel and load-bearing result is that GOPMaxLogit aligns best with human ratings, with PCC 0.442 (low confidence) and 0.456 (high confidence) on SpeechOcean762 (Table 2). This estimate is presented as the winner among four GOP variants, but the selection is made on the same test set used to compute the correlations, and no confidence intervals, significance tests, or correction for multiple comparisons are reported. If the observed gap over GOPDNN (0.278/0.295) is within sampling noise, the central perceptual-alignment claim is unsupported. The classification comparisons are additionally weakened by Section 2.5's practice of optimizing the GOP threshold on the test set by maximizing MCC, which can inflate and differentially bias the MCC values used to argue that logit-based methods outperform probability-based GOP. Because the strongest novel evidence is the correlation result, the absence of uncertainty quantification around that result is the most load-bearing concern; the simulated-error issue in MPC, while relevant, does not affect the SpeechOcean PCC finding.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares probability-based GOP (GOPDNN) with four logit-based GOP variants—GOPMaxLogit, GOPMargin, GOPVarLogit, and a hybrid GOPCombined—for phoneme-level mispronunciation detection and pronunciation assessment. Experiments are run on two L2 English datasets: MPC (Dutch children; simulated phoneme-level errors) and SpeechOcean762 (Mandarin speakers; human phoneme accuracy annotations). Classification is evaluated with accuracy, precision, recall, F1, MCC, and ROC AUC, and correlation with human ratings is measured via PCC and MSE after a second-order polynomial mapping. The main claims are that logit-based methods improve classification on MPC, GOPMaxLogit most strongly aligns with human perception on SpeechOcean762, and GOPCombined balances probability and logit information.","tokens_in":8673,"tokens_out":3898,"duration_ms":43362,"significance":"If the empirical claims hold, the contribution is practically valuable: a simple, non-trainable, model-agnostic family of GOP scores computable from any CTC-based acoustic model, with open-source code and evaluation on a public benchmark (SpeechOcean762) plus an in-house dataset. The paper also raises an interesting point about softmax overconfidence in GOP. However, the central empirical claims currently rest on test-set threshold optimization, winner selection on the test set without uncertainty quantification, and simulated errors in MPC. These issues must be resolved before the abstract-level claims about classification superiority and perceptual alignment are fully supported.","major_comments":[{"comment":"The classification thresholds are selected by maximizing MCC on the same test set used to report all classification results (Section 2.5: 'we optimized the GOP threshold by selecting the percentile that maximized MCC'). This procedure can inflate MCC for every method and, more importantly, can differentially favor methods whose score distributions happen to provide a convenient percentile. Because the abstract claim 'Logit-based methods outperform probability-based GOP in classification' is grounded in Tables 1 and 2, the threshold-selection procedure is load-bearing. Please use a held-out development set for threshold selection, or otherwise report the chosen threshold and evaluate at a fixed or pre-registered threshold, and add an estimate of optimism (e.g., cross-validated MCC).","section":"Section 2.5 / Tables 1-2"},{"comment":"The central perceptual-alignment result—GOPMaxLogit achieves the highest PCC with human phoneme accuracy ratings (0.442 low confidence, 0.456 high confidence)—is a winner-take-all selection made on the same test set used to compute the correlations. No confidence intervals, significance tests, or multiple-comparison corrections are reported, so the observed gaps over GOPDNN (0.278/0.295) and GOPVarLogit (0.341/0.357) could be within sampling noise. Please provide bootstrap confidence intervals for each PCC and, given the correlations are computed on the same utterances, a test for the difference between correlated correlation coefficients (e.g., a Williams test) or a held-out model-selection step.","section":"Section 3 / Table 2"},{"comment":"Because MPC lacks annotated mispronunciations, the authors 'introduced simulated pronunciation errors by modifying phoneme sequences,' including substitutions such as /D/ → /d/, /T/ → /s/, and diphthong simplification. The classification comparison on MPC—where logit-based GOPMargin achieves the largest MCC—therefore depends on the assumption that these synthetic substitutions are representative of natural L2 errors. If such substitutions introduce stronger or weaker acoustic cues than natural mispronunciations, the reported improvements may not generalize. Please validate the simulated errors (e.g., through human listening checks or comparison with naturally occurring L2 errors) or explicitly reframe the MPC results as a synthetic stress test rather than evidence for real-world classification gains.","section":"Section 2.4 / Table 1"},{"comment":"GOPCombined has a free parameter α ∈ [0,1], but the value used to produce the results in Tables 1 and 2 is never reported, nor is the selection procedure for α described. Since the abstract and discussion highlight GOPCombined as a balancing hybrid, the reader cannot assess whether its reported performance comes from the proposed combination or from an α tuned in hindsight on the test data. Please report α for each dataset and provide either a sensitivity analysis across α or a principled selection rule (e.g., development-set optimization).","section":"Section 2.2.4 / Eq. (6)"}],"minor_comments":[{"comment":"The discussion states that GOPMaxLogit 'achiev[es] the highest PCC scores (Table 1)', but the PCC results appear in Table 2; this cross-reference should be corrected.","section":"Section 4"},{"comment":"The notation l_t^{(p)} in Eq. (3) is not defined precisely; please state explicitly that it is the raw pre-softmax logit for phoneme p at frame t, and align this notation with the usage in Eqs. (4)–(6).","section":"Section 2.2 / Eq. (3)"},{"comment":"The threshold-optimization description is vague about the search space: please specify the percentile grid and the step size used when 'selecting the percentile that maximized MCC.'","section":"Section 2.5"},{"comment":"The abstract claims that the findings 'suggest that hybrid GOP methods incorporating uncertainty modeling and phoneme-specific weighting improve pronunciation assessment,' but the paper does not implement phoneme-specific weighting; either soften this claim or add the corresponding analysis.","section":"Abstract / Section 2.2.4"},{"comment":"The dataset name is inconsistently capitalized as 'Speechocean762' in some places and 'SpeechOcean762' in others; please standardize throughout, including table captions and figure labels.","section":"Global"}],"recommendation":"major_revision","confidential_remarks":"This is a competent empirical study with a practical contribution and open-source code, and the topic fits the journal/venue scope. My main editorial concern is that the headline claims are currently ahead of the evidence: the threshold optimization on the test set, the absence of uncertainty quantification for the winning PCC, and the synthetic-error basis of the MPC results are load-bearing. The paper is likely acceptable after these empirical points are addressed, but I would not recommend acceptance in the present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does one thing well: it takes four simple logit-based confidence scores, applies them to GOP, and compares them against softmax GOP on two L2 datasets. The empirical comparison is new, the code and model are public, and the paper is honest that results depend on dataset characteristics. The most interesting finding is that GOPMaxLogit correlates with human phoneme ratings on SpeechOcean762 (PCC 0.442/0.456) noticeably better than GOPDNN (0.278/0.295). If that holds, it is a cheap, model-agnostic upgrade for CAPT systems.\n\nThe soft spots are real but not disqualifying. The load-bearing correlation result is selected on the same test set used to compute it, with five metrics in play and no confidence intervals or significance tests. The gap could be sampling noise. The classification comparison is similarly weakened by optimizing thresholds on the test set to maximize MCC. On MPC the results rely on simulated pronunciation errors, which may not generalize to real L2 speech. Alpha in GOPCombined is not reported, and the paper makes no serious attempt to compare with the prior state of the art beyond noting that a multidimensional model reaches PCC 0.69.\n\nThat said, the paper does not overclaim. It repeatedly notes that performance depends on dataset, and the per-phoneme error analysis is a genuinely useful diagnostic. The weaknesses are fixable: a referee would ask for confidence intervals, a proper validation split or multiple-comparison correction, and a more careful treatment of the threshold optimization.\n\nThis is a modest but legitimate empirical paper for the CAPT community. It deserves peer review rather than desk rejection; with the requested re-analysis it could become a solid reference. I would send it out.","headline":"A useful, honest empirical comparison of logit-based GOP variants, but the headline MaxLogit–human correlation result is presented without uncertainty quantification and is selected on the test set.","tokens_in":9186,"tokens_out":2111,"would_cite":true,"duration_ms":25263,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Logit-based GOP scores improve mispronunciation detection, with the maximum-logit variant correlating best with human perception.","keywords":["goodness of pronunciation","GOP","logit-based GOP","mispronunciation detection","pronunciation assessment","CTC phoneme recognition","softmax posterior probabilities","L2 English speech"],"falsifier":"On an independently annotated corpus of naturally mispronounced L2 speech, compute the Pearson correlations of GOPMaxLogit and GOPDNN with human phoneme ratings; if GOPMaxLogit does not exceed GOPDNN's correlation, the paper's main perceptual-alignment claim would be overturned.","tokens_in":8280,"feed_emoji":"🎤","tokens_out":8543,"duration_ms":73251,"temperature":0.7,"pith_summary":"This paper tests whether goodness-of-pronunciation (GOP) scores computed from raw logits, instead of softmax posteriors, improve mispronunciation detection and agreement with human raters. On two L2 English datasets — Dutch children and Mandarin speakers — logit-based scores generally outperform probability-based GOP in classification, but the best score differs by dataset. The maximum-logit GOP correlates most strongly with human phoneme accuracy ratings on SpeechOcean762, while margin-based GOP performs best on the Dutch children's corpus. If the results hold, pronunciation-training systems could use a fast, non-trainable score that captures perceived pronunciation quality better than softmax-based GOP.","feed_headline":"Raw logit scores outperform softmax GOP in mispronunciation detection","feed_subtitle":"Maximum-logit GOP matches human phoneme ratings best, while margin GOP wins on children's speech.","key_machinery":"The machinery is a family of four GOP scores defined directly on raw logits: GOPMaxLogit (the maximum target-phoneme logit over aligned frames), GOPMargin (the mean difference between the target logit and the strongest competing logit), GOPVarLogit (the variance of target logits over the segment), and GOPCombined (a weighted sum of GOPMargin and the softmax GOPDNN). These scores are extracted after CTC-segmentation forced alignment with a fine-tuned wav2vec2 phoneme recognizer, so they need no extra training or parameters. The load-bearing mechanism is that raw logits preserve the model's unnormalized confidence, avoiding the overconfidence and gradient saturation of softmax normalization, which yields better separation between correct and mispronounced phonemes.","core_discovery":"The paper's central claim is that raw logits carry more discriminative information than softmax probabilities for scoring pronunciation, and that logit-based GOP scores are a viable non-trainable alternative for mispronunciation detection. Across the two corpora, logit-based scores beat probability-based GOP in classification, and GOPMaxLogit gives the strongest correlation with human phoneme accuracy judgments on SpeechOcean762, with Pearson correlations of 0.442 for low-confidence and 0.456 for high-confidence phoneme ratings. On the MPC corpus, GOPMargin wins on accuracy and MCC, illustrating that dataset acoustic variability determines which logit score is best. The paper also finds that softmax GOP over-detects mispronunciations with high recall but low precision, and correlates poorly with human ratings, while a hybrid of margin and softmax scores balances the two families.","pith_inferences":["The dataset dependence suggests a natural next step: a phoneme-specific or uncertainty-weighted combination of margin and max-logit scores may beat any single variant on both classification and human correlation.","Because MPC mispronunciations were simulated by rule-based phoneme substitutions, the reported improvements there should be revalidated on naturally occurring L2 errors, which may be harder to separate.","The gap between classification accuracy and human correlation implies that pronunciation-assessment systems should be evaluated on both; optimizing for classification alone can yield scores that agree with expert perception less well than a slightly weaker classifier."],"forward_implications":["Any CTC-based acoustic model can supply logit-based GOP scores without retraining or added parameters, making the approach fast and model-agnostic.","Logit-based scores separate correct from mispronounced phonemes more cleanly than GOPDNN, which over-detects errors, so they should improve precision in pronunciation-training feedback.","Maximum-logit GOP tracks human phoneme accuracy ratings more closely than probability-based GOP, making it a better target for perceptually meaningful pronunciation scores.","The best logit score is dataset-dependent, so deployment across learner populations should select or combine scores rather than assume one fixed metric."],"supporting_citations":[{"why":"Defines the original GOP score based on likelihood ratios that the logit variants replace.","marker":"[9]"},{"why":"Provides the logit-margin idea that GOPMargin and GOPCombined build on.","marker":"[25]"},{"why":"Defines the DNN GOP from mean softmax posterior probabilities, the probability-based baseline compared throughout.","marker":"[27]"},{"why":"Supplies the CTC segmentation algorithm used for forced alignment of phoneme boundaries.","marker":"[28]"},{"why":"Provides the fine-tuned wav2vec2 CTC phoneme recognizer from which raw logits are extracted.","marker":"[29]"},{"why":"Provides the MPC corpus of Dutch children's English speech with simulated mispronunciations used in the first experiment.","marker":"[30]"},{"why":"Provides SpeechOcean762 with human phoneme accuracy annotations used for the correlation and classification evaluation.","marker":"[31]"},{"why":"Reports the highest prior PCC on SpeechOcean762, the reference point for the paper's correlation results.","marker":"[33]"}],"fun_headline_variants":["Logit GOP beats softmax for mispronunciation detection","Max-logit GOP matches human phoneme ratings best","Hybrid GOP scores balance logit and probability features","Logit-based GOP scores improve mispronunciation detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The MPC results hinge on the assumption that simulated pronunciation errors made by rule-based phoneme substitutions (such as /D/ to /d/ and diphthong simplification) behave like real L2 mispronunciations; if simulated errors are easier or harder to detect, the reported advantages of logit-based scores on MPC may not generalize to natural speech.","fun_headline_variants_meta":{"raw":{"variants":["Logit GOP beats softmax for mispronunciation detection","Max-logit GOP matches human phoneme ratings best","Hybrid GOP scores balance logit and probability features","Logit-based GOP scores improve mispronunciation detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000298,"raw_usage":{"total_tokens":1671,"prompt_tokens":834,"completion_tokens":837,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":773}},"tokens_in":450,"tokens_out":837,"duration_ms":7208,"temperature":1.0,"reasoning_tokens":773,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:39:22.404127+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On an independently annotated corpus of naturally mispronounced L2 speech, compute the Pearson correlations of GOPMaxLogit and GOPDNN with human phoneme ratings; if GOPMaxLogit does not exceed GOPDNN's correlation, the paper's main perceptual-alignment claim would be overturned.","supporting_citations":[{"cited_title":"What do adult english learners say about their pronuncia- tion and linguistic self-confidence?","cited_arxiv_id":null,"evidence_quote":"Defines the original GOP score based on likelihood ratios that the logit variants replace."},{"cited_title":"An improved goodness of pronunciation (gop) measure for pronunci- ation evaluation with dnn-hmm system considering hmm transi- tion probabilities","cited_arxiv_id":null,"evidence_quote":"Provides the logit-margin idea that GOPMargin and GOPCombined build on."},{"cited_title":"Mitigating neural network overconfidence with logit normalization,","cited_arxiv_id":null,"evidence_quote":"Defines the DNN GOP from mean softmax posterior probabilities, the probability-based baseline compared throughout."},{"cited_title":"Comparing non-native and native speech: Are l2 productions more variable?","cited_arxiv_id":null,"evidence_quote":"Supplies the CTC segmentation algorithm used for forced alignment of phoneme boundaries."},{"cited_title":"Perception of speech sounds in school-aged children with speech sound disorders,","cited_arxiv_id":null,"evidence_quote":"Provides the fine-tuned wav2vec2 CTC phoneme recognizer from which raw logits are extracted."},{"cited_title":"On the learning property of lo- gistic and softmax losses for deep neural networks,","cited_arxiv_id":null,"evidence_quote":"Provides the MPC corpus of Dutch children's English speech with simulated mispronunciations used in the first experiment."},{"cited_title":"Logit margin mat- ters: Improving transferable targeted adversarial attack by logit calibration,","cited_arxiv_id":null,"evidence_quote":"Provides SpeechOcean762 with human phoneme accuracy annotations used for the correlation and classification evaluation."},{"cited_title":"A new DNN-based high qual- ity pronunciation evaluation for computer-aided language learn- ing (CALL),","cited_arxiv_id":null,"evidence_quote":"Reports the highest prior PCC on SpeechOcean762, the reference point for the paper's correlation results."}],"review_version":1}