{"id":"8256fec9-e1e2-4f62-bc35-2826bed68808","arxiv_id":"2507.10985","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper proposes detecting mispronunciations by measuring acoustic distance between a learner's speech and a voice-cloned corrected version, but the evaluation lacks a held-out split and baselines.","lead":"This paper tests whether a voice-cloned version of a learner's speech, generated with corrected pronunciation, can be compared with the original audio to flag mispronounced words. The idea is promising for language-learning apps, but the reported experiments are too weakly validated to support the claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on an unverified premise: that cloning the learner's voice from text corrects pronunciation while preserving voice.","rationale":"The reader's weakest_assumption and my own analysis converge: the clone's pronunciation-correctness is the load-bearing component. I looked for a more basic internal inconsistency, but the paper's pipeline is coherent if that premise holds. I also considered the in-sample/threshold-leakage issue; it is real but secondary, since even a clean held-out evaluation would not rescue the method if the clone reference is not verified. The proposed test directly probes the premise with an independent phonetic transcription of the clone, which is exactly the missing evidence. Since the reader already identified this and rejected the paper on it, my stress test does not change the verdict.","tokens_in":5870,"tokens_out":2570,"duration_ms":31588,"concrete_test":"Take a random sample (e.g., 100 words per speaker) from L2-ARCTIC for the four speakers, including all annotated mispronounced words. Generate ElevenLabs clones of the full utterances, then run forced alignment with an English phoneme recognizer (e.g., Montreal Forced Aligner) on both original and clone audio against canonical dictionary pronunciations. Compute the phoneme error rate of the clone relative to canonical pronunciation and the preservation rate of the original mispronunciations. If the clone is not phonemically correct on the mispronounced target words (e.g., clone phoneme error rate above a few percent, or errors matching the original's substitutions), the central reference assumption fails and the measured deviations cannot be interpreted as mispronunciation signals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"At Sec. 3.1, the clone is assumed to be 'corrected pronunciation,' and Sec. 1 describes it as 'ideal pronunciation conditions,' but no systematic verification of the clone's phonetics is reported. The only evidence is a manual inspection of one word ('Fraud', Sec. 4) where the original is labeled /AW1/ and the clone /AO1/. That single case does not establish that ElevenLabs reliably corrects mispronounced words across speakers, phonemes, and contexts. If the clone preserves the learner's error (possible when the cloned voice is conditioned on the learner's articulation) or introduces its own synthesis artifacts, then the distances in Dcorrect and Dincorrect are dominated by voice-conversion noise, and the threshold/KDE classifier (Eqs. 4-7) has no valid reference signal. This is load-bearing because every reported result, including Table 1, inherits the reference. The evaluation also appears in-sample: thresholds and KDEs are fit on Dcorrect and Dincorrect and then evaluated without a described held-out split, and three of four speakers are below the majority-class accuracy, but the unverified clone-reference assumption is the more fundamental threat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a mispronunciation detection method that compares a learner's original utterance with a voice-cloned version of the same text generated by ElevenLabs, under the assumption that the clone preserves the learner's voice but corrects pronunciation. For each word, the method extracts MFCC envelopes from both audio streams, resamples them to a common length, computes per-coefficient DTW distances, and averages them into a normalized distance. In a calibration phase, these distances are partitioned by ground-truth correctness labels, used to fit class-conditional KDEs (K_COR, K_INC) and to set class-specific 90th-percentile thresholds (tau_C, tau_I). At runtime, a word is classified as CORRECT, INCORRECT, or AMBIGUOUS depending on which KDE is larger and whether the distance falls below or above the selected threshold. Experiments on four L2-ARCTIC speakers report per-speaker accuracies between 0.461 and 0.646 and F1 scores up to 0.684, with a qualitative case study on the word \"Fraud\".","tokens_in":6147,"tokens_out":3050,"duration_ms":37213,"significance":"If the central premise were validated, the approach would be attractive because it offers a personalized reference signal without language-specific phonetic rules or large annotated training sets, and the use of a public corpus (L2-ARCTIC) is a strength. The pipeline is clearly described, and the qualitative example suggests the intended mechanism can operate in at least one instance. However, the significance is currently undermined by two load-bearing gaps: the claim that the cloned audio contains corrected pronunciation is supported only by a single manual inspection, and the reported evaluation does not establish any held-out train/test separation. Moreover, the quantitative results in Table 1 are weak: for NJS the accuracy (0.486) is well below the majority-class baseline (163/253 ≈ 0.644), and for ERMS and MBMPS the accuracy is also at or below the respective majority baselines. These issues prevent the paper from demonstrating the claimed effectiveness.","major_comments":[{"comment":"The evaluation appears to be in-sample. The thresholds tau_C and tau_I in Eq. (4) are computed as the 90th percentiles of D_correct and D_incorrect, and the KDEs K_COR and K_INC in Sec. 3.6 are fit to the same labeled partitions; Algorithm 1 then applies these to each word without describing any train/test split. Section 4 reports only aggregate precision, recall, F1, and accuracy, with no statement that evaluation words or utterances were held out from threshold/KDE fitting. If the reported numbers include training data, they are optimistically biased and cannot support the claim of effective detection. The authors must specify a split (e.g., utterance-level or word-level) and re-report Table 1 on held-out data; ideally, confidence intervals should also be provided given the small per-speaker supports.","section":"Sec. 3.5, 3.6, Algorithm 1, Sec. 4"},{"comment":"The load-bearing assumption that the ElevenLabs clone provides 'ideal pronunciation conditions' or 'corrected pronunciation' is never verified. Section 3.1 simply loads the clones, and the only evidence is the manual inspection of one word ('Fraud') in Section 4. The paper does not report any systematic phonetic transcription, forced alignment, or human listening evaluation of the cloned outputs across the four speakers, 245+ words per speaker, or different phoneme contexts. If the clone preserves the learner's original mispronunciation or introduces synthesis artifacts, then the distances D_correct and D_incorrect no longer reflect pronunciation quality, and the threshold/KDE classifier loses its reference signal. The authors should verify clone pronunciation correctness quantitatively (e.g., by comparing forced-alignment phone labels of the clone against canonical ARPABET transcriptions) and report the agreement rate per speaker.","section":"Sec. 1 and Sec. 3.1"},{"comment":"The reported accuracies are at or below the majority-class baseline for three of four speakers, which directly contradicts the abstract's claim of 'effective' mispronunciation pinpointing. For NJS, accuracy is 0.486 versus a majority baseline of 163/253 ≈ 0.644; for MBMPS, 0.461 versus 154/245 ≈ 0.629; for ERMS, 0.537 versus 133/244 ≈ 0.545. Even if a held-out evaluation were added, the current numbers provide little evidence that the model distinguishes mispronounced words from correct ones beyond always predicting the majority class. The authors should report the majority-class baseline and discuss how their method improves upon it, or temper the central claim accordingly.","section":"Table 1"},{"comment":"The explainability case study is anecdotal. The manual observation that the original 'Fraud' contains /AW1/ and the clone contains /AO1/ is presented as evidence that the clone corrects pronunciation, but this is a single word from a single speaker. The paper does not state how the phonetic labels in Figure 3 were obtained (manual transcription? forced alignment?), nor whether the same correction behavior occurs across the other 26,000+ utterances in L2-ARCTIC or even across the other three speakers in the evaluation. A quantitative comparison of original vs. clone phone error rates, computed with a standard forced aligner, is needed to support the premise.","section":"Sec. 4"}],"minor_comments":[{"comment":"The sentence 'Given the sets of training distances partitioned by prediction correctness D_correct and D_incorrect, we their probability density functions' is missing a verb; it should read 'we estimate their probability density functions.'","section":"Sec. 3.6"},{"comment":"The caption of Figure 4 contains a typo: 'Bribary' should be 'Bribery'.","section":"Fig. 4"},{"comment":"References [2] and [4] appear to be the same paper (Korzekwa et al., 'Computer-assisted pronunciation training—speech synthesis is almost all you need') with different formatting; one duplicate should be removed.","section":"References"},{"comment":"The method is described as 'Multi-Feature Dynamic Time Warping,' but the implementation computes independent 1D DTW distances per MFCC coefficient after resampling; the terminology is somewhat misleading because it is not a true multi-dimensional DTW.","section":"Sec. 3.4"}],"recommendation":"reject","confidential_remarks":"The two main threats—unverified clone correctness and in-sample evaluation—are both load-bearing and would require substantial new experiments (systematic phonetic verification of clones, a proper held-out split, baseline comparisons) rather than local edits. Given the weak quantitative results in Table 1, I do not see a path to acceptance without essentially rerunning the evaluation, which is beyond a minor or even major revision. The paper would be better positioned as a preliminary study if the authors resubmit with corrected methodology."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nShort version: this is a clearly written paper with an appealing idea, but the load-bearing assumption that ElevenLabs cloning a learner's utterance from text produces a corrected, same-voice pronunciation is never validated, and the experiments as described are in-sample with no baselines. I would not trust the numbers, and the approach needs substantial rework before it warrants publication.\n\nWhat is actually new: the specific pairing of a commercial voice-cloning API with DTW-based frame-level deviation to localize mispronunciations is an incremental but useful twist on prior work (Das & Gutierrez-Osuna; Nguyen et al.). The paper honestly cites those, and it does not pretend the broad “compare to a synthetic native-like reference” idea is new. The pipeline is well structured, and the qualitative “Fraud” example is a nice illustration of what the method could do.\n\nThe soft spots are serious. First, the core premise: the clone is assumed to contain the corrected pronunciation. There is no systematic verification—no phonetic transcription of the synthesized audio, no checks across speakers or phonemes. One manually inspected word (“Fraud”) is not evidence that the API reliably fixes mispronunciations. If the clone preserves the learner's error or introduces artifacts, the measured distances mean nothing.\n\nSecond, the evaluation is circular as reported. D_correct and D_incorrect are built using ground-truth labels, and the thresholds and KDEs are fit to those same distances. The Experiments section never states a held-out split, so Table 1 likely overstates performance. Even as reported, three of four speakers are below majority-class accuracy. That is a red flag.\n\nThird, there are no baselines—no comparison to GOP scoring or any existing CAPT method. The claim of effectiveness is unsupported.\n\nWhat is salvageable: the idea of a personalized, voice-preserving reference signal is worth exploring. But the authors need to (1) validate the clone's phonetic correctness, (2) run a proper train/test split with cross-validation, (3) add baselines, and (4) report per-speaker majority-class performance honestly.\n\nWho it is for: someone working on CAPT or accent modification might find the concept worth a skim, but the manuscript currently should not be cited as evidence.\n\nRecommendation: if I were the editor, I would desk-reject this version. The central assumption is unverified, and the experimental results are weak. I would encourage the authors to address the validation and evaluation issues and resubmit.","headline":"Promising reference-signal idea that never verifies its own reference signal; the reported evaluation is in-sample and below trivial baselines for three of four speakers.","tokens_in":6662,"tokens_out":3360,"would_cite":false,"duration_ms":40307,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mispronounced words can be found by comparing a learner's own speech to a voice-cloned version of the same words spoken correctly in the learner's voice.","keywords":["pronunciation analysis","voice cloning","speech processing","language learning","acoustic comparison","mispronunciation detection","dynamic time warping","MFCC features"],"falsifier":"Force-align or automatically transcribe the cloned audio for a word the learner is known to mispronounce, such as the 'fraud' case study: if the clone's transcription contains the same substituted phone as the learner's production (or phones that do not match the canonical form), the reference signal is corrupted and the deviation-based labels cannot be trusted. A cheaper quantitative check: if the distance distributions for correctly pronounced words and mispronounced words become statistically indistinguishable once the clone is generated by a different TTS voice or cloning recipe, the discriminative signal was an artifact of one synthesis pipeline.","tokens_in":5696,"feed_emoji":"🎙️","tokens_out":12820,"duration_ms":136525,"temperature":0.7,"pith_summary":"The paper proposes that a language learner's mispronunciations can be located without phonetic rules or per-language training data by measuring the acoustic distance between the learner's original utterance and a voice-cloned version of the same words spoken correctly in the learner's own voice. The hypothesis is that the regions of maximal acoustic deviation between the original and cloned audio mark the mispronounced segments. The authors test this on annotated non-native English speech, generating clones of each utterance, aligning the real and cloned audio word by word, and computing frame-level distances between their MFCC (mel-frequency cepstral coefficient) feature envelopes. Mispronounced words show systematically larger distances than correctly pronounced words, and a per-word threshold labels speech as correct, incorrect, or ambiguous. If the approach holds, pronunciation feedback becomes personalized and language-agnostic, needing a text-to-speech clone of the learner rather than expert phonetic rules.","feed_headline":"A voice clone of yourself can reveal your mispronounced words","feed_subtitle":"The gap between a learner's speech and a correctly pronounced clone flags the words a tutor should correct.","key_machinery":"The load-bearing mechanism is the pairing of a voice clone with a normalized multi-feature DTW distance. The clone is produced by a commercial text-to-speech service that reproduces the learner's voice while reading the target text with correct pronunciation, giving each learner a personalized 'correct version' of their own utterance. For each aligned word, 13-dimensional MFCC (mel-frequency cepstral coefficient) envelopes are extracted from both versions, the two time series are resampled to a common length, and a dynamic time warping distance (the minimal accumulated cost of aligning two time series) is computed per coefficient and averaged, with normalization by duration and dimensionality. Calibration fits kernel density estimates and 90th-percentile thresholds to the distances of correctly and incorrectly pronounced training words; at runtime each new word's distance $\\bar{d}_j$ selects a class-appropriate threshold, yielding a CORRECT, INCORRECT, or AMBIGUOUS verdict. The whole argument depends on these deviations being interpretable as pronunciation error rather than voice idiosyncrasy or synthesis artifact.","core_discovery":"The paper's central claim is that a voice-cloned, correctly pronounced version of a learner's own utterance can serve as a personalized reference signal: the words whose real and cloned versions diverge most in acoustic feature space are the words the learner mispronounced. Concretely, the authors compute, for each word, a multi-feature dynamic time warping (DTW) distance between the 13-dimensional MFCC envelopes of the original and the synthesized audio, calibrate class-specific thresholds from annotated training distances, and then label unseen words as CORRECT, INCORRECT, or AMBIGUOUS depending on where the distance falls relative to the threshold selected by kernel density classification. Across four non-native speakers, mispronounced words exhibit larger mean distances than correctly pronounced ones, and the model's per-word labels track human-annotated errors; a worked case study on the word 'fraud', where the speaker's vowel is realized as /AW1/ instead of the target /AO1/, shows the kind of first-language interference error the method flags. The authors offer this as a proof of concept that mispronunciation detection can bypass predefined phonetic rule inventories and avoid extensive per-language training data.","pith_inferences":["Since the paper never phonetically transcribes the cloned audio, a direct check of the foundation would be to force-align the clone: if the clone reproduces the learner's substituted phone rather than the canonical one (or adds its own artifacts), the deviation signal would measure something other than pronunciation error.","The per-coefficient DTW distances could double as an error typology: substitution errors should shift the spectral envelope shape across many coefficients, whereas deletions and additions should show up mainly as timing and alignment discontinuities, potentially separating error types without phonetic labels.","Because the reference is the learner's own voice, the same deviation geometry could expose prosodic problems such as misplaced stress or non-native rhythm, which phoneme-inventory methods cannot see; the paper lists this as future work.","If clone correctness is verified, the approach becomes a zero-shot pronunciation checker for under-resourced languages: any language a TTS system can speak and any learner whose voice can be cloned gets a personalized reference without annotated training data."],"forward_implications":["Pronunciation feedback becomes available for any language or dialect that a text-to-speech service can read and any voice it can clone, with no phonetic rule inventory required.","Because the calibration is learned from annotated distances rather than hand-built rules, the same pipeline transfers across learners with different first languages without retraining per language.","The three-way decision rule lets a tutoring system abstain from feedback on words whose acoustic evidence is inconclusive, rather than forcing a guess.","Per-speaker distance distributions differ enough that individualizing the model improves results, pointing toward fully personalized calibration as the next step.","The runtime cost is limited to cloning an utterance, aligning words, and computing frame distances, which the authors identify as the basis for eventual real-time interactive practice."],"supporting_citations":[{"why":"The commercial voice-cloning platform that generates each learner's pronunciation-corrected reference; the method's core premise is that this output is both correctly pronounced and speaker-identical.","marker":"[5]"},{"why":"L2-ARCTIC, the annotated non-native English speech corpus that supplies the word and phone alignments plus the ground-truth mispronunciation labels used to calibrate thresholds and evaluate detection.","marker":"[8]"},{"why":"Prior work showing that synthetic mispronunciations built by phoneme manipulation and voice cloning improve detection accuracy in computer-assisted pronunciation training; the paper's stated key enabler.","marker":"[9]"},{"why":"Das and Gutierrez-Osuna's speech-reconstruction framework, which the paper extends from reconstructed native-like speech to personalized voice clones as the reference.","marker":"[6]"},{"why":"Nguyen et al.'s synthesis of native-accented versions of non-native speech via knowledge distillation and TTS ground truth, the direct predecessor for using a synthetic voice as the reference.","marker":"[7]"},{"why":"The argument that speech synthesis can overcome the scarcity of annotated mispronounced data, the motivation for avoiding per-language training data.","marker":"[4]"},{"why":"The speech-processing reference that supplies the MFCC feature extraction used for the acoustic comparison.","marker":"[11]"},{"why":"Silverman's density estimation text, the basis for the kernel density estimation step that chooses which threshold applies at runtime.","marker":"[12]"}],"fun_headline_variants":["Voice cloning spots your mispronunciations","Clone voice to find your speech errors","Your voice clone reveals mispronounced words","Voice-clone comparison flags pronunciation errors","Clone yourself to hear your errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire pipeline rests on the unverified premise that the cloned audio actually pronounces the target words correctly while preserving the learner's voice; if the clone repeats the learner's mispronunciation or introduces its own artifacts, the measured acoustic deviation points at something other than pronunciation error.","fun_headline_variants_meta":{"raw":{"variants":["Voice cloning spots your mispronunciations","Clone voice to find your speech errors","Your voice clone reveals mispronounced words","Voice-clone comparison flags pronunciation errors","Clone yourself to hear your errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1334,"prompt_tokens":863,"completion_tokens":471,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":409}},"tokens_in":479,"tokens_out":471,"duration_ms":4990,"temperature":1.0,"reasoning_tokens":409,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:19:28.329930+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Force-align or automatically transcribe the cloned audio for a word the learner is known to mispronounce, such as the 'fraud' case study: if the clone's transcription contains the same substituted phone as the learner's production (or phones that do not match the canonical form), the reference signal is corrupted and the deviation-based labels cannot be trusted. A cheaper quantitative check: if the distance distributions for correctly pronounced words and mispronounced words become statistically indistinguishable once the clone is generated by a different TTS voice or cloning recipe, the discriminative signal was an artifact of one synthesis pipeline.","supporting_citations":[{"cited_title":"Elevenlabs text-to-speech api,","cited_arxiv_id":null,"evidence_quote":"The commercial voice-cloning platform that generates each learner's pronunciation-corrected reference; the method's core premise is that this output is both correctly pronounced and speaker-identical."},{"cited_title":"L2-arctic: A non-native english speech corpus,","cited_arxiv_id":null,"evidence_quote":"L2-ARCTIC, the annotated non-native English speech corpus that supplies the word and phone alignments plus the ground-truth mispronunciation labels used to calibrate thresholds and evaluate detection."},{"cited_title":"Capturing accents: An approach to personalize pronunciation training for learners with different l1 backgrounds,","cited_arxiv_id":null,"evidence_quote":"Prior work showing that synthetic mispronunciations built by phoneme manipulation and voice cloning improve detection accuracy in computer-assisted pronunciation training; the paper's stated key enabler."},{"cited_title":"Improving mispronunciation detection using speech reconstruction,","cited_arxiv_id":null,"evidence_quote":"Das and Gutierrez-Osuna's speech-reconstruction framework, which the paper extends from reconstructed native-like speech to personalized voice clones as the reference."},{"cited_title":"Improving pronunciation and accent conversion through knowledge distillation and synthetic ground-truth from native tts,","cited_arxiv_id":null,"evidence_quote":"Nguyen et al.'s synthesis of native-accented versions of non-native speech via knowledge distillation and TTS ground truth, the direct predecessor for using a synthetic voice as the reference."},{"cited_title":"Computer- assisted pronunciation training—speech synthesis is almost all you need,","cited_arxiv_id":null,"evidence_quote":"The argument that speech synthesis can overcome the scarcity of annotated mispronounced data, the motivation for avoiding per-language training data."},{"cited_title":"Bäckström, O","cited_arxiv_id":null,"evidence_quote":"The speech-processing reference that supplies the MFCC feature extraction used for the acoustic comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Silverman's density estimation text, the basis for the kernel density estimation step that chooses which threshold applies at runtime."}],"review_version":1}