{"id":"39a22b94-d558-4afc-b604-c5b704b6ef05","arxiv_id":"2505.24304","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A sequence-to-sequence voice conversion model trained on a native rater's shadowing utterances can spot unintelligible segments in L2 speech, beating an ASR baseline on the native rater but not on all listeners.","lead":"This paper builds a model that predicts which parts of a non-native English speaker's speech a native listener would find hard to understand, using the listener's own shadowing repetitions as training data. The idea is to replace conventional speech-recognition scores, which focus on how close to native pronunciation the speech is, with a measure that tracks what real listeners struggle with.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significance or recall is reported for the only human-judgment comparison; the 2.7-point precision advantage over ASR on a single native rater is too thin to support the central claim.","rationale":"The reader correctly identified the DTW-derived D-labels as a load-bearing component, since they are used for both training and the objective evaluation. My stress-test sharpens the concern: even taking the manual EN-A annotations as the gold standard, the paper's only direct evidence for the central claim is a single precision number with no recall, no confidence interval, and no significance test. The model is trained on the same rater's shadowing, and the DTW pipeline's own precision against EN-A is only 45.9, meaning the model's 43.2 is near the ceiling of what its training target can provide. The comparison with ASR favors the proposed method on EN-A but not on EN-J or EN-O, and with only one native rater the general claim about native judgments is overreaching. These issues do not require rejection, but they make the central claim conditional on additional statistical reporting and ideally on additional native raters. The reader's conditional verdict remains appropriate, so I recommend no change to the verdict.","tokens_in":7865,"tokens_out":10401,"duration_ms":148704,"concrete_test":"Compute word-level recall and F1 for the multi-task and ASR methods against EN-A's manual annotations on the 300 test utterances, and bootstrap (or McNemar) 95% confidence intervals for the precision difference. If the interval includes zero, or recall favors ASR by a margin that cancels the precision gain, the central claim fails. Additionally, report the same statistics for EN-J and EN-O to show the method's advantage is not rater-specific.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Table 2's EN-A column: multi-task precision 43.2 vs. ASR 40.5. This is the only comparison against actual native human annotations; the Table 1 objective F1 uses D-labels generated by the same DTW pipeline (Sec. 2.1) that provided the training targets in Eq. (10), so it cannot independently validate perception alignment. The reported subjective metric is precision only; without recall or F1, a higher precision can simply reflect predicting fewer, safer segments. No confidence intervals, significance tests, or per-utterance variance are reported, and EN-A is the single rater whose shadowing produced the training labels. Moreover, Table 2 shows the DTW pipeline itself has only 45.9 precision against EN-A, so the multi-task model's 43.2 is close to the ceiling of its training signal. If the 2.7-point gap is within sampling noise, or if multi-task recall is substantially lower, the claim that the method 'aligns more closely with native judgments' is not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a perception-based L2 speech intelligibility indicator that uses a native rater's shadowing data within a sequence-to-sequence voice conversion framework. A two-stage reverse-shadowing procedure (Sec. 2.1) generates frame-wise disfluency labels by DTW alignment, and these labels are used both as training targets for a multi-task seq2seq VC model (Sec. 3.2) and as a comparison point in the subjective evaluation. Two methods are presented: an alignment-failure-based method (Sec. 3.1) and a multi-task learning method with disfluency label predictors. Evaluations include word-level F1 against automatic D-labels (Table 1) and precision against manual annotations from three rater groups (Table 2). The central claim is that the proposed approach aligns more closely with native judgments than traditional ASR-based metrics.","tokens_in":8044,"tokens_out":6378,"duration_ms":76049,"significance":"The idea of leveraging a rater's shadowing and a seq2seq VC model to predict perception-based unintelligibility is creative and potentially useful for CALL systems, moving beyond native-likeness measures. The paper provides an ablation study, multiple rater groups, and a public demo, which are strengths. However, the current evidence is insufficient: the only human-judgment comparison that favors the proposed method is precision on a single native rater, with no recall, F1, or significance testing, and the method underperforms the ASR baseline on the other two rater groups. If the evaluation were strengthened, this could be a valuable contribution, but as it stands the central claim is not yet supported.","major_comments":[{"comment":"The subjective evaluation reports only precision. A method can achieve high precision by predicting very few segments, so precision alone is insufficient to support the claim of closer alignment with native judgments. Please report recall and F1 for each method and rater, together with confidence intervals or significance tests. The current numbers show that the multi-task model is worse than the ASR baseline on EN-J (28.1 vs 32.8) and EN-O (31.7 vs 37.6), so the claimed advantage is at best rater-dependent and not statistically demonstrated.","section":"Sec. 5.2, Table 2"},{"comment":"The row labeled \"Ground Truth\" is not a human ground truth but the 2-stage shadowing + DTW pipeline described in Sec. 2.1, which is the same procedure used to generate the D-labels that train the multi-task model (Eq. 10). The multi-task model's precision on EN-A (43.2) is close to the precision of its own training-label generator (45.9), and EN-A is the rater whose shadowing produced those training labels. Consequently, the EN-A comparison does not independently validate perception alignment; the EN-J and EN-O results, which come from raters not involved in label generation, show the multi-task model underperforming the ASR baseline. This undermines the abstract's general claim that the method aligns more closely with native judgments.","section":"Sec. 5.2, Table 2"},{"comment":"The precision computation lacks a defined matching criterion between predicted and manually annotated unintelligible segments. Without specifying the overlap rule (e.g., frame-level intersection-over-union, word-level match with tolerance, or a required temporal overlap), the reported precision values are not reproducible and the comparison across methods is not well defined. Please provide the exact matching procedure used to compute the precision scores in Table 2.","section":"Sec. 5.2"},{"comment":"The objective evaluation uses D-labels as ground truth, which are generated by the same Sec. 2.1 procedure that provides the training targets for the multi-task model. This evaluation therefore measures agreement with the automatic labeling procedure rather than with human perception, and the 30.3 vs 28.0 word-level F1 difference is not tested for significance. The central claim should be based on the perception-based evaluation, or the objective result should be framed only as a proxy agreement with the automatic labeler.","section":"Sec. 5.1, Table 1"}],"minor_comments":[{"comment":"The threshold tau in Eq. (3) is never specified, and no sensitivity analysis is provided. Please report the chosen value and, ideally, show how the alignment-based method's performance varies with tau.","section":"Sec. 3.1, Eq. (3)"},{"comment":"The loss weight lambda is set to 10 without any ablation. A brief study of lambda would strengthen the multi-task learning results.","section":"Sec. 3.2, Eq. (10)"},{"comment":"The feature selection (HuBERT 9th layer for source, PPG-BNF for target) is justified only by the statement that alignment did not converge with HuBERT on both sides. Please give more detail or a quantitative comparison to support this choice.","section":"Sec. 4.2"},{"comment":"The phrase \"global, multilingual contexts\" overstates the experimental scope, which covers only Japanese learners of English and a small set of raters. Consider tempering this claim.","section":"Abstract and Conclusion"},{"comment":"There are minor typographical issues: \"Session\" should be \"Section\" in Sec. 5.1 and Sec. 5.2, and the phrase \"detailed in Session 3.2\" should be corrected.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely intended for a speech conference and presents a novel application, but the evidence for the central claim is currently weak. The most serious issue is the conflation of the DTW-based labeler with ground truth in Table 2, combined with the lack of recall/F1 and significance testing. If the authors can provide a more complete evaluation and temper the claims, a revised version could be publishable. I would encourage the editor to request a revision rather than reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The interesting thing here is the combination: using a native rater's reverse shadowing to generate fine-grained disfluency labels, then training a seq2seq voice conversion model with a multi-task disfluency detection head to predict unintelligible segments. That specific pipeline is new, and the authors are honest that the alignment-breakdown variant underperforms. The multi-task model with all three losses does reach a word-level F1 of 30.3 vs 28.0 for an ASR baseline on the objective D-label task, and the ablation clearly shows each loss helps.\n\nThe problem is the central claim: \"aligns more closely with native judgments than traditional ASR.\" In the subjective evaluation against actual human annotations, the multi-task model beats ASR only for EN-A, the American rater whose shadowing produced the training labels, and loses to ASR for EN-J and EN-O. That 43.2 vs 40.5 gap is 2.7 points of precision on a single rater, with no confidence intervals or significance tests. Precision alone is also tricky: a model can inflate precision by predicting fewer, safer segments, and recall isn't reported for this comparison.\n\nThere's also a circularity concern. The D-labels used for training and the objective evaluation come from the same two-stage shadowing + DTW pipeline that is listed as \"Ground Truth\" in Table 2. The pipeline itself only reaches 45.9 precision against EN-A, so the model's 43.2 is essentially learning to approximate the pipeline, not human perception independently. The manual annotation was done later and by the same rater, which helps a bit, but the evaluation is not fully independent.\n\nThese are addressable: report recall and F1 for Table 2, compute significance across utterances and raters, and frame the claim as \"closer to one native rater's judgments\" rather than \"human judgments\" broadly. The idea is worth pursuing, and the dataset is a real asset.\n\nWho should read this: people working on CALL or shadowing-based intelligibility assessment. It's a solid workshop-level contribution, not a definitive solution. A serious referee could push the authors to fix the evaluation, so I'd send it to review rather than desk-reject, but I'd expect major revision.","headline":"Clever idea, thin evidence: the multi-task VC model beats ASR on only one of three human rater groups, and the gain is 2.7 points of precision with no significance test.","tokens_in":8629,"tokens_out":2692,"would_cite":true,"duration_ms":31339,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a multi-task sequence-to-sequence voice-conversion model, trained on a native rater's shadowing of L2 speech, identifies unintelligible segments more closely matching that rater's judgment than an ASR-based metric…","keywords":["L2 speech intelligibility","shadowing","sequence-to-sequence voice conversion","disfluency detection","computer-assisted language learning","multi-task learning","ASR-based intelligibility metrics","reverse shadowing"],"falsifier":"Re-run the subjective annotation experiment with several native English raters who were not involved in generating the shadowing labels, and compare mean precision; the central claim fails if the multi-task model's mean precision does not exceed the ASR baseline's, since the paper's reported advantage rests on a single native rater (EN-A).","tokens_in":7622,"feed_emoji":"🗣️","tokens_out":7288,"duration_ms":77504,"temperature":0.7,"pith_summary":"The paper tries to establish that a learner's intelligibility to a native listener can be predicted by training a voice-conversion model on the native listener's shadowing of that learner's speech, rather than by measuring how close the learner's pronunciation is to a native accent. The proposed multi-task sequence-to-sequence model converts the learner's reading into the rater's shadowing while jointly detecting disfluent frames, and the paper reports that it identifies unintelligible words more accurately than an ASR-based baseline in a word-level F1 comparison. In a subjective annotation experiment, the model's predicted unintelligible segments match the native rater's manual annotations with higher precision than the ASR baseline does, although the same is not true for the two non-native annotator groups. A reader should care because this points toward feedback that reflects whether a real listener is likely to understand the learner, not whether the learner sounds native.","feed_headline":"Shadowing-trained model beats ASR at spotting unintelligible words","feed_subtitle":"The multi-task model hits F1 30.3 vs 28.0 and precision 43.2 vs 40.5 on the native rater.","key_machinery":"The central machinery is the two-stage reverse shadowing label and its embedding into a sequence-to-sequence voice converter. First, dynamic time warping between the rater's unscripted shadowing and script-shadowing yields a frame-level disfluency label via edit-distance thresholding, and a second alignment maps that label onto the learner's reading. The model then treats the learner's speech as source and the rater's shadowing as target, using a monotonic-alignment-search-based voice converter with a length regulator, and adds two 5-layer CNN disfluency-label predictors, one after the encoder and one after the decoder, with focal loss. An inverse length regulation projects the predicted disfluency labels back to the source side, so the model learns to point at the learner-side frames that caused the rater's breakdown.","core_discovery":"On the paper's own terms, the discovery is that a perception-based intelligibility indicator can be encoded as a voice-conversion task: the model is asked to convert L2 read speech into the native rater's first shadowing, and the mismatches and disfluencies the model must handle reveal where comprehension breaks down. The full multi-task model, trained with both reconstruction losses and two CNN-based disfluency predictors driven by focal loss, reaches a word-level F1 of 30.3 against 28.0 for an ASR-based mispronunciation baseline on the same test set. In subjective evaluation with manual annotations, it reaches precision 43.2 against the native rater EN-A, above the ASR baseline's 40.5; against the non-native annotator groups EN-J and EN-O the ASR baseline scores higher. The paper interprets this as evidence that the method captures the specific native rater's auditory perception better than ASR does, while remaining less aligned with how other listeners perceive the same utterances.","pith_inferences":["A testable extension the paper leaves implicit is whether training on multiple raters' shadowings would produce a consensus intelligibility map that outperforms ASR for all listener groups, not only for the shadowing rater.","The rater-specific nature of the result suggests a CALL deployment could let learners choose the listener profile they want to be understood by, such as a native or another non-native speaker, and receive feedback optimized for that audience.","The alignment-failure variant underperforms on precision, which suggests a broader design principle: letting the alignment module fail and reading off the failures is less effective than explicitly supervising disfluency detection through auxiliary losses.","One consequence worth testing is whether the 45.9 precision of the DTW-derived label itself is close to human inter-rater agreement; if it is, the label-generation step may already be near the practical ceiling for this task."],"forward_implications":["If the method is right, CALL systems can point learners to the specific words or frames a listener is likely to miss, instead of giving only a holistic score.","If the method is right, feedback can be tied to a particular rater's perceptual profile, because the model trained on one rater's shadowing aligns better with that rater's annotations.","If the method is right, the technique could generalize to other languages and accents, since voice conversion and shadowing do not require pronunciation-error lexicons or expert phonetic annotation.","If the method is right, the sequence-to-sequence framework gives a single model that both reconstructs the rater's shadowing and emits disfluency labels, combining perceptual simulation with interpretable localization."],"supporting_citations":[{"why":"Supplies the two-stage reverse shadowing procedure whose DTW-derived labels are the training targets and evaluation reference for the proposed model.","marker":"[7]"},{"why":"Provides the multi-granularity annotation method and the dataset construction used to generate frame-level and word-level disfluency labels.","marker":"[8]"},{"why":"Supplies the automatic alignment search based non-autoregressive voice conversion architecture that the proposed model adapts for alignment and length regulation.","marker":"[13]"},{"why":"The prior pilot study that first applies seq2seq voice conversion to L2 speech shadowing and informs the feature embedding choices for source and target.","marker":"[20]"},{"why":"Introduces focal loss, used in the disfluency label predictors to handle the class imbalance of disfluent frames.","marker":"[23]"},{"why":"Provides the PPG-like bottleneck feature used as the target-side acoustic representation for stable alignment.","marker":"[24]"},{"why":"Supplies the LibriSpeech-trained ASR system used as the baseline for mispronunciation-based intelligibility detection.","marker":"[25]"}],"fun_headline_variants":["Mimicking a native's shadowing spots L2 comprehension gaps","Perception-based model outdoes ASR on L2 intelligibility","Voice conversion turns shadowing into a comprehension probe","Shadowing data helps model find L2 words that confuse listeners"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on the assumption that the two-stage shadowing pipeline with dynamic time warping produces accurate frame-level disfluency labels on the learners' recordings; those DTW-derived labels are used both as training targets for the proposed model and as the ground-truth reference in part of the evaluation, yet the pipeline itself achieves only 45.9 precision against the native rater's manual annotations.","fun_headline_variants_meta":{"raw":{"variants":["Mimicking a native's shadowing spots L2 comprehension gaps","Perception-based model outdoes ASR on L2 intelligibility","Voice conversion turns shadowing into a comprehension probe","Shadowing data helps model find L2 words that confuse listeners"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1503,"prompt_tokens":902,"completion_tokens":601,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":531}},"tokens_in":518,"tokens_out":601,"duration_ms":7391,"temperature":1.0,"reasoning_tokens":531,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:25:24.362775+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the subjective annotation experiment with several native English raters who were not involved in generating the shadowing labels, and compare mean precision; the central claim fails if the multi-task model's mean precision does not exceed the ASR baseline's, since the paper's reported advantage rests on a single native rater (EN-A).","supporting_citations":[{"cited_title":"Dataset We utilized the shadowing dataset described in [8, 9]","cited_arxiv_id":null,"evidence_quote":"Supplies the two-stage reverse shadowing procedure whose DTW-derived labels are the training targets and evaluation reference for the proposed model."},{"cited_title":"Objective evaluation We present our objective evaluation results in Table 1","cited_arxiv_id":null,"evidence_quote":"Provides the multi-granularity annotation method and the dataset construction used to generate frame-level and word-level disfluency labels."},{"cited_title":"Accent, intelligibility, and com- prehensibility: Evidence from four l1s,","cited_arxiv_id":null,"evidence_quote":"Supplies the automatic alignment search based non-autoregressive voice conversion architecture that the proposed model adapts for alignment and length regulation."},{"cited_title":"Non- autoregressive sequence-to-sequence voice conversion,","cited_arxiv_id":null,"evidence_quote":"The prior pilot study that first applies seq2seq voice conversion to L2 speech shadowing and informs the feature embedding choices for source and target."},{"cited_title":"wav2vec 2.0: a framework for self-supervised learning of speech representa- tions,","cited_arxiv_id":null,"evidence_quote":"Introduces focal loss, used in the disfluency label predictors to handle the class imbalance of disfluent frames."},{"cited_title":"Wavlm: Large-scale self- supervised pre-training for full stack speech processing,","cited_arxiv_id":null,"evidence_quote":"Supplies the LibriSpeech-trained ASR system used as the baseline for mispronunciation-based intelligibility detection."}],"review_version":1}