{"id":"9bd40eb1-96bc-435f-85c9-6d42768d65cd","arxiv_id":"2506.19315","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"JCAPT, a Mamba-based joint APA and MDD model with phonological features and think tokens, improves mispronunciation detection and several scoring aspects on speechocean762 over JAM.","lead":"JCAPT combines a Mamba speech encoder, phonological features, and learnable think tokens into one system for grading L2 pronunciation and detecting mispronounced phones. On the public speechocean762 benchmark, it reports higher MDD F1 than earlier joint models, though some of the claimed component gains are not supported by its own ablation tables.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ablation tables contradict the claimed consistent superiority: removing phonological features raises utterance completeness PCC (0.644 vs 0.551) and MDD F1 (52.21 vs 51.05), so the central contribution claim is not established.","rationale":"The reader's stated weakest assumption was direct comparability of the JAM/Joint-CAPT-L1 baselines due to the unreleased pipeline. That is a valid external reproducibility concern. However, the more decisive weakness is internal: the paper's own Tables 2 and 3 refute the sentence 'the full model consistently outperforms all ablated versions' on the very metrics used to motivate the model, especially completeness and MDD F1. This does not automatically invalidate the head-to-head comparison with JAM, but it does invalidate the causal claim that the proposed combination of Mamba, phonological features, and think tokens is what drives the gains. Because the central claim in the abstract and conclusion is specifically about the combined approach yielding consistent improvements, this internal contradiction is the most load-bearing issue. The reader's verdict of CONDITIONAL remains appropriate: the authors can resolve it by releasing per-seed results, running paired significance tests, and revising the overclaims; no rejection is warranted on this evidence alone.","tokens_in":7829,"tokens_out":6278,"duration_ms":67671,"concrete_test":"Perform paired significance testing across the five reported seeds for full JCAPT versus each ablation on every Table 2 and Table 3 metric. If w/o phonological remains significantly better or statistically tied on utterance completeness and MDD F1 after proper paired analysis, the §3.4 and abstract claims must be revised to state that the full combination helps only some aspects, and per-seed values with standard deviations should be reported. If full JCAPT is significantly better after this analysis, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 states that the full JCAPT 'consistently outperforms all ablated versions' in both APA and MDD, but Tables 2 and 3 contradict this. Compared with w/o phonological: utterance completeness PCC is 0.551 vs 0.644, fluency 0.834 vs 0.840, prosody 0.824 vs 0.826, and utterance total 0.806 vs 0.808. For MDD, full F1 is 51.05 vs 52.21 for w/o phonological and 51.92 for w/o both; full recall is 40.23 vs 42.00. Thus the two headline benefits claimed in the abstract, MDD and completeness, are not attributable to the proposed combination; a simpler variant without phonological features does better on both. Since the paper's novelty is precisely the combination of Mamba, phonological features, and think tokens, the central claim that this combination yields a practically better CAPT system is internally unsupported by the reported numbers. The reader's baseline-comparability concern about JAM/Joint-CAPT-L1 results is real and secondary; even if those baselines are valid, the causal attribution in the paper is contradicted by its own ablation evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents JCAPT, a joint model for automatic pronunciation assessment (APA) and mispronunciation detection and diagnosis (MDD). It combines a bidirectional Mamba encoder with canonical phoneme embeddings, phonological attributes, and appended learnable think tokens, then applies multi-level scoring heads. Experiments on speechocean762 report lower MSE and higher PCC than the JAM baseline, a large gain in utterance completeness (PCC 0.551 vs 0.205), and higher MDD F1 (51.05 vs 45.01). The authors conclude that the combination of Mamba, phonological features, and think tokens yields consistent improvements and interpretability.","tokens_in":8094,"tokens_out":3626,"duration_ms":35314,"significance":"If the main results were robust, the MDD F1 improvement and completeness gain would be practically meaningful for CAPT. The paper uses a public benchmark with standard metrics and reports averages over five runs, which is a reasonable evaluation skeleton. However, the central claim of consistent superiority is not supported by the reported numbers: the ablation evidence contradicts the causal attribution, one prosody metric is below the published baseline, and no variance or significance testing is provided. The paper would be significant after the claims are revised and the best configuration is confirmed.","major_comments":[{"comment":"The statement that the full model 'consistently outperforms all ablated versions' is false on the reported numbers. Removing phonological features improves utterance completeness PCC (0.644 vs 0.551), fluency (0.840 vs 0.834), prosody (0.826 vs 0.824), utterance total (0.808 vs 0.806), MDD F1 (52.21 vs 51.05), and MDD recall (42.00 vs 40.23). The paper's headline benefits, MDD and completeness, are thus not attributable to the proposed combination, and the causal claim in §3.3 and the conclusion ('integrating phonological features... yields more accurate' performance) is internally unsupported.","section":"§3.4, Tables 2 and 3"},{"comment":"The claim that JCAPT 'consistently outperforms previous models across all evaluation levels' is contradicted by Table 1: utterance prosody PCC is 0.824 for JCAPT versus 0.829 for JAM. Since prosody is one of the five utterance-level aspects, the correct statement is that JCAPT improves most metrics but not all; the authors should identify which differences are statistically significant.","section":"§3.3, Table 1"},{"comment":"The comparison to JAM and Joint-CAPT-L1 assumes that identical feature extraction, forced alignment, phone sets, data splits, and metric implementations were used. The paper says 'we adopted the same procedures' but provides no code, reruns, or precise versions of the baseline pipelines. Without this, the reported gains could reflect implementation differences rather than the proposed architecture; please release code or rerun the baselines in the same harness.","section":"§3.2, Table 1"},{"comment":"Section 3.2 reports five independent runs but only averages; no standard deviations, confidence intervals, or significance tests are given. This is load-bearing because several central comparisons are small (e.g., phoneme MSE 0.066 vs 0.066, PCC 0.720 vs 0.716) and the ablation 'wins' noted above are not checked for statistical reliability. Please report variance and test the full-vs-ablated and JCAPT-vs-JAM differences.","section":"§3.2"}],"minor_comments":[{"comment":"The caption contains a typo: 'Abalation Studies' should be 'Ablation Studies'.","section":"Table 3 caption"},{"comment":"The word 'Lately' at the start of the third ablation paragraph should be 'Finally' or 'Lastly'.","section":"§3.4"},{"comment":"The notation 'P hnonehot' appears to be a formatting error for the one-hot phoneme vector; please correct it.","section":"§2.1"},{"comment":"The claim that this is the first study to combine phonological attribution, SSM-based modeling, and prompting in CAPT should be softened or qualified, since reference [11] already applies a hierarchical selective SSM to CAPT; the authors should clarify the incremental contribution.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The main concern is internal consistency, not external consensus. I would not recommend rejection, because a corrected manuscript with the best configuration identified, significance testing, and revised claims could be acceptable. The authors should also be asked to share code to resolve baseline comparability. The novelty framing in the introduction is somewhat promotional but can be fixed by precise claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick read of JCAPT. The interesting bit is the assembly: Mamba encoder + phonological attributes + think tokens for joint APA/MDD on speechocean762, and the headline MDD F1 gain over JAM (51.05 vs 45.01) is a real, practically useful number if it holds. They also report phoneme- and word-level gains that look consistent. The evaluation is against a public benchmark, and they did five runs, so the effort is honest enough.\n\nBut the stress-test note is right, and the paper itself gives us the problem: Section 3.4 claims the full model 'consistently outperforms all ablated versions,' and Tables 2 and 3 say otherwise. Removing phonological features actually raises utterance completeness PCC (0.644 vs 0.551 for full), fluency (0.840 vs 0.834), prosody (0.826 vs 0.824), utterance total (0.808 vs 0.806), and MDD F1 (52.21 vs 51.05) with higher recall. Removing think tokens also beats full on completeness and total. So the two headline benefits — MDD and completeness — are not attributable to the proposed combination; a simpler variant does better on both. That is a load-bearing flaw, not a quibble. The architecture novelty is exactly the combination, and the combination does not consistently help.\n\nThe secondary concern about baseline comparability with JAM is also real: they say they 'adopted the same procedures' but give no code or reruns, so I can't fully trust the cross-paper comparison. That said, if the baselines are comparable, the MDD gain is plausible; the internal ablation contradiction is the bigger issue.\n\nWhat's good: the method is clearly described, the dataset is standard, the metrics are appropriate, and the authors reported the numbers that contradict them. That last point almost counts in their favor — the data is on the table. They just need to fix the overclaim, either by re-running ablations or by rewriting the claim to say the full model improves specific aspects but is not uniformly best.\n\nWho is this for? CAPT researchers who want to know whether Mamba + phonological features + think tokens is worth trying. The answer from this paper is 'maybe,' but not for the reasons the authors give. It deserves a serious referee because the evaluation is concrete and the flaw is fixable; I'd send it back for major revision, not desk-reject it. If they address the ablation inconsistency and release code, it becomes a useful incremental contribution.","headline":"Plausible assembly with a real MDD gain, but the paper's own ablation tables contradict its central claim that the full model is consistently better.","tokens_in":8574,"tokens_out":2853,"would_cite":false,"duration_ms":26335,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By fusing phonological attributes and learnable think tokens into a bidirectional Mamba encoder, JCAPT reports gains on both pronunciation scoring and mispronunciation detection on speechocean762: MDD F1 rises from 45.01% to 51.05% and…","keywords":["computer-assisted pronunciation training","automatic pronunciation assessment","mispronunciation detection and diagnosis","Mamba","phonological features","think tokens","multi-task learning","speechocean762"],"falsifier":"Rerun JAM and JCAPT on speechocean762 from the same raw audio, forced alignments, feature versions, and five-seed protocol; if the paired MDD F1 difference is not near the reported 6.04 points, or if reproducing JAM's 45.01% requires a different feature pipeline than the one used for JCAPT, then the outperformance claim is unsupported.","tokens_in":7651,"feed_emoji":"🗣️","tokens_out":9534,"duration_ms":90177,"temperature":0.7,"pith_summary":"This paper is about making computer-assisted pronunciation training (CAPT) systems give better, more detailed feedback to second-language learners. It proposes JCAPT, a single model that handles both automatic pronunciation assessment (APA) and mispronunciation detection and diagnosis (MDD), and claims that combining a bidirectional Mamba encoder with phonological feature embeddings and appended learnable 'think tokens' improves both tasks at once. On the speechocean762 benchmark, the reported gains are an MDD F1 of 51.05% versus 45.01% for the previous joint model JAM, and an utterance-level completeness correlation of 0.551 versus 0.205. The authors take the small ablation drops to mean that fine-grained phoneme-level reasoning, rather than better acoustic features alone, is what drives the improvement.","feed_headline":"Mispronunciation detection F1 jumps to 51 percent on L2 speech benchmark","feed_subtitle":"A joint pronunciation-training model also lifts utterance completeness correlation from 0.205 to 0.551 on speechocean762.","key_machinery":"The machinery is a bidirectional Mamba encoder, i.e., a stack of selective state space model layers that process long phone sequences in linear time. Its input is a fused sequence: phone-level acoustic embeddings built from GOP, wav2vec 2.0, HuBERT, and WavLM features are added to canonical phoneme embeddings that concatenate a one-hot phone vector with a phonological attribute vector. A set of learnable think tokens is appended to the sequence so the encoder can perform extra internal computation before predictions. From the encoder output, aspect-specific attention pooling builds utterance-level representations, and separate heads produce phoneme-, word-, and utterance-level APA scores as well as phoneme-level MDD classifications. The argument for the gains is that the three components are complementary: phonological attributes ground the representations linguistically, think tokens add temporal reasoning capacity, and Mamba supplies efficient long-range context.","core_discovery":"The central claim is that a joint APA and MDD framework built on Mamba can get better results than prior systems by adding two things to a standard phone-level feature pipeline: phonological attribute vectors that describe each canonical phone (such as voicing, manner, and place of articulation), and a set of learnable think tokens appended at the end of the encoder input. The paper reports that on speechocean762 JCAPT reaches phoneme-level MSE 0.066 and PCC 0.720, word-level stress PCC 0.270, utterance-level completeness PCC 0.551, and MDD F1 51.05%, compared with 45.01% for JAM. Ablations removing phonological features, think tokens, or both show the full model is best on most granularities, with think tokens mainly helping MDD recall and phonological features mainly helping phoneme- and word-level assessment. The authors frame this as the first CAPT system to bring together phonological attribution, selective state space modeling, and think-token prompting in one architecture.","pith_inferences":["An untested implication is that think tokens function as a trainable 'deliberation buffer'; one could compare hidden states before and after the appended tokens to see whether they accumulate phone-error information, a probe the paper does not run.","All experiments use read-aloud Mandarin-accented English from one benchmark, so whether the completeness and MDD gains transfer to spontaneous speech, other L1 backgrounds, or other languages is an open empirical question.","The ablations show that removing think tokens slightly improves precision while lowering recall, suggesting a simple confidence threshold or calibration step could push the reported 51.05% F1 higher without any architectural change.","Because the paper fixes the multi-task trade-off weight at 0.3 from the earlier JAM setup, sweeping this weight could reveal how much of the gain comes from the loss balance rather than from the architecture, a comparison the paper does not make."],"forward_implications":["On the paper's numbers, a joint CAPT system can gain more than six F1 points in mispronunciation detection without changing the acoustic front end, by changing the encoder and the symbolic inputs.","An utterance-level aspect that was nearly uncorrelated with human scores (completeness, PCC 0.205 in JAM) reaches 0.551 under JCAPT, so hard global scores are not necessarily limited by the acoustic features.","Since the Mamba encoder scales linearly in sequence length, the same joint recipe is a candidate for longer read-aloud or spontaneous L2 utterances at lower cost than transformer-based baselines.","The ablations point to a complementary split of labor: phonological features mostly lift phoneme- and word-level accuracy, while think tokens mostly lift MDD recall, and removing both degrades results more than removing either alone."],"supporting_citations":[{"why":"JAM is the main baseline and supplies the feature extraction and evaluation procedures JCAPT adopts.","marker":"[10]"},{"why":"Joint-CAPT-L1 is the earlier joint APA/MDD baseline whose numbers JCAPT must beat.","marker":"[9]"},{"why":"Provides the phonological attribute representation fused into the canonical phone embeddings.","marker":"[12]"},{"why":"Introduces the contemplative think-token mechanism that JCAPT adapts for phone-level reasoning.","marker":"[15]"},{"why":"Defines the Mamba selective state space architecture used as the bidirectional encoder.","marker":"[16]"},{"why":"Provides the Dual-Mamba bidirectional design that the encoder is inspired by.","marker":"[25]"},{"why":"Supplies the speechocean762 dataset, human scores, phone transcriptions, and the 39-phone label set.","marker":"[26]"}],"fun_headline_variants":["Mamba joint model improves MDD F1 to 51% on L2 benchmark","Phonological features and think tokens aid MDD in joint CAPT","On speechocean762, joint model hits MDD F1 51.05%","First joint CAPT with Mamba, phonological features, and think tokens","Joint APA-MDD model with Mamba sets new MDD benchmark on speechocean762"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the previously published baseline numbers were computed with the exact same feature extraction, alignment, data splits, and metric definitions as JCAPT, because the paper compares against those numbers without rerunning the baselines itself.","fun_headline_variants_meta":{"raw":{"variants":["Mamba joint model improves MDD F1 to 51% on L2 benchmark","Phonological features and think tokens aid MDD in joint CAPT","On speechocean762, joint model hits MDD F1 51.05%","First joint CAPT with Mamba, phonological features, and think tokens","Joint APA-MDD model with Mamba sets new MDD benchmark on speechocean762"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000706,"raw_usage":{"total_tokens":3152,"prompt_tokens":887,"completion_tokens":2265,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":2158}},"tokens_in":503,"tokens_out":2265,"duration_ms":17200,"temperature":1.0,"reasoning_tokens":2158,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:06:40.297297+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun JAM and JCAPT on speechocean762 from the same raw audio, forced alignments, feature versions, and five-seed protocol; if the paired MDD F1 difference is not near the reported 6.04 points, or if reproducing JAM's 45.01% requires a different feature pipeline than the one used for JCAPT, then the outperformance claim is unsupported.","supporting_citations":[{"cited_title":"An effective pronunciation as- sessment approach leveraging hierarchical transformers and pre- training strategies,","cited_arxiv_id":null,"evidence_quote":"JAM is the main baseline and supplies the feature extraction and evaluation procedures JCAPT adopts."},{"cited_title":"Preserving phonemic distinctions for ordinal regres- sion: A novel loss function for automatic pronunciation assess- ment,","cited_arxiv_id":null,"evidence_quote":"Joint-CAPT-L1 is the earlier joint APA/MDD baseline whose numbers JCAPT must beat."},{"cited_title":"An approach to mispronun- ciation detection and diagnosis with acoustic, phonetic and lin- guistic (apl) embeddings,","cited_arxiv_id":null,"evidence_quote":"Provides the phonological attribute representation fused into the canonical phone embeddings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the contemplative think-token mechanism that JCAPT adapts for phone-level reasoning."},{"cited_title":"Towards Efficient and Multifaceted Computer-assisted Pronunciation Training Leveraging Hierarchical Selective State Space Model and Decoupled Cross-entropy Loss","cited_arxiv_id":"2502.07575","evidence_quote":"Defines the Mamba selective state space architecture used as the bidirectional encoder."},{"cited_title":"Improved mispro- nunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression classifiers,","cited_arxiv_id":null,"evidence_quote":"Provides the Dual-Mamba bidirectional design that the encoder is inspired by."},{"cited_title":"Context-aware Goodness of Pronunciation for Computer-Assisted Pronunciation Training","cited_arxiv_id":"2008.08647","evidence_quote":"Supplies the speechocean762 dataset, human scores, phone transcriptions, and the 39-phone label set."}],"review_version":1}