{"id":"2213a186-2e2f-42a1-8d37-6bcf22c0b17b","arxiv_id":"2509.02915","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LoRA fine-tuning of the Phi-4 multimodal LLM on Speechocean762 yields pronunciation scores and phoneme-level transcripts from a single model, with PCC up to 0.74 for accuracy and PER down to 0.11.","lead":"This paper shows that a multimodal large language model, fine-tuned with LoRA, can score pronunciation and detect mispronunciations in one model. The approach uses a single training run on a standard dataset and avoids the separate-task architectures typical of prior systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MDD evaluation conflates ASR with mispronunciation detection; the reported F1/PER do not validate the claimed simultaneous MDD capability.","rationale":"The reader's weakest assumption focuses on whether the pretrained audio encoder/projector already encode L2-relevant features, leaving open the possibility that LoRA-only tuning might not suffice. But the paper's own experiments show LoRA-only tuning does reach the reported performance, so that premise is not demonstrated to fail. A more fundamental issue is that the MDD task is operationalized as pure speech recognition. The paper's evaluation section defines MDD metrics (WER, PER, F1) on the basis of transcriptions, not on mispronunciation labels. Speechocean762 provides phone-level mispronunciation annotations, yet the paper never uses them to compute detection accuracy; it only compares recognized phonemes to reference phoneme sequences. This conflates ASR with MDD. If the model cannot compare its output to the target text, it cannot diagnose which phonemes are mispronounced. Therefore the central claim of performing both APA and MDD simultaneously is unsupported for the MDD component, regardless of whether LoRA tuning works. This is a concrete, checkable flaw rather than a speculative concern about pretrained features. The appropriate verdict remains CONDITIONAL (the paper can be corrected by adding a proper MDD evaluation), so the reader's verdict is unchanged, but the reason for conditionality shifts.","tokens_in":10496,"tokens_out":5000,"duration_ms":55578,"concrete_test":"Re-evaluate the model using the official Speechocean762 mispronunciation evaluation protocol: force-align each model phoneme output to the target sentence, map predicted phones to the 46-unit L2 phone set, and compute mispronunciation detection F1 against the dataset's phone-level mispronunciation labels (as done in Ryu et al., Interspeech 2023). If the F1 is near the no-training baseline or substantially lower than the reported PER-derived F1, the MDD claim fails; if it is comparable to prior MDD systems, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the model performs MDD is not actually supported. In Section 4.3, MDD is evaluated using WER, PER, and F1 computed by comparing the model's word/phoneme transcriptions to reference transcriptions from Speechocean762. However, the model is prompted only to 'Transcribe the audio utterance' (Appendix 7.2), with no target sentence or mispronunciation labels provided. The metrics therefore measure ASR accuracy at the word and phoneme level, not the detection/diagnosis of mispronunciations relative to an intended utterance. The F1-score for 'mispronunciation detection' is not defined against the dataset's mispronunciation annotations, and no alignment or scoring protocol is described. Consequently, Table 3's MDD results (e.g., PER 0.114, F1 0.724) are compatible with the model simply being a good phoneme recognizer, not a mispronunciation detector. Since the abstract and title emphasize simultaneous APA and MDD, the load-bearing support for the MDD half of the claim is missing.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a unified automatic pronunciation assessment (APA) and mispronunciation detection and diagnosis (MDD) system by LoRA fine-tuning the multimodal LLM Phi-4-multimodal-instruct on the Speechocean762 dataset. The model is trained with task-specific control tokens and prompts to output four sentence-level pronunciation scores (accuracy, fluency, prosodic, total) as well as word-level and phoneme-level transcriptions. The authors compare two fine-tuning strategies: LoRA-only and unfreezing the audio encoder/projector. They report Pearson correlation coefficients up to 0.733 for fluency, WER down to 0.139, PER down to 0.114, and F1 up to 0.724, concluding that LoRA-only tuning is comparable to audio-layer unfreezing. They also report a moderate negative correlation between PER and accuracy scores.","tokens_in":10769,"tokens_out":4699,"duration_ms":52933,"significance":"If the MDD claim were properly validated, the contribution would be a simple, resource-efficient recipe for building a unified CAPT system from an MLLM, and the LoRA-only versus unfreezing comparison would be a useful practical datapoint. The APA results are plausible and competitive on several dimensions, and the paper is honest about its computational limitations. However, the MDD evaluation as presented does not measure mispronunciation detection or diagnosis: the model is prompted to transcribe, not to compare against a target sentence, and no alignment or scoring protocol against the Speechocean762 mispronunciation annotations is described. The abstract also overstates the correlation results. The central claim of simultaneous APA and MDD is therefore not yet supported; the paper's strongest defensible contribution is an APA-scoring and general ASR experiment on a public benchmark.","major_comments":[{"comment":"The MDD evaluation does not validate the claimed MDD capability. The MDD prompt is 'Transcribe the audio utterance, providing both a word-level transcript and phoneme-level breakdown.' No target sentence or canonical phoneme sequence is provided, and no alignment or scoring protocol is described. The F1 formulas in (3)-(5) define TP/FP in terms of 'correctly identified mispronunciations,' but the paper never explains how a mispronunciation is identified from an unconstrained transcription against the dataset's reference phoneme annotations. Consequently, the reported WER, PER, and F1 in Table 3 are measures of ASR/phoneme-recognition performance, not of mispronunciation detection or diagnosis. The MDD half of the central claim is therefore unsupported.","section":"§4.3 and Appendix 7.2"},{"comment":"The abstract's statement 'PCC > 0.7' is not supported by Table 3 for the LoRA model as a whole: Table 3 gives LoRA total PCC 0.668, accuracy 0.645, and only fluency and prosodic PCC above 0.7 at some epochs. Similarly, §4.4's claim that 'The LoRA approach consistently yielded superior results compared to the Unfreeze strategy across all metrics' is contradicted by Table 3: at epoch 4 Unfreeze accuracy is 0.743 versus LoRA 0.645, and Unfreeze WER is 0.142 versus LoRA 0.148. The data support 'comparable' or 'mixed,' not 'consistently superior.'","section":"Abstract and §4.4"},{"comment":"The paper does not describe a validation-based epoch selection procedure. Table 3 reports all epochs, but §4.4 and Table 4 compare 'four epochs' (LoRA and Unfreeze) even though the best LoRA APA/MDD numbers occur at epochs 2-3. If hyperparameters or epochs are selected using the test set, the reported comparisons are biased; if not, the selection rule should be stated. This matters for the central LoRA-versus-unfreeze comparison and for the comparison with prior work in Table 4.","section":"§4.2-§4.4 and Table 4"}],"minor_comments":[{"comment":"The caption states 'All p-values for APA tasks are under 0.05, except for underlined values,' but the table does not display p-values or underlining. Either include them or remove the claim.","section":"Table 3 caption"},{"comment":"The training description gives batch size 8 and gradient accumulation step 8, but not the effective batch size, number of training steps, warmup, or learning-rate schedule. Please add these details for reproducibility.","section":"§4.2"},{"comment":"The text says Phi-4 'incorporates a Mixture-of-LoRAs structure,' but the proposed method then fine-tunes LoRA adapters. Clarify whether the authors add new LoRA adapters on top of the model's existing Mixture-of-LoRAs or replace/reuse them.","section":"§2.4"},{"comment":"The exclusion of the 'completeness' metric because all test values are 10 is plausible, but the sentence should also state whether the same was true for the training set, since excluding a metric from training on this basis is unusual.","section":"§4.1"},{"comment":"The label 'prosodic' in the appendix and 'prosody' in Table 2 are used inconsistently. Please unify the terminology.","section":"Table 2 and Appendix 7.1"},{"comment":"The paper uses 'simultaneously' for APA and MDD, but the two tasks are triggered by different control tokens and prompts. Clarify whether 'simultaneous' means a single shared model/training run rather than parallel generation in one forward pass.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core APA experiment is reasonable, and the LoRA-only versus unfreezing comparison is a useful empirical datapoint, though the claim of consistent superiority is overstated. The load-bearing problem is the MDD evaluation: as written, the paper does not measure mispronunciation detection at all. The authors should either add a proper MDD evaluation using the Speechocean762 mispronunciation annotations and a target-sentence-conditioned protocol, or revise the title/abstract/claims to remove MDD and present the work as APA plus speech recognition. If the latter, the relevance to a CAPT journal should be reconsidered. I would also encourage them to release code and model checkpoints for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is an honest, straightforward fine-tuning study showing that LoRA-only adaptation of Phi-4-multimodal gives reasonable pronunciation-scoring correlations on Speechocean762, and that unfreezing the audio encoder doesn't help. The part that doesn't hold up is the claim that the same setup also does MDD. The evaluation never feeds the model the target sentence, so the WER/PER/F1 numbers are ASR/phone-recognition metrics, not mispronunciation detection.\n\nWhat's genuinely useful: the control-token setup for APA vs MDD in one model, the direct comparison of LoRA-only vs unfreezing audio layers (the null result is informative), and use of the standard train/test split. The authors also deserve credit for the limitation section; they admit no full fine-tuning and no suprasegmental modeling.\n\nSoft spots, in order. First, the MDD evaluation in Section 4.3 is the big one. Speechocean762 has mispronunciation annotations, but the model is prompted to 'transcribe the audio utterance' without the intended sentence. That is ASR, not MDD. The precision/recall/F1 are never defined against the dataset's labels; the formulas define TP/FP conceptually but no alignment or scoring protocol is given. So Table 3's F1 of 0.724 is plausibly just a phoneme recognition number. The abstract's 'simultaneous APA and MDD' claim rests on this, which means the central claim is overstated. Second, the abstract says 'PCC > 0.7' but accuracy and total PCC for LoRA are below 0.7 (0.54-0.68). The strong correlation holds only for fluency/prosodic at the best epoch, not all metrics. Minor but sloppy. Third, Table 4 compares both models at 4 epochs even though LoRA peaks at epoch 3; that slightly biases the SOTA comparison in favor of Unfreeze for accuracy. They should compare best epochs or report all. Fourth, no code, no uncertainty bars, and apparently single runs. Common for this type of paper, but noted. The correlation analysis in Figure 5 is descriptive and weak, but clearly labeled as such, so fine.\n\nBottom line: the APA half is evidence, the MDD half isn't. The paper deserves a serious referee because the core question is worth asking and the fix is straightforward: redo MDD evaluation with target sentences and the dataset's mispronunciation labels. Without that, this is a solid demo of pronunciation scoring, not a joint APA+MDD system. It is the kind of paper that would benefit from peer review rather than desk rejection; I would send it out and require the MDD evaluation to be reworked or the claim softened.","headline":"Useful fine-tuning comparison for pronunciation scoring, but the MDD claim is really just ASR; the paper needs a proper MDD evaluation before the joint claim holds.","tokens_in":11222,"tokens_out":3240,"would_cite":false,"duration_ms":35719,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a Multimodal Large Language Model adapted only through LoRA can, in a single training pass, both score pronunciation and detect mispronunciations, reaching PCC above 0.7 and phoneme error rates below 15 percent on Spe","keywords":["automatic pronunciation assessment","mispronunciation detection and diagnosis","multimodal large language model","low-rank adaptation","computer-assisted pronunciation training","Speechocean762","phoneme error rate","English L2 learners"],"falsifier":"Run the same LoRA fine-tuning recipe on an out-of-domain L2 corpus, such as Korean- or Spanish-accented English, and check whether the accuracy-score Pearson correlation stays above 0.7; if it drops toward zero, the claimed transferability to general L2 learners is contradicted. Alternatively, replacing the pretrained audio encoder with a randomly initialized encoder and repeating the recipe would test whether the pretrained speech features are load-bearing.","tokens_in":10425,"feed_emoji":"🗣️","tokens_out":7011,"duration_ms":70586,"temperature":0.7,"pith_summary":"This paper tries to establish that a single pretrained multimodal large language model, adapted with low-rank (LoRA) fine-tuning, can do both jobs of computer-assisted pronunciation training at once: scoring how native-like an utterance is, and transcribing what was actually said at word and phoneme level. The authors show this on the Speechocean762 corpus of Mandarin-accented English, using prompts and task-control tokens instead of task-specific model heads, and report score correlations with human raters above 0.7 in several dimensions and word/phoneme error rates below 0.15. They also report that fine-tuning only the LoRA adapter matches or beats additionally unfreezing the audio encoder and projector, which matters because it makes the method cheap and accessible. If true, the result removes the need for the separate datasets, architectures, and training pipelines previously used for joint APA and MDD systems.","feed_headline":"One LoRA-tuned speech model scores pronunciation and catches errors in one pass","feed_subtitle":"Fine-tuning only the adapter yields scores that track human raters and phoneme errors below 15 percent.","key_machinery":"The carrying mechanism is Low-Rank Adaptation (LoRA) applied to Phi-4-multimodal-instruct, a pretrained speech-language MLLM that already contains an audio encoder, an audio projector, and a mixture-of-LoRAs structure. Only the low-rank adapter weights (or, in one variant, also the audio encoder and projector) are updated during supervised fine-tuning, while the large language model stays frozen. Two control tokens, <|APA|> and <|MDD|>, are prepended to prompts to switch the same model between scoring and transcription during inference, and both outputs are produced in the pretrained model's token vocabulary (including ARPABET phoneme symbols), so no new output heads or phoneme tokens are ne","core_discovery":"The central claim is that pronunciation assessment does not need a bespoke joint architecture: the pretrained Phi-4-multimodal-instruct model, carrying an audio encoder, projector, and LoRA adapter attached to a language model, can be taught both APA and MDD from a single dataset using only supervised fine-tuning of the adapter (or optionally the audio layers). With the Speechocean762 training set, the model predicts four pronunciation scores (accuracy, fluency, prosodic, total) in JSON and outputs orthographic and phoneme transcriptions in the same pass, with control tokens <|APA|> and <|MDD|> selecting the task. The best LoRA run produces Pearson correlations up to 0.733 with human scores","pith_inferences":["A testable extension of the paper's setup is to apply the same LoRA-trained adapter to other accented-English corpora (for example, Korean or Spanish L1) with no additional training; if the pretrained audio encoder is as transferable as assumed, accuracy-score correlation should remain in the same range.","The control-token mechanism could generalize to other CAPT subskills, such as stress, intonation, or rhythm scoring, by defining new tokens and rubrics instead of adding new output heads; the paper does not implement this.","The reported negative correlation between predicted accuracy scores and phoneme error rate could be operationalized as an explainability signal: a deployment could generate learner feedback by diffing the model's phoneme transcript against the target, something the paper leaves implicit.","The LoRA-only versus unfreeze comparison covers only four epochs, so the relative advantage of LoRA-only on MDD metrics is specific to that training budget; longer unfreezing is an open experimental question."],"forward_implications":["A single model can serve both APA and MDD in one inference pass, so a CAPT system can output a pronunciation score and a phoneme-level transcript from the same audio and prompt.","LoRA-only tuning reaches comparable or better MDD metrics than unfreezing the audio encoder and projector, so users with one GPU can adapt a large MLLM without full fine-tuning.","Because ARPABET symbols are ordinary alphabetic tokens, phoneme-level transcription can be handled without training or adding new phoneme tokens.","The negative correlation between predicted accuracy scores and phoneme error rate supports the interpretation that the scored output is tied to actual mispronunciation detection, not just dataset statistics.","The same prompt-and-control-token recipe should transfer to other learner populations and languages using the model's multilingual capabilities, though the paper only tests Mandarin-L1 English speakers."],"supporting_citations":[{"why":"Supplies the pretrained Phi-4-multimodal-instruct architecture (audio encoder, projector, mixture-of-LoRAs) whose LoRA-only adaptation is the method.","marker":"[1]"},{"why":"Provides all training and test data: human pronunciation scores and phoneme-level mispronunciation annotations for Mandarin-accented English.","marker":"[25]"},{"why":"Defines the joint APA+MDD multi-task learning baseline that the paper's unified MLLM approach is designed to replace; used for comparison.","marker":"[20]"},{"why":"Supplies the transformer/GOP-based multi-aspect pronunciation assessment baseline compared in Table 4.","marker":"[13]"},{"why":"Supplies a prior multimodal-LLM pronunciation assessment system and its reported PCC numbers; comparison target.","marker":"[11]"},{"why":"Provides GPT-4o zero-shot and Azure PA comparison numbers for pronunciation scoring.","marker":"[21]"},{"why":"Motivates the unfreeze experiment by suggesting that updating all audio layers may help; official documentation.","marker":"[17]"},{"why":"Defines the CMUDict/ARPABET phone set used as the phoneme transcription target in the MDD task.","marker":"[22]"}],"fun_headline_variants":["LoRA-only tuning nails pronunciation scoring and error detection","Single adapter, dual task: pronunciation scores and mispronunciation spotting","Lightweight LoRA replaces complex joint training for pronunciation assessment","One adapter does it all: pronunciation scoring and error diagnosis","Minimal fine-tuning yields robust pronunciation evaluation"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The method assumes that the audio-processing parts of the pretrained model already understand accented non-native speech well enough that updating only a small adapter is sufficient to reach usable pronunciation scores; the paper does not validate this premise independently on accented speech.","fun_headline_variants_meta":{"raw":{"variants":["LoRA-only tuning nails pronunciation scoring and error detection","Single adapter, dual task: pronunciation scores and mispronunciation spotting","Lightweight LoRA replaces complex joint training for pronunciation assessment","One adapter does it all: pronunciation scoring and error diagnosis","Minimal fine-tuning yields robust pronunciation evaluation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":1068,"prompt_tokens":768,"completion_tokens":300,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":221}},"tokens_in":512,"tokens_out":300,"duration_ms":4040,"temperature":1.0,"reasoning_tokens":221,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:15:25.969747+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same LoRA fine-tuning recipe on an out-of-domain L2 corpus, such as Korean- or Spanish-accented English, and check whether the accuracy-score Pearson correlation stays above 0.7; if it drops toward zero, the claimed transferability to general L2 learners is contradicted. Alternatively, replacing the pretrained audio encoder with a randomly initialized encoder and repeating the recipe would test whether the pretrained speech features are load-bearing.","supporting_citations":[{"cited_title":"Abouelenin, A","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained Phi-4-multimodal-instruct architecture (audio encoder, projector, mixture-of-LoRAs) whose LoRA-only adaptation is the method."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the joint APA+MDD multi-task learning baseline that the paper's unified MLLM approach is designed to replace; used for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the transformer/GOP-based multi-aspect pronunciation assessment baseline compared in Table 4."},{"cited_title":"Pronunciation Assessment with Multi-modal Large Language Models","cited_arxiv_id":"2407.09209","evidence_quote":"Supplies a prior multimodal-LLM pronunciation assessment system and its reported PCC numbers; comparison target."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the unfreeze experiment by suggesting that updating all audio layers may help; official documentation."},{"cited_title":"Weide et al","cited_arxiv_id":null,"evidence_quote":"Defines the CMUDict/ARPABET phone set used as the phoneme transcription target in the MDD task."}],"review_version":1}