{"id":"d77f1fe0-261d-4958-9ffe-1dc790ae5e13","arxiv_id":"2505.21148","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A speech LLM fine-tuned with a fair-average loss outperforms BERT and wav2vec2 baselines on holistic L2 oral proficiency scoring, and transfers across test parts and datasets.","lead":"The paper tests whether a speech-based large language model, Qwen2-Audio, can grade spoken English proficiency of non-native speakers more accurately than previous speech and text-only systems. If the results hold, automated oral assessment could become cheaper, more consistent, and easier to deploy across different test formats.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper never states whether Linguaskill train/test splits are by candidate; if the same speakers appear in both, the matched and cross-part gains (Tables 3-4) could be inflated by speaker memorization, undermining the SOTA claim.","rationale":"I agree with the reader's weakest assumption. The paper is a competent empirical study with a clear method, reproducible training recipe, and a genuinely public S&I evaluation; those are real strengths. The central claim, however, rests on the Linguaskill test results in Tables 3 and 4, and the paper never establishes that training and test sets contain disjoint speakers. Because Linguaskill is a multi-part test taken by the same candidates, a submission-level split can easily leak speaker identity into training. A 7B speech LLM is exactly the kind of model that would exploit such leakage, and the cross-part 'read-aloud transfers to all parts' result (PCC 0.911) is suspiciously strong if speakers overlap. This is not an internal inconsistency; it is an unverified precondition for the generalization claim. The concrete fix is a candidate-ID check and a candidate-disjoint rerun. Since this is checkable and the authors may already have done the right split, the appropriate verdict remains CONDITIONAL rather than REJECT. I would not change the reader's verdict.","tokens_in":9255,"tokens_out":4490,"duration_ms":50965,"concrete_test":"Ask the authors to report the candidate IDs and split criterion for Linguaskill, then recompute Tables 3 and 4 under a strict candidate-disjoint split (partition by speaker before assigning submissions to train/dev/test, while keeping all five parts of a candidate together). If any candidate appears in both training and test, the current numbers are not a valid generalization estimate. If the Qwen2Audio advantage over BERT and wav2vec2 shrinks or disappears under the disjoint split, the central SOTA claim fails; if it persists, the concern is resolved. For S&I, separately evaluate on a held-out candidate-disjoint test set rather than using the calibration dev set for final reporting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that a LoRA-adapted Qwen2Audio grader is state-of-the-art for holistic L2 scoring (PCC 0.954/0.938 on LinGen/LinBus, Table 3) depends on the test set being a true out-of-speaker evaluation. Section 3.1 (Table 1) reports only the number of test submissions and hours, never the number of unique candidates or the split criterion. Linguaskill has five scored parts per candidate; if split is performed at submission level, the same candidate's responses can appear in both training and test partitions. Qwen2Audio is a 7B-parameter model with LoRA adaptation, so it has ample capacity to memorize voice, content, and proficiency-related cues tied to a speaker. This would inflate all matched results in Table 3, and it would be especially damaging to the cross-part analysis in Table 4: a grader trained only on Part 2 read-aloud responses from candidates whose Part 1/3/4/5 responses are in the test set could appear to 'generalize' (overall PCC 0.911) simply by recognizing the speaker. The same logic applies to the S&I cross-task results in Table 5, since the S&I dev set is also used for calibration and final reporting. The claim is therefore not secure until candidate-level disjointness is verified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether speech large language models can serve as holistic L2 oral proficiency graders. Using Qwen2-Audio-7B-Instruct, it compares three training schemes—classification with cross-entropy loss, classification with a proposed 'fair average' loss, and regression—combined with hard and soft decoding strategies. Experiments are conducted on the private Linguaskill corpus and the public Speak & Improve Corpus 2025. The authors report that the LoRA-adapted Qwen2Audio grader with fair-average loss and soft decoding outperforms BERT-cascade and wav2vec2 end-to-end baselines on Linguaskill LinGen and LinBus (PCC 0.954 and 0.938, respectively), and that it generalizes across test parts and across tasks to S&I. The paper also reports zero-shot results showing moderate correlation without task-specific training.","tokens_in":9490,"tokens_out":5569,"duration_ms":68125,"significance":"If the results hold, this is a meaningful advance: a single multimodal LLM grader that consumes audio directly can outperform cascaded ASR-plus-BERT systems and self-supervised speech encoders on holistic L2 scoring, with promising generalization from read-aloud training to spontaneous speech. The proposed fair-average loss is a simple and sensible way to handle half-point human scores, and the paper includes a useful zero-shot comparison. However, the strength of the empirical claim depends on two currently unverified assumptions: that train/test splits are disjoint at the candidate level, and that the reported improvements over strong baselines are statistically reliable given the small PCC differences and the absence of error bars. Both issues are verifiable and should be addressed before the state-of-the-art claim is accepted.","major_comments":[{"comment":"The paper reports train/dev/test splits by number of test submissions and hours, but never states whether the splits are disjoint at the level of individual candidates. Linguaskill has five scored parts per candidate, so if a candidate's responses appear in both training and test partitions, the Qwen2Audio model—with its large capacity and LoRA adaptation—could memorize speaker-identifying acoustic and proficiency cues. This would inflate the matched results in Table 3 and would be especially damaging to the cross-part generalization claim in Table 4, where a grader trained only on Part 2 read-aloud responses reaches an overall PCC of 0.911 on LinGen; it could also inflate the cross-task numbers in Table 5. Please report the number of unique candidates per split, confirm that no candidate appears in more than one split, and if overlap exists, re-run the evaluation on candidate-disjoint splits.","section":"Section 3.1, Table 1"},{"comment":"The headline improvements over the BERT baseline are small: on LinGen the PCC difference is 0.954 vs. 0.942 (Δ = 0.012) and on LinBus it is 0.938 vs. 0.928 (Δ = 0.010). All numbers come from a single training run, with no multiple seeds, error bars, confidence intervals, or significance tests. Given that the paper's central claim is that speech LLMs 'outperform all previous competitive baselines,' the authors should report variance across at least a few training runs and provide a significance test for the metric differences (for example, a bootstrap or permutation test over test responses). Without this, the superiority over BERT on LinGen/LinBus is not statistically established.","section":"Section 3.3, Table 3"},{"comment":"For the S&I corpus, the development set is used to extract linear calibration factors, and the same S&I dev set also serves as the test set in Table 5. Reporting calibrated metrics on the calibration set can inflate performance; although linear calibration primarily affects RMSE, it can still shift PCC/SRC slightly, and using the same set for calibration and final reporting is not standard practice. Please clarify whether a separate S&I test set exists, and if not, report uncalibrated metrics on a held-out portion of the dev set or use cross-validation so that the numbers in the S&I column are not optimistically biased.","section":"Section 3.1, Table 5"}],"minor_comments":[{"comment":"The cross-entropy loss as written is ∑_c y_{i,c} log(ŵ_{i,c}), but y_{i,c} is the predicted probability and ŵ_{i,c} is the one-hot reference label. The correct form is −∑_c ŵ_{i,c} log y_{i,c}, with a minus sign. Please fix the notation.","section":"Section 2.1.1, Eq. (1)"},{"comment":"The text says 'Table 6 evaluates the cross-part performance of the trained graders,' but the table is labeled Table 4. Please correct the cross-reference.","section":"Section 3.4"},{"comment":"The caption says 'Audio2Qwen graders,' but the model is referred to as Qwen2Audio throughout the paper. Please align the naming.","section":"Figure 1 caption"},{"comment":"The regression row is labeled 'reg-' with a trailing hyphen; this should be 'Reg' for consistency with the other rows.","section":"Table 2"},{"comment":"The prompt omits the actual text for Options B through E, showing ellipses. For reproducibility, please include the full prompt in an appendix or supplementary material.","section":"Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"The central concern—candidate-level split integrity—is checkable by the authors, who are affiliated with the group that created the Linguaskill and S&I datasets. The editor should request explicit confirmation of candidate disjointness and, if necessary, results on a candidate-disjoint split before further review. The small PCC differences and single-run results also require significance testing; this is a standards issue for the field and should be treated as a blocking requirement rather than a stylistic preference."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful headline: this is the first paper to apply a speech LLM to holistic L2 oral proficiency scoring, and the results are consistent enough that it deserves a serious referee. The authors show Qwen2Audio with LoRA tuning outperforms the BERT cascade and wav2vec2 baselines on both general and business Linguaskill sets, and the cross-part and cross-task generalization results are genuinely interesting.\n\nWhat is actually new: the application itself, plus a small training trick they call fair average loss, which is basically a soft-label regression on class probabilities. That is not a big technical leap, but it is sensible and works better than hard classification or a linear regression head. The zero-shot result (PCC ~0.64) and the read-aloud-only grader transferring to other parts (0.911 overall PCC) are the kind of empirical findings that could be useful beyond this specific dataset.\n\nThe methodology is generally clean, and the baselines are reasonable. The S&I results on a public corpus help, and the zero-shot numbers are independent of training, so they should be taken as reliable evidence of speech LLM capability.\n\nThe soft spots are real but not fatal. The biggest one, which the stress-test note correctly flags, is that the paper never states whether the Linguaskill train/dev/test splits are by candidate. Linguaskill has five parts per candidate; if the split is at submission level, the same speaker's responses appear in both train and test. With a 7B model, that could inflate the matched and cross-part results. The authors need to clarify this. It is a missing detail, not proof of leakage, but it blocks the SOTA claim.\n\nSecond, all results come from a single training run with no error bars or significance tests. The headline gain over BERT on LinGen is 0.012 PCC, which could easily be within run-to-run noise. Third, for S&I, the dev set is used both for calibration and for final reporting, which is a minor protocol weakness. The \"all previous competitive baselines\" phrasing is a bit strong, but the baselines they compare against are the standard ones from the literature, so I would call it a wording issue, not a fatal one.\n\nBottom line: I largely agree with the reader's conditional verdict. The paper is a solid, competent empirical study that advances the application of speech LLMs to a new assessment task. It deserves peer review, with the split protocol and significance testing as the key conditions.","headline":"This is the first solid application of speech LLMs to holistic L2 oral scoring, with consistent gains, but the missing candidate-level split description leaves the headline SOTA claim unproven.","tokens_in":10087,"tokens_out":3137,"would_cite":true,"duration_ms":32672,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A speech large language model trained with a fair-average loss becomes the best automatic grader of L2 English oral proficiency, reaching a Pearson correlation of 0.954 on a general English test set and 0.938 on a business English test set.","keywords":["spoken language assessment","speech large language models","L2 oral proficiency","fair average loss","Qwen2-Audio","LoRA adaptation","cross-task generalization","CEFR scoring"],"falsifier":"Retrain the Qwen2Audio-fa grader on a speaker-disjoint split of Linguaskill and Speak & Improve, ensuring no candidate appears in both training and test sets, and check whether the PCC on LinGen remains near 0.954 or drops substantially.","tokens_in":9018,"feed_emoji":"🎙️","tokens_out":8138,"duration_ms":74901,"temperature":0.7,"pith_summary":"The paper sets out to establish that speech large language models, which take audio as input and generate text, can grade the holistic speaking proficiency of second-language English learners as accurately as or better than established automatic graders. It trains the Qwen2Audio model with three targets — classification, classification with a fair-average loss, and regression — and evaluates on the Linguaskill and Speak & Improve corpora. The best system, trained with the fair-average loss and decoded with soft scoring, reaches Pearson correlations of 0.954 on general English and 0.938 on business English, outperforming a BERT-based cascade and a wav2vec2 end-to-end grader. If this result is right, speech LLMs are the new state of the art for this task and remove the need for a separate automatic speech recognition step.","feed_headline":"Speech LLM grader hits 0.954 in spoken-English scoring","feed_subtitle":"Trained with a fair-average loss, Qwen2Audio beats BERT and wav2vec2 baselines and generalises across tasks.","key_machinery":"The central object is a grader built from Qwen2Audio, a pre-trained audio-language model, adapted with LoRA adapters of rank 16 on both encoder and decoder, adding 10 million parameters. The load-bearing mechanism is the fair-average loss: the model's logits over six letter grades are passed through a softmax, each grade's probability is weighted by its numeric score (1 to 6), and the weighted average is trained against the human reference score with mean squared error. This preserves intermediate half-point scores that ordinary classification rounding would discard, and the same weighted average is used at decoding time as the soft decoding strategy.","core_discovery":"The paper claims that a speech LLM can serve directly as a holistic L2 oral proficiency grader, and that its predictions align with human scores better than prior systems do. Using Qwen2Audio-7B-Instruct with LoRA adapters, the authors frame grading as a classification task over letter grades mapped to numeric scores, and train with a fair-average loss that computes the softmax-weighted score and minimizes squared error against the reference. On submission-level evaluation across all five test parts of Linguaskill, this grader achieves RMSE of 0.323 and PCC of 0.954 on LinGen and RMSE of 0.356 and PCC of 0.938 on LinBus, beating the wav2vec2 attention-pooling grader and the BERT cascade. The paper further reports that the trained grader generalises across test parts and across corpora, including a low-resource setting where training uses only the small Speak & Improve training set.","pith_inferences":["A natural next step is to apply the same fair-average training to analytic scoring dimensions, since the model already encodes acoustic and linguistic cues needed for separate pronunciation, fluency, grammar, and vocabulary scores.","If the speaker-leakage concern is addressed, the zero-shot correlations near 0.64 suggest an emergent proficiency-judging ability in speech LLMs; probing prompt variations or adding few-shot demonstrations could make zero-shot grading practical for new tests.","The fair-average loss is a generic recipe for tasks with half-step or ordinal human labels, so it could transfer to other rating domains such as essay scoring or medical image grading.","A direct validation would compare the model's agreement with human inter-rater agreement on the same submissions; if the model reaches human-level consistency, it could serve as a second rater in operational assessment."],"forward_implications":["Automatic L2 speaking assessment can be built directly from audio, without an ASR module, avoiding the propagation of transcription errors.","The Qwen2Audio-fa grader becomes the strongest single system on Linguaskill-style data, with PCC gains over the BERT cascade of about 0.01 on both LinGen and LinBus.","A grader trained only on read-aloud sentences transfers to spontaneous and short-answer parts of the test, so read-aloud data are a viable and easier-to-collect starting point.","In the low-resource S&I setting, the speech LLM reaches PCC 0.833 versus 0.753 for the BERT baseline, showing an advantage when training data are limited.","Models trained on one corpus transfer to the other in both directions, indicating robustness to test format and topic domain."],"supporting_citations":[{"why":"Supplies the wav2vec2 end-to-end grader and part of the BERT cascade baselines that the speech LLM must beat.","marker":"[18]"},{"why":"Provides the attention pooling mechanism used to strengthen the wav2vec2 baseline.","marker":"[19]"},{"why":"Provides the pre-trained Qwen2Audio speech LLM used in every experiment.","marker":"[22]"},{"why":"Defines the CEFR proficiency-level descriptors that form the prompt given to the model.","marker":"[28]"},{"why":"Defines the Linguaskill corpus from which the main training and test sets are drawn.","marker":"[29]"},{"why":"Provides the Speak & Improve corpus used for low-resource training and cross-task evaluation.","marker":"[30]"},{"why":"Provides the development set used for linear calibration and for the reported S&I evaluation.","marker":"[31]"},{"why":"Supplies the BERT text-encoder grading approach used in the cascaded baseline.","marker":"[32]"},{"why":"Provides the speech recognition front end for the cascaded BERT baseline.","marker":"[33]"}],"fun_headline_variants":["Speech LLM scores 0.954 PCC on L2 oral test","Qwen2Audio beats BERT, wav2vec2 for L2 speaking scores","Speech LLM generalises across L2 oral proficiency tasks","LLM-based grader edges out cascades for L2 speech","Tuned speech LLM lifts automatic L2 oral scoring accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results depend on the training and test splits being speaker-disjoint; the paper splits by test submission and does not state whether any candidate contributes to both sides, which could allow the model to memorise voices rather than learn proficiency.","fun_headline_variants_meta":{"raw":{"variants":["Speech LLM scores 0.954 PCC on L2 oral test","Qwen2Audio beats BERT, wav2vec2 for L2 speaking scores","Speech LLM generalises across L2 oral proficiency tasks","LLM-based grader edges out cascades for L2 speech","Tuned speech LLM lifts automatic L2 oral scoring accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1581,"prompt_tokens":878,"completion_tokens":703,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":610}},"tokens_in":494,"tokens_out":703,"duration_ms":7957,"temperature":1.0,"reasoning_tokens":610,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:34:17.345396+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the Qwen2Audio-fa grader on a speaker-disjoint split of Linguaskill and Speak & Improve, ensuring no candidate appears in both training and test sets, and check whether the PCC on LinGen remains near 0.954 or drops substantially.","supporting_citations":[{"cited_title":"Transformer based end-to-end mispronunciation detection and diagnosis,","cited_arxiv_id":null,"evidence_quote":"Supplies the wav2vec2 end-to-end grader and part of the BERT cascade baselines that the speech LLM must beat."},{"cited_title":"Explore wav2vec 2.0 for Mispronunciation Detection,","cited_arxiv_id":null,"evidence_quote":"Provides the attention pooling mechanism used to strengthen the wav2vec2 baseline."},{"cited_title":"Assessment of L2 Oral Proficiency Using Self-Supervised Speech Representation Learning,","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained Qwen2Audio speech LLM used in every experiment."},{"cited_title":"Rating short L2 essays on the CEFR scale with GPT-4,","cited_arxiv_id":null,"evidence_quote":"Defines the CEFR proficiency-level descriptors that form the prompt given to the model."},{"cited_title":"Can GPT-4 do L2 analytic assessment?","cited_arxiv_id":null,"evidence_quote":"Defines the Linguaskill corpus from which the main training and test sets are drawn."},{"cited_title":"Emergent Abilities of Large Language Models,","cited_arxiv_id":null,"evidence_quote":"Provides the development set used for linear calibration and for the reported S&I evaluation."},{"cited_title":"Ludlow,Official Quick Guide to Linguaskill","cited_arxiv_id":null,"evidence_quote":"Provides the speech recognition front end for the cascaded BERT baseline."}],"review_version":1}