{"id":"1caea206-08e3-4b63-8137-17e07842d63f","arxiv_id":"2506.01808","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A separately-trained speech projector and text LoRA can be merged with only 1K steps of multimodal fine-tuning to yield a competitive ASR, ST, and spoken QA system.","lead":"The paper describes a system that transcribes, translates, and answers questions from English speech into Chinese, Italian, and German. The trick is to train a speech adapter and a text adapter separately, then merge them with a short fine-tuning step to get a strong multilingual speech assistant.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SQA results rely on a self-contradicted metric: Table 8's official BERT scores diverge sharply from Table 2's LLM-judge accuracy, so the claimed SQA gains over speech-only models and near-topline behavior are not established.","rationale":"The reader's weakest_assumption is exactly this metric issue, and my independent read lands on the same point. The paper itself, in Appendix C.2.1, reports that the organizers' evaluation setup yields scores that differ significantly from the authors' own protocol, and Table 8 quantifies the divergence: for the submitted A.1+B fluent model on en-de, the official BERT score is 0.532 versus 85.2% LLM-judge accuracy. Because the SQA claim is the part of the central claim most exposed to this divergence, it is the load-bearing concern for accepting the paper's headline results. I also note secondary issues—the statement about outperforming SeamlessM4T-v2-large across all metrics is contradicted by Table 2, and the task ratios in Table 11 were selected using both dev and test sets—but these do not threaten the structural A+B+C claim. That claim is supported by relevant ablations: Table 10 includes a randomly initialized LoRA control, shows that pretrained LoRA weights add further gains, and demonstrates that the short 1K-step adaptation is consistently beneficial across tasks and languages. Thus the appropriate verdict remains CONDITIONAL: the parallel-training-plus-short-alignment recipe is plausible, but the reported SQA numbers must be reconciled with the official metric before the specific claimed gains can be trusted.","tokens_in":19057,"tokens_out":4727,"duration_ms":54434,"concrete_test":"Run the organizers' BERT-score evaluation (bert_score version 0.3.13, default model, as in Table 8) on every SQA/QA row in Table 2, including the text-only Llama-3.1-8B-Instruct zero-shot and B LoRA rows, and compare the resulting ranking with the LLM-as-judge ranking. If the ordering changes—for example, if A.1+B fluent no longer approaches or exceeds the text topline, or the en-de/en-it gains disappear—then the reported SQA conclusion is a metric artifact. If the ranking is preserved despite the scale shift, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the merged model approaches the text-only topline for SQA is carried entirely by the LLM-as-judge numbers in Table 2. Appendix C.2.1 and Table 8 show that under the organizers' BERT-score protocol, the same outputs score much lower on multilingual SQA: A.1+B (fluent) reports 85.2% judge accuracy on en-de but only 0.532 BERT score. This is not a uniform scale offset: invalid-question BERT scores are near 0.99 while valid multilingual scores are markedly lower, so the two metrics are not tracking the same behavior. The authors acknowledge the discrepancy and provide no correlation or reconciliation, yet the abstract and conclusions present the SQA performance as a central success. If the yes/no LLM judge rewards fluent paraphrase rather than factual correctness, the claimed SQA gains are a metric artifact. The structural A+B+C recipe is separately supported by the ablations, so this concern targets the SQA component of the central claim rather than the whole pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes NAVER LABS Europe's constrained submission to the IWSLT 2025 instruction-following speech processing track. The proposed pipeline trains a speech projector (A) on ASR/ST or ASR/ST/SQA data and text LoRA adapters (B) on MT/QA data in parallel, then merges them via a short joint instruction-tuning stage (C) of 1K steps. Results are reported for ASR, ST, and SQA on the ACL 60-60 and SpokenSQuAD test sets, with ablations on the adaptation stage, synthetic data, and task ratios. The central structural claim is that the parallel-training-plus-short-alignment recipe works: the multimodal models outperform the projector-only models on ASR and ST and make the speech model capable of SQA, which the projector alone could not do.","tokens_in":19216,"tokens_out":5269,"duration_ms":53605,"significance":"If confirmed, the A+B+C recipe is a practically useful and inexpensive way to add speech capabilities to a frozen instruction-tuned LLM without training a full multimodal model from scratch. The paper's strengths include clear descriptions of the data pipeline, detailed hyperparameter reporting, and a valuable control in Table 10 comparing a randomly initialized LoRA with the pretrained LoRA during the merging step. The paper is also transparent about the discrepancy between its LLM-as-a-judge SQA scores and the organizers' BERT-score results. However, the SQA conclusions are carried entirely by the LLM-as-judge metric, which the paper itself shows diverges sharply from the official protocol, and one explicit comparative claim is contradicted by the paper's own table. These issues need to be resolved before the central claims can be accepted.","major_comments":[{"comment":"The sentence in the Final Discussion that the models 'outperform SeamlessM4T-v2-large across all language pairs and metrics' is directly contradicted by Table 2. For en-it BLEU, SeamlessM4T-v2-large scores 43.54 while A.1+B (fluent) scores 40.48; for ASR WER, SeamlessM4T-v2-large scores 17.6 while the best multimodal models score 18.6. The claim should be corrected to identify the specific language pairs and metrics where the submission is better, and the exceptions should be acknowledged.","section":"Section 5.2, Final Discussion and Table 2"},{"comment":"The SQA results in Table 2 are not supported by the paper's own Table 8. For the submitted model A.1+B (fluent) on valid en-de SQA, Table 2 reports 85.2% LLM-as-judge accuracy, while Table 8 reports a BERT score of 0.532 using the organizers' protocol. Table 8 also shows near-0.99 BERT scores on invalid questions, so the gap is not a uniform offset between metrics. Since the claims that multimodal training enables SQA and approaches the text-only topline are based entirely on the LLM-as-judge numbers, the SQA component of the central claim is not currently established. Please validate the LLM-as-judge metric against the reference-based metric, report both sets of numbers in the main table, or substantially soften the SQA claims.","section":"Appendix C.2.1, Tables 2 and 8"},{"comment":"Table 11 reports ASR/ST/SQA task-ratio sweeps, and the text states that the ratio was selected as the one with the best average WER and BLEU over 'both ACL 60-60 dev and test set.' Because Table 2 reports results on the ACL 60-60 eval set, the reported ASR/ST numbers for the final configuration are affected by test-set selection. This makes the gains of the chosen task ratio optimistic and weakens the ablation's validity. Please rerun the sweep using only the dev set, or explicitly state that the final test-set numbers include selection bias.","section":"Appendix C.2, 'Task Ratios' and Table 11"}],"minor_comments":[{"comment":"The paragraph says 'Table 11 presents the results of our investigation into the inclusion of potentially noisy synthetic textual data,' but the synthetic-data rows (No synthetic data, Only synthetic data) appear in Table 10, not Table 11. The cross-reference should be corrected.","section":"Appendix C.2, 'Inclusion of Synthetic Textual Data'"},{"comment":"The entry '34, 003†' contains an extra space inside the number; this is a formatting typo.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The core recipe and ablations are useful for the IWSLT system-report venue, but the SQA metric discrepancy and the test-set tuning need to be addressed explicitly in revision. I would not oppose publication after these issues are fixed; the paper should not be accepted with the current overclaims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The training recipe is the real contribution, and it is more solid than the headline numbers. Training a speech projector and a text LoRA in parallel, then merging them with a 1K-step joint instruction-tuning pass, is a clean and cheap idea, and the paper supports it with the right ablations: a random-LoRA control, with/without textual tasks, with/without SQA data, synthetic vs non-synthetic data, and step counts. The fluent-answer regeneration for SpokenSQuAD is a practical fix that others will reuse. As a system paper for a constrained IWSLT track, this is honest engineering work rather than a fundamental advance.\n\nNow the soft spots, in rough order of size.\n\nFirst, the paper's claim that the best model outperforms SeamlessM4T-v2-large across all language pairs and metrics is false on its own Table 2: en-it BLEU is 40.48 for A.1+B (fluent) vs 43.54 for Seamless. That sentence needs to go or be qualified.\n\nSecond, the task ratios in Table 11 were chosen using both the ACL 60-60 dev and test sets. That makes the reported averages selected numbers, not test numbers. Reporting them as plain results overstates what the recipe gives out of the box.\n\nThird, the SQA story. The central claim that the merged model gets close to the text topline for SQA rests on the LLM-as-judge metric in Table 2. Appendix C.2.1 and Table 8 show that the organizers' BERT-score protocol gives much lower scores for valid multilingual questions (e.g., en-de 0.532) while invalid questions score near 0.99. The two metrics are not tracking the same thing, and the paper says so, but it still presents the SQA gains as a central success. If the judge rewards fluent paraphrase rather than factual correctness, the SQA improvement is partly a metric artifact. I don't think this sinks the whole A+B+C story, because the ASR/ST gains are measured with standard WER/BLEU/COMET and the ablation tables are consistent. But the SQA part needs either a correlation analysis, the official BERT scores in the main table, or a narrowed claim.\n\nThe Seamless-as-data-generator-and-baseline point does not bother me; that is normal for constrained tracks and it is not circular.\n\nWho this is for: people building low-cost multilingual speech instruction-following systems and IWSLT-style evaluations. I would cite it for the recipe and for the cautionary metric divergence.\n\nRecommendation: send it to serious peer review, but expect major revision on the three points above. As written, I would take it conditionally, not as-is.","headline":"The parallel-projector-plus-LoRA recipe is genuinely useful, but the SQA numbers and the Seamless comparison need to be fixed before I trust the headline claims.","tokens_in":19847,"tokens_out":5112,"would_cite":true,"duration_ms":49364,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A speech projector and text LoRA adapters, trained apart, fuse in 1,000 joint steps into a model that beats speech-only systems on ASR, ST, and spoken QA.","keywords":["speech instruction following","spoken question answering","speech translation","speech projector","LoRA adapters","joint instruction tuning","multimodal large language models","ASR"],"falsifier":"Score the submitted A.1+B fluent model's outputs on the multilingual SpokenSQuAD test set with the organizers' BERT-score settings; if the en-de, en-it, and en-zh BERT scores remain near the reported 0.532-0.666 range while the LLM judge reports 76-85% accuracy, the SQA claim fails under the official metric.","tokens_in":18769,"feed_emoji":"🎙️","tokens_out":8777,"duration_ms":80812,"temperature":0.7,"pith_summary":"This paper is a system description for a constrained multilingual speech instruction-following track: one model must recognize English speech, translate it into German, Italian, or Chinese, and answer questions about the spoken content in those languages. The claim it argues for is that the speech and text sides of such a model do not need to be trained jointly from the start. A speech-to-LLM embedding projector (A) is trained on speech-to-text data while text-only LoRA adapters (B) are trained on text-to-text data in parallel. A short 1,000-step joint instruction-tuning stage (C) then aligns the two, and the fused model improves ASR and translation over the speech-projector-only system and, on spoken QA, approaches the text-only baseline. The practical interest is that the expensive per-modality training is decoupled and the merge is cheap, so hyperparameter effort can concentrate on the final alignment stage.","feed_headline":"Train speech and text separately; 1K joint steps beat speech-only models","feed_subtitle":"The fused model recognizes, translates, and answers questions from English speech in one pass.","key_machinery":"The load-bearing pieces are three. (1) A speech projector: a four-layer transformer encoder that maps averaged speech representations from a large pretrained multilingual speech encoder into the embedding space of a frozen instruction-tuned 8-billion-parameter LLM, letting the LLM consume audio as if it were text. (2) Text LoRA adapters: low-rank updates to the attention and feed-forward weights of the same frozen LLM, trained on text-only translation and QA data. (3) The C-stage alignment: loading both modules and running 1K instruction-tuning steps over interleaved speech and text batches, with the textual batches acting as a stabilizing signal; the authors ablate that a randomly initialized LoRA gives less gain than the pretrained text LoRA. Averaging every third frame of the speech encoder output also shortens sequences so larger batches fit in memory.","core_discovery":"The paper's central discovery is that separately trained modality adapters can be merged into a competent multimodal system by a brief alignment stage. On its own, the speech projector trained with ASR, ST, and SQA data could transcribe and translate but essentially could not perform spoken question answering; the text-only LoRA could handle QA on text but had no speech input. After the two are loaded together and instruction-tuned for 1K steps on mixed speech and text batches, the resulting model answers spoken questions at close to the text-only topline according to the authors' LLM-as-a-judge evaluation, improves ASR by about 1-2 WER points and translation BLEU over projector-only models, and handles all three tasks from the same set of weights. The authors interpret this as evidence that text and speech adaptation can be optimized independently and then aligned with a small amount of joint multimodal data.","pith_inferences":["Editorial extension: the same parallel-then-align recipe may transfer to other modalities; a vision or audio adapter and a text-specialized LoRA could be pretrained independently and fused with a short joint stage, provided the alignment data is representative.","Editorial extension: the paper's own comparison between its LLM judge and the organizers' BERT-score protocol indicates the near-topline spoken-QA result is not yet settled; before using this model as a QA system, one should re-score outputs with the official metric.","Editorial extension: the fluent-answer regeneration step was motivated by domain shift from extractive answers, but the same idea could be tested as a general data augmentation for any extractive QA dataset feeding an LLM, since it converts exact spans into natural sentence-level supervision."],"forward_implications":["A 1,000-step joint tuning stage is enough to fuse separately pretrained modality adapters; the paper's ablations show 2K steps mainly saturate or hurt, so the merge is cheap.","Adding the text modality during fusion improves speech translation most in the low-resource direction (Italian), suggesting textual supervision acts as a helpful regularizer when speech data are scarce.","Regenerating extractive QA answers as fluent sentences before training sharply improves spoken QA scores, at a small ASR cost, and is the variant chosen for the submitted system.","The speech projector alone, even when trained with SQA data, fails to produce SQA answers; the reasoning-style task only becomes learnable once the text LoRA and the projector are aligned in the C stage."],"supporting_citations":[{"why":"Supplies the speech encoder whose representations the projector maps into the LLM.","marker":"[5]"},{"why":"Provides the frozen instruction-tuned LLM that both the projector and the LoRA adapters attach to.","marker":"[8]"},{"why":"Defines the LoRA method used to train the text-side adapter.","marker":"[9]"},{"why":"Provides the spoken question-answering data, including the answers later regenerated as fluent sentences.","marker":"[15]"},{"why":"Serves as the validation and evaluation set for ASR and speech translation scoring.","marker":"[27]"},{"why":"One of the training corpora for ASR and ST/MT.","marker":"[33]"},{"why":"The parliamentary speech corpus used as additional ASR and ST/MT training data.","marker":"[12]"},{"why":"The library whose LLM-as-a-judge scripts produce the SQA accuracy scores in the main table.","marker":"[24]"},{"why":"The reference-free metric used to filter multilingual question-answer pairs and to score translation quality.","marker":"[25]"},{"why":"The reference the authors cite for interleaving text batches as a stabilizing signal during multimodal training.","marker":"[21]"}],"fun_headline_variants":["Separate speech and text training, then 1K joint steps unify them","1K joint steps fuse separately trained speech and text adapters","Quick fusion: 1K steps align speech and text for spoken QA","Independently trained modules merged with brief joint tuning","Speech and text adapters trained apart, tuned together in 1K steps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The spoken-QA gains are measured with an LLM-as-a-judge accuracy score that the paper's own Table 8 shows can diverge sharply from the official BERT-score metric for the same outputs, so if that judge is not a faithful measure, the central SQA claim is not established.","fun_headline_variants_meta":{"raw":{"variants":["Separate speech and text training, then 1K joint steps unify them","1K joint steps fuse separately trained speech and text adapters","Quick fusion: 1K steps align speech and text for spoken QA","Independently trained modules merged with brief joint tuning","Speech and text adapters trained apart, tuned together in 1K steps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000704,"raw_usage":{"total_tokens":3131,"prompt_tokens":855,"completion_tokens":2276,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":2183}},"tokens_in":471,"tokens_out":2276,"duration_ms":15685,"temperature":1.0,"reasoning_tokens":2183,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:32:33.026473+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score the submitted A.1+B fluent model's outputs on the multilingual SpokenSQuAD test set with the organizers' BERT-score settings; if the en-de, en-it, and en-zh BERT scores remain near the reported 0.532-0.666 range while the LLM judge reports 76-85% accuracy, the SQA claim fails under the official metric.","supporting_citations":[{"cited_title":"LoRA: Low-rank adaptation of large language 7 NAVER LABS Europe Submission to the Instruction-following Track models","cited_arxiv_id":null,"evidence_quote":"Defines the LoRA method used to train the text-side adapter."},{"cited_title":"Spoken squad: A study of mitigating the impact ofspeechrecognitionerrorsonlisteningcomprehension","cited_arxiv_id":null,"evidence_quote":"Provides the spoken question-answering data, including the answers later regenerated as fluent sentences."},{"cited_title":"Evaluating multilingual speech translation under realistic condi- tions with resegmentation and terminology","cited_arxiv_id":null,"evidence_quote":"Serves as the validation and evaluation set for ASR and speech translation scoring."},{"cited_title":"BERGEN: A benchmarking library for retrieval-augmented generation","cited_arxiv_id":null,"evidence_quote":"The library whose LLM-as-a-judge scripts produce the SQA accuracy scores in the main table."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The reference-free metric used to filter multilingual question-answer pairs and to score translation quality."},{"cited_title":"Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularization","cited_arxiv_id":"2503.22577","evidence_quote":"The reference the authors cite for interleaving text batches as a stabilizing signal during multimodal training."}],"review_version":1}