{"id":"d5d270a2-20ed-4ea6-83f0-c3e09d4f057f","arxiv_id":"2607.25716","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A SpeechLLM-based ASR system can be fine-tuned with federated learning and LoRA adapters, reaching word error rates close to centralized training on English and Italian while transmitting only a small fraction of the model.","lead":"This paper reports a privacy-friendly way to train speech recognition models that use large language models: instead of sending audio to a server, each device updates a small set of adapter parameters and shares only those updates. On English and Italian read-speech benchmarks, the federated model nearly matches a centralized model's word error rate.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central-baseline comparison is underspecified: Section IV-C gives the FL budget (100 rounds, 30% clients, 10 local epochs, about 300 full-data passes) but never reports the central baselines' optimizer, schedule, or update budget, so the claimed near-parity could reflect an order-of-magnitude…","rationale":"The reader's weakest assumption correctly identifies the missing central-training budget as the main threat to the paper's central claim. My read of the full text supports this: Section IV-C is explicit about the FL protocol but silent on the central control, and the figures and Table II contain no training schedule for the centralized runs. The arithmetic in Section V-E (19.7 vs 18.4 is a 1.3 pp deficit, not a 0.3 pp outperformance) reinforces that the numeric reporting needs external verification, but it is secondary to the missing control. The paper does have independent support in the form of a concrete, reproducible-looking FL protocol (Flower, public repository) and a parameter-count argument that communication is reduced by updating 8.4M rather than 85.1M parameters; that part of the claim is not threatened by my concern. However, the headline comparison 'near-centralized performance' cannot be evaluated without knowing how much compute the central baselines received. This does not require rejecting the paper; it requires a conditional acceptance with a requested baseline specification and matched-budget rerun, which is exactly the reader's verdict. No adjustment to the reader's verdict is needed.","tokens_in":10504,"tokens_out":9449,"duration_ms":84834,"concrete_test":"Open the repository linked in Section IV-C and inspect the central-training configuration for the WavLM/Whisper SpeechLLM baselines: optimizer, learning-rate schedule, and total epochs or steps. Count total per-sample updates and compare with the FL budget (100 rounds x 30% clients x 10 local epochs, about 300 full-data passes on LibriSpeech/MLS). If the central config is absent or uses far fewer updates, rerun the central SpeechLLM baseline on LibriSpeech train-clean-100 with the matched update budget and the same optimizer/batch size, then measure WER on test-clean. The near-parity claim stands only if the matched central WER remains at or above the reported 6.1%; a central WER materially below 6.1% would show the FL-vs-central gap is an artifact of an undertrained baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that federated LoRA+projector training matches centralized SpeechLLM training depends on a controlled central baseline. Section IV-C specifies the federated budget in detail: 100 rounds, 30% of speakers sampled per round, 10 local epochs per client. Because selected clients hold disjoint speaker data, each round trains on 30% of the corpus for 10 epochs, i.e., about 3 full-data passes; over 100 rounds this is roughly 300 full-data passes. Nowhere does the paper report the central baselines' total epochs or steps, optimizer, learning-rate schedule, or early-stopping criterion; the linked GitHub repository is not versioned in the preprint. If the central runs in Figures 2-5 and Table II used a conventional 20-30 epoch fine-tune, the federated runs consumed an order of magnitude more updates, so the 6.1% and 6.0% WER references would not be strong baselines and the 'near-centralized performance' claim would be a compute artifact, not evidence that PEFT-only federated optimization is communication-efficient. A related reporting error in Section V-E (19.7% vs 18.4% is a 1.3 percentage-point gap, and federated learning does not outperform central training there) further shows the numeric claims need independent verification. The issue is not internal inconsistency in the FL method but an absent control for the main comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a federated learning framework for SpeechLLM-based end-to-end ASR. The architecture freezes a speech encoder (WavLM-Large or Whisper-Medium) and a TinyLlama-1.1B backbone, training only LoRA adapters and a linear projector; federated aggregation is performed with a modified FedAvg that uses a unified exponential learning-rate decay. Experiments are reported on LibriSpeech-100 (English) and the Italian portion of MLS, comparing federated training against centralized training, comparing WavLM and Whisper encoders, and comparing SpeechLLM against full fine-tuning and adapter baselines. The central claim is that federated training with only PEFT parameters updated achieves near-centralized WER on English and competitive WER on Italian while reducing communication costs.","tokens_in":11008,"tokens_out":3809,"duration_ms":36744,"significance":"If the claims hold, the paper would provide a useful empirical datapoint: a frozen SpeechLLM with only LoRA and projector parameters can be federated across per-speaker clients and approach centralized performance on clean read speech. The experimental design is clear in its use of two languages, two encoders, and a PEFT ablation, and the communication-efficiency argument is underpinned by explicit parameter counts. The paper also ships a public code repository, which is a reproducibility strength even though the preprint does not pin a version. However, the central empirical comparison is currently under-specified: the federated budget is detailed while the centralized baseline budget is not, and at least one numeric claim in the multilingual section is internally inconsistent. These issues bear directly on the paper's main conclusion and must be resolved before the results can be accepted at face value.","major_comments":[{"comment":"The central-training comparison is not controlled. Section IV-C specifies the federated budget in detail (100 rounds, 30% of clients sampled per round, 10 local epochs per client), which implies roughly 300 full-data passes over the corpus. Nowhere do the authors report the central baselines' optimizer, learning-rate schedule, number of epochs or steps, or early-stopping criterion. If the centralized runs in Figures 2–5 and Table II used a conventional 20–30 epoch fine-tune, the federated runs consumed an order of magnitude more updates, and the 'near-centralized performance' claim in Section VI would reflect a compute advantage rather than an efficient method. Please report the full central training budget and, ideally, compare at matched update counts or state why the comparison is fair despite the budget asymmetry.","section":"§IV-C, §V-A–V-E"},{"comment":"The multilingual results are numerically inconsistent. The text states that federated training achieves 19.7% WER on MLS Italian and centralized training 18.4%, then calls the gap 'negligible 0.3 percentage points (actually outperforming)'. The arithmetic is 19.7 − 18.4 = 1.3 percentage points, and the gap favors centralized training, not federated learning. This contradicts the claim and the conclusion of near-parity on Italian; the numbers and the wording must be corrected.","section":"§V-E, Fig. 5"},{"comment":"All reported WERs appear to come from single runs, yet several central claims rest on small absolute differences: 6.4% vs 6.1% in Section V-A, 6.6% vs 6.0% in Section V-C, and 18.7% vs 17.5% in Section V-C. Without multiple seeds, error bars, or a significance test, statements such as 'effectively matching' and 'negligible gap' are not statistically supported. Please provide variance estimates or repeated-run results for the key comparisons, or explicitly justify why single runs are sufficient.","section":"§V-A–V-E, Table II"}],"minor_comments":[{"comment":"The sentence 'All FL results marked as —-,—- ... compared with those achieved with central training marked as ⋆,⋆' appears to have lost its formatting; please define the symbols in the text and in each figure legend so the comparisons are unambiguous.","section":"§IV-C"},{"comment":"The text reports the federated Italian WER as 'approximately 22%' while the figure label shows 22.6%; please harmonize the numbers.","section":"§V-B, Fig. 3(b)"},{"comment":"There is a typo in 'both datsets' and a stray period in '18.4.%'; please proofread the final version.","section":"§V-E"},{"comment":"The text says 'only the adapter parameters are updated' while Section III-A and III-E state that LoRA and projector parameters are trained; please clarify whether 'adapter' refers to the projector only or to both the projector and LoRA modules.","section":"§III-D"},{"comment":"The symbol N is used both for the total number of clients in Figure 1 and for the number of participating clients in Equation (1); please use distinct notation to avoid ambiguity.","section":"§III-E, Eq. (1)"},{"comment":"The abstract claims the 'first systematic study' of federated SpeechLLM training, but the related work already discusses FL for PEFT-based ASR; please temper the novelty statement or explicitly distinguish the present contribution from [8], [19]–[22].","section":"§II, Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core idea is plausible, but the uncontrolled central baseline and the arithmetic error in Section V-E are load-bearing for the main claims. I would be willing to accept after the authors provide full central training details, correct the multilingual numbers, and add at least basic variance reporting for the headline comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful empirical recipe, not a breakthrough. The components are all known—FedAvg, LoRA, projector, WavLM/Whisper encoders, TinyLlama—but the specific combination for federated SpeechLLM ASR on LibriSpeech-100 and MLS Italian is new, and the results look plausible. Federated training reaches 6.4% WER on English with WavLM (central 6.1%) and 6.6% with Whisper (central 6.0%). The encoder ablation and the comparison with full fine-tuning and adapters are informative; the fact that full WavLM fine-tuning fails to converge in FL while LoRA+projector works is a genuinely useful datapoint.\n\nThe main soft spot is exactly what the stress-test flags: the central baseline is underspecified. Section IV-C gives the FL budget (100 rounds, 30% clients, 10 local epochs), which works out to roughly 300 full-data passes, but never states the central training budget, optimizer, schedule, or early stopping. If central runs used a conventional 20–30 epoch fine-tune, the near-parity claims reflect a compute advantage, not evidence that the federated optimizer is efficient. This matters, but it does not sink the paper: the headline claim is about communication cost reduction, and updating only 8.4M of roughly 2.2B parameters is a clear bandwidth story independent of the central baseline. The near-parity wording should still be qualified without a controlled budget.\n\nThere is also a concrete numeric error in Section V-E. The text says the MLS Italian gap is 0.3 points and claims FL is “actually outperforming,” but the stated numbers (19.7 vs 18.4) imply a 1.3-point gap in favor of central. The multilingual setup also shows LibriSpeech federated WER at 16.8% versus 6.1% central, a 10.7-point gap that does not fit the conclusion’s “near-centralized performance” summary, which only really holds for the monolingual case. No error bars or seeds are reported anywhere, so variance is unknown.\n\nNone of this kills the contribution. It is an honest empirical study with a plausible recipe and a clear statement of the communication cost advantage. The missing baseline details and the V-E inconsistency are exactly what peer review should catch and fix. I would send it out; the community needs more concrete federated ASR recipes like this, even if the gains are incremental.","headline":"Useful empirical recipe for PEFT-only federated SpeechLLM ASR, with plausible results; the central-baseline budget is missing and Section V-E has a numeric inconsistency that needs fixing.","tokens_in":11378,"tokens_out":2104,"would_cite":true,"duration_ms":19245,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a SpeechLLM-based ASR system can be trained federatively, with raw audio kept on devices, by exchanging only low-rank adapter and projector parameters, reaching near-centralized word error rates on English and…","keywords":["federated learning","automatic speech recognition","SpeechLLM","LoRA","parameter-efficient fine-tuning","FedAvg","exponential learning rate decay","multilingual ASR"],"falsifier":"Retrain the centralized models with the same effective number of optimizer updates as the federated pipeline (100 rounds, 30% of clients per round, 10 local epochs per client) and compare WER; if central WER drops below the reported 6.1% on LibriSpeech or 20.1% on MLS Italian, the claimed parity is an artifact of unequal compute rather than an efficient federated method.","tokens_in":10356,"feed_emoji":"🎙️","tokens_out":9865,"duration_ms":77291,"temperature":0.7,"pith_summary":"This paper tries to establish that a speech-language-model ASR stack can be trained in a federated way, with raw audio staying on client devices, by updating only the small projection layer and low-rank adapters while the speech encoder and language model remain frozen. The authors propose a modified FedAvg schedule with a unified exponential learning-rate decay, and report that on English LibriSpeech the federated model reaches word error rates within a few tenths of a point of centrally trained models, with a somewhat larger but still competitive gap on Italian MLS. If the finding holds, privacy-preserving ASR could be built on billion-parameter speech LLMs without shipping data to a server and without paying the communication cost of full-model updates.","feed_headline":"Federated speech LLMs near centralized ASR with tiny updates","feed_subtitle":"Updating just LoRA and projector weights keeps word error rates within a point or two of server-trained models.","key_machinery":"The load-bearing mechanism is a federated aggregation loop that transmits only the trainable parameters—the two-stage linear projector and the LoRA low-rank matrices attached to the frozen language model backbone—plus a modified FedAvg rule in which every client uses the same exponentially decaying learning rate, $\\eta_t = \\eta_0 \\gamma^{\\lfloor t/\\tau \\rfloor}$ with $\\eta_0 = 0.001$, $\\gamma = 0.9$, and $\\tau = 10$. This schedule is what distinguishes Adaptive FedAvg from vanilla FedAvg in the comparison, and the architecture's separation of frozen backbone from small trainable adapters keeps communication and local computation low enough for federated training to be practical.","core_discovery":"The central discovery is that federated optimization of a SpeechLLM-based ASR system is stable and effective when the trainable parameter set is restricted to the projector and LoRA modules, and when client learning rates follow a shared exponential decay. With WavLM as encoder, federated training reaches 6.4% WER on LibriSpeech test-clean against 6.1% for centralized training; with Whisper-medium, 6.6% against 6.0%. On Italian MLS, the federated model trails centralized training by roughly 2 percentage points with WavLM (22.6% vs 20.1%) and by about 1 point with Whisper (18.7% vs 17.5%). The paper also reports that full fine-tuning of the encoder does not converge in the federated setting, while the modular SpeechLLM converges, and that a multilingual joint run attains near-parity on Italian but a larger gap on English.","pith_inferences":["Editorial inference: if the parity holds under matched compute budgets, the practical baseline for private ASR shifts from small on-device models to federated fine-tuning of large speech LLMs; the main remaining bottleneck would be client-side memory for running the frozen backbone.","Editorial inference: the unified decay schedule can be read as a form of global learning-rate control that reduces client drift, and a testable extension is to replace the round-indexed schedule with a client-adaptive or loss-based schedule to handle stronger non-IID conditions.","Editorial inference: the multilingual result, near-parity on Italian but a larger gap on English, suggests the joint federated model underfits the larger or lower-error language; a testable extension is to weight client aggregation by language or use per-language learning rates.","Editorial inference: the communication-cost argument would be strengthened by reporting bytes-per-round directly; a reader could estimate it from the adapter size, but a direct measurement would let operators trade round count against bandwidth."],"forward_implications":["Federated SpeechLLM ASR can reach word error rates within a few tenths of a point of centralized training on LibriSpeech and within about one to two points on MLS Italian in the monolingual setups.","Communication cost scales with the roughly 8.4 million trainable parameters of the adapter path, not the 1.1-billion-parameter backbone, so federated deployment of larger LLM backbones remains feasible as long as adapters stay small.","The unified exponential learning-rate schedule yields faster early convergence than vanilla FedAvg, with 9.7% versus 19.7% WER at round 20 on LibriSpeech and a better final model at round 100.","Encoder choice matters: Whisper-medium, with multilingual pretraining, shows smaller centralized-federated gaps than WavLM, especially on Italian, suggesting that multilingual robustness transfers to federated optimization.","Full fine-tuning of the speech encoder is not viable under this federated protocol because it fails to converge, so parameter-efficient adaptation is not just a bandwidth convenience but a stability requirement."],"supporting_citations":[{"why":"It supplies the weighted aggregation rule in Equation (1), the core of the federated update loop.","marker":"[52]"},{"why":"It defines Low-Rank Adaptation, the trainable low-rank matrices inserted into the LLM layers.","marker":"[35]"},{"why":"It introduces the WavLM-large speech encoder used in the primary federated experiments.","marker":"[43]"},{"why":"It introduces the Whisper-medium speech encoder, whose multilingual pretraining is linked to smaller federated gaps.","marker":"[45]"},{"why":"It provides the 1.1-billion-parameter language model backbone that stays frozen during federated training.","marker":"[46]"},{"why":"It describes the two-stage projection and average-pooling adapter that maps encoder embeddings into the LLM input space.","marker":"[51]"},{"why":"It supplies the English read-speech corpus and the test-clean evaluation set.","marker":"[53]"},{"why":"It supplies the Italian read-speech corpus used for the second case study.","marker":"[54]"},{"why":"It gives the adapter-based parameter-efficient baseline in Table II that the SpeechLLM approach is compared against.","marker":"[8]"}],"fun_headline_variants":["Federated speech LLMs nearly match centralized with only LoRA and projector","Tiny updates let federated speech LLMs rival centralized ASR","Federated speech LLMs: 1-2 point WER gap with minimal client updates","Efficient federated training lets speech LLMs come within a point of centralized","Federated speech LLMs: efficient updates, near-centralized WER"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that federated training is competitive depends on the centralized baseline having been given a fair training budget; the paper reports the federated schedule in detail but omits the central baseline's optimizer, schedule, and number of updates, so the parity comparisons could reflect an under-trained central model.","fun_headline_variants_meta":{"raw":{"variants":["Federated speech LLMs nearly match centralized with only LoRA and projector","Tiny updates let federated speech LLMs rival centralized ASR","Federated speech LLMs: 1-2 point WER gap with minimal client updates","Efficient federated training lets speech LLMs come within a point of centralized","Federated speech LLMs: efficient updates, near-centralized WER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001123,"raw_usage":{"total_tokens":4663,"prompt_tokens":931,"completion_tokens":3732,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":3629}},"tokens_in":547,"tokens_out":3732,"duration_ms":22134,"temperature":1.0,"reasoning_tokens":3629,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:24:00.103558+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the centralized models with the same effective number of optimizer updates as the federated pipeline (100 rounds, 30% of clients per round, 10 local epochs per client) and compare WER; if central WER drops below the reported 6.1% on LibriSpeech or 20.1% on MLS Italian, the claimed parity is an artifact of unequal compute rather than an efficient federated method.","supporting_citations":[{"cited_title":"Communication-efficient learning of deep networks from decentralized data,","cited_arxiv_id":null,"evidence_quote":"It supplies the weighted aggregation rule in Equation (1), the core of the federated update loop."},{"cited_title":"LoRA: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"It defines Low-Rank Adaptation, the trainable low-rank matrices inserted into the LLM layers."},{"cited_title":"Slam-llm: A modular, open-source multimodal large language model framework and best practice for speech, language, audio and music processing,","cited_arxiv_id":null,"evidence_quote":"It describes the two-stage projection and average-pooling adapter that maps encoder embeddings into the LLM input space."},{"cited_title":"Librispeech: an asr corpus based on public domain audio books,","cited_arxiv_id":null,"evidence_quote":"It supplies the English read-speech corpus and the test-clean evaluation set."},{"cited_title":"Efl-peft: A communication efficient federated learning framework using peft sparsification for asr,","cited_arxiv_id":null,"evidence_quote":"It gives the adapter-based parameter-efficient baseline in Table II that the SpeechLLM approach is compared against."}],"review_version":2}