{"id":"b423b587-0398-4563-99f8-6942b17fd4f3","arxiv_id":"2501.19377","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A single audio-plus-text LLM jointly performs voice trigger detection, device-directed speech detection, dialog act classification, and ASR, with reported EER reductions of 64% and 22% over dedicated baselines.","lead":"SELMA is a single speech-enabled language model that handles voice trigger detection, device-directed speech detection, dialog act classification, and speech recognition at once, using a Qwen-Audio backbone with low-rank adapters. The authors report large relative error-rate gains on the two detection tasks compared with dedicated systems, while matching ASR quality only approximately.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 64% VT and 22% DDSD gains assume the prior-paper baselines were evaluated on the same in-house test sets as SELMA; Section III-B only states the test sets are 'similar to' or 'updated versions', so test-set mismatch could drive the improvements.","rationale":"The reader's weakest_assumption correctly identifies the most load-bearing issue: the reported relative improvements presuppose that SELMA and the baselines were evaluated on equivalent test sets. Section III-B uses the phrases 'similar to' for the VT test data and 'updated version' for the DDSD test data, and the UAD/ODLD baselines are not re-run by the authors. This is not an internal inconsistency in the architecture, but it is a correctness risk for the headline claim, because a relatively small shift in test-set difficulty can change EER by more than the claimed margin. I agree with the reader's assessment, and I add that the lack of confidence intervals compounds the problem: even with 130k VT queries, the published point estimates cannot rule out overlap under a different test sample. The concrete test of re-running all baselines on the exact SELMA evaluation sets would settle whether the 64% and 22% relative gains are real. Since the reader already assigned CONDITIONAL with medium confidence and the concern matches that verdict, no adjustment is needed.","tokens_in":9821,"tokens_out":4002,"duration_ms":38779,"concrete_test":"Have the authors re-run UAD [17], ODLD [1], GPT2-XL+Whisper [9], and Qwen 7B+Whisper [9] on the exact SELMA VT/DDSD test sets and report EER with bootstrap 95% confidence intervals. If UAD/ODLD cannot be recreated, at minimum re-run the two [9] baselines on the updated DDSD test and on the exact VT test set, and state explicitly how the VT test set used here overlaps with the evaluation sets of [3] and [17]. If the relative improvements persist under matched evaluation, the central claim holds; if not, the abstract's 64%/22% figures should be revised or qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Table II is that SELMA 1 beats dedicated VT and DDSD baselines by large relative margins: VT EER 0.12% vs UAD 0.33% (64% relative), and DDSD EER 7.78% vs GPT2-XL+Whisper 10.00% / ODLD 12.32% (22% and 37% relative). For this comparison to be valid, every baseline must be scored on the exact same test distribution as SELMA. The paper does not establish this. Section III-B says the VT test data is 'similar to' the sets in [3] and [17] rather than identical, and the DDSD test is an 'updated version' of the in-house test from [9]. The UAD and ODLD numbers are taken from prior publications; only the two [9] DDSD baselines are stated to share training data with SELMA, with no statement that they were re-evaluated on the updated DDSD test set. If the baselines' numbers come from older or differently sampled test distributions, the relative improvements in the abstract and Table II could reflect test-set difficulty rather than model superiority. The absence of confidence intervals or bootstrap estimates makes this risk harder to dismiss: even on 130k VT queries, a distribution shift in baseline FAR/FRR could erase the claimed 64% relative gain. The architecture and ablations are otherwise sensible and the 5.5M trainable parameter count is credible, but the headline comparison is not yet anchored to a shared evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SELMA is a speech-enabled language model that integrates a LoRA-adapted Whisper audio encoder with a Qwen LLM to perform five virtual-assistant tasks (ASR, voice-trigger detection, device-directed speech detection, text-based DDSD, and dialog-act classification) in a single end-to-end system. The architecture concatenates a mean-pooled global audio representation with the full sequence of audio tokens, optionally gated, and is trained with task-specific text prompts. The paper reports large relative EER improvements over dedicated baselines (64% for VT detection, 22% for DDSD) while maintaining competitive ASR WER, and ablation studies isolate the contributions of pooling, gating, auxiliary tasks, and ASR. The main claims are strength and simplicity of a single model replacing a multi-component pipeline.","tokens_in":10158,"tokens_out":4923,"duration_ms":45129,"significance":"If the reported gains hold under a controlled comparison, SELMA would be a significant step toward unifying several virtual-assistant front-end tasks in one multimodal LLM. The approach is parameter-efficient (5.5M trainable parameters out of an 8B-scale model) and uses a simple mean-pooling-plus-sequence design that clearly improves over the configuration with only one of the two. The ablation set is thorough, covering model components, data modalities, and task groupings, and the DET curves provide a more complete view than EER numbers alone. The main weakness is that the headline comparisons against UAD and ODLD rely on baseline numbers from prior work evaluated on test sets that are only 'similar to' or 'updated versions' of the SELMA test sets, and all metrics are single-run point estimates without uncertainty intervals. The paper is otherwise internally consistent, and the central architecture is plausible, so the claims are defensible but not yet fully anchored.","major_comments":[{"comment":"The headline EER improvements (VT: 0.12% vs. UAD 0.33%, a claimed 64% relative gain; DDSD: 7.78% vs. GPT2-XL+Whisper 10.00% and ODLD 12.32%) depend on comparing SELMA to baseline numbers that were not re-computed on the same test distributions. Section III-B states that the VT test set is 'similar to' those in [3] and [17], and that the DDSD test is an 'updated version' of the in-house test from [9]; the text does not state that UAD (from [17]) and ODLD (from [1]) were evaluated on these exact test sets. Even for the DDSD baselines from [9], which share training data, the paper does not confirm they were re-evaluated on the updated test set. To support the relative-improvement claims, the authors should re-run all baselines on the exact SELMA test sets, or else report a detailed analysis of how scores shift across the compared test distributions.","section":"Section III-B and Table II"},{"comment":"No confidence intervals, bootstrap estimates, or significance tests are reported for any EER or WER in Table II or for the DET curves in Figure 3. All numbers appear to come from a single training run. The VT gap (0.12% vs. 0.33% EER) is sizable, but without uncertainty bounds the reader cannot judge whether the differences, particularly the smaller DDSD gaps (7.78% vs. 8.76%) and the ablation differences (e.g., SELMA 1 vs. SELMA 4 at 7.78% vs. 7.63% EER), are statistically reliable. The authors should provide confidence intervals for the primary results, for example via bootstrap re-sampling of test utterances or multiple seeds.","section":"Table II and Figure 3 (general evaluation)"},{"comment":"The ASR transcripts used as ground-truth text for the VT and DDSD training corpora are generated by an auxiliary model with the same architecture as the main SELMA model. This self-training setup is disclosed, but its impact on the VT/DDSD results is not analyzed. If the auxiliary model's transcripts contain systematic errors that correlate with its own weaknesses, the main model could be learning a biased text representation, and the conclusion that 'joint ASR improves VT/DDSD' (supported by the SELMA 1 vs. SELMA 8 ablation) would be less direct. I recommend adding an analysis of the sensitivity of VT/DDSD accuracy to transcript quality, for instance by comparing against human-annotated transcripts on a held-out subset, or by evaluating a version of SELMA that uses a different, non-SELMA ASR to generate the training transcripts.","section":"Section III-B (ASR transcript generation)"}],"minor_comments":[{"comment":"Prompts #2 and #5 are identical ('What does the person say and what type of dialog act is this?'), which appears to be a typographical error; if two tasks are intended to share a prompt, this should be stated explicitly, otherwise one of the rows likely was meant to have a different label.","section":"Table I"},{"comment":"The text says SELMA 1 uses 'a concatenation of the mean pooled audio representation sequence and the sequence itself,' but in Section II-B the concatenation is defined as H_{1:K+1} = [R; H_{1:K}], i.e., the pooled vector is prepended to the sequence. Please align the wording with the formal definition.","section":"Section III-C (SELMA 1 description)"},{"comment":"The ODLD baseline is cited as '[1]', which in the reference list is the Siri team blog post on voice-trigger systems; if ODLD is described in more than one place, a more specific citation would help readers locate the exact model architecture.","section":"Table II / References"},{"comment":"The caption does not specify whether the DET curves are computed on the same test sets as the numbers in Table II; please add a sentence stating the test set and the number of queries used for each curve.","section":"Figure 3"},{"comment":"The paper states 'The overall system has 5.5M trainable parameters,' but it is not clear whether this figure refers to all SELMA variants or only SELMA 1. Since SELMA 4 adds a gating network and SELMA 5 replaces mean pooling with a Q-Former, the parameter counts may differ; please report the count for each variant.","section":"Section III-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically sound in its internal ablations and architecture, but the main comparative claims rest on test sets that are not demonstrably identical to those used for the baselines. Because the test sets are in-house and proprietary, it is particularly important for the authors to either re-run the baselines themselves or provide a clear statement of comparability. The absence of uncertainty estimates is a further concern that a responsive revision can address. The duplicate prompt in Table I and a few other presentation issues are easy to fix. Overall, the contribution has merit and is likely to be publishable after a careful revision that addresses the comparability and statistical issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a legitimate incremental advance in speech-enabled LLMs for virtual assistant pipelines. SELMA jointly trains VT detection, DDSD, ASR, and two auxiliary tasks in a single Qwen-Audio-Chat backbone, with LoRA adapters on both the audio encoder and the LLM, plus a concatenated mean-pooled representation. The ablation study is honest and informative: SELMA 2 shows pooling alone tanks ASR, SELMA 8 shows removing ASR hurts VT/DDSD, and the 5.5M trainable parameter footprint is credible.\n\nWhat is new is the combination rather than any single component. Prior work like [9] and [20] focused on DDSD alone; here the multi-task unification and the concatenation of pooled and sequence-level audio representations is a sensible engineering contribution. The DET curves show SELMA 1 beating UAD across operating points, which is a nice visual.\n\nThe soft spot is exactly what the reader flagged. The headline VT and DDSD gains are not anchored to a shared test distribution. The VT test is described as 'similar to' [17]/[3], and the DDSD test is an 'updated version' of the set in [9]. The UAD and ODLD numbers are taken from earlier papers; only the two [9] DDSD baselines are said to use the same training data, and even those are not explicitly stated to have been re-scored on the updated test set. If the baselines come from slightly older or differently sampled distributions, the 64% relative gain could shrink. The absence of confidence intervals or bootstrap estimates makes this risk hard to dismiss. Also, the auxiliary model's transcripts are used as training labels for VT/DDSD—a mild circularity, but the authors acknowledge it and it is a practical workaround.\n\nOverall, the central claim holds up as a demonstration that a single LoRA-tuned speech LLM can handle multiple assistant tasks at once with competitive accuracy. The evaluation is weaker than the architecture work, but not fatally so. This is systems engineering, not a new scientific mechanism, and it should be reviewed as such.\n\nFor peer review: yes, send it. The ablation and the unification deserve referee time, and the test-set mismatch concern is fixable by re-running baselines or releasing detailed test-set descriptions. I would bring it to a reading group and would cite it if I were working on speech LLMs for on-device assistants.","headline":"A solid, incremental multi-task speech LLM paper whose headline gains hinge on baseline/test-set comparisons that the paper leaves under-anchored.","tokens_in":10681,"tokens_out":2046,"would_cite":true,"duration_ms":20573,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single end-to-end speech-language model can replace the dedicated models for voice-trigger detection, device-directed speech detection, and ASR in a virtual assistant, reporting a 64% relative improvement on voice-trigger EER and a 22%…","keywords":["multi-task learning","multimodal large language model","virtual assistant","voice trigger detection","device-directed speech detection","automatic speech recognition","low-rank adaptation","speech-enabled language model"],"falsifier":"Rerun the UAD, ODLD, and [9] baselines on the exact VT and DDSD test sets used for SELMA; if their Equal-Error Rates fall to SELMA's levels on identical audio, the claimed relative gains would disappear.","tokens_in":9605,"feed_emoji":"🎙️","tokens_out":5676,"duration_ms":48774,"temperature":0.7,"pith_summary":"SELMA is a single end-to-end model that takes audio and text as inputs to a large language model and performs five virtual-assistant tasks at once: voice-trigger detection, device-directed speech detection, automatic speech recognition, text-based device-directed speech detection, and dialog-act classification. The paper argues that this one model matches or beats dedicated single-task models, reporting a 64% relative Equal-Error Rate improvement on voice-trigger detection, a 22% relative improvement on device-directed speech detection, and word error rates near a fine-tuned Whisper baseline. If true, the typical assistant pipeline that chains separate trigger, directedness, and ASR components could be collapsed into a single model, simplifying deployment and maintenance. The contribution is an architecture and training recipe, not a new theoretical result.","feed_headline":"Single speech LLM cuts voice-trigger errors by 64 percent","feed_subtitle":"SELMA also improves device-directed speech detection by 22% and keeps word error rates near the Whisper baseline.","key_machinery":"The load-bearing component is the concatenated mean-pooled representation: the audio encoder's output sequence H is averaged over time to give a single vector R, which is then prepended to the sequence, producing a combined representation that gives the LLM simultaneous access to global utterance context and fine-grained temporal detail. The audio encoder and the LLM backbone are adapted with LoRA modules on their query and value matrices, so only 5.5M parameters (about 0.84% of the model) are trained. A gating network can optionally modulate the audio features, but the concatenation alone, without gating, is the configuration that achieves the headline results.","core_discovery":"The paper shows that a single multimodal LLM, with the audio encoder and the language model jointly fine-tuned using low-rank adaptation, can handle the acoustic and semantic demands of all three primary assistant tasks simultaneously. The key to its reported success is the way audio is represented: the model keeps the full variable-length sequence of audio representations for tasks like ASR, and concatenates a mean-pooled global summary so tasks like voice-trigger and device-directedness detection can exploit the whole-utterance context. On the in-house test sets, this configuration reaches 7.78% EER on DDSD and 0.12% EER on VT detection, with a WER of 0.125, improving on dedicated baselines. Removing either the sequence or the pooled summary degrades performance, showing that both views are load-bearing.","pith_inferences":["The pooled-plus-sequence trick could transfer to other audio-LLM tasks, such as speaker verification or emotion recognition, where both global and local acoustic cues matter.","A streaming variant that updates the pooled summary as audio arrives could extend SELMA to always-on wake-word scenarios without waiting for the full utterance.","The architecture's value is independent of the exact baseline percentages; a head-to-head re-run on identical test audio is what would pin down the true size of the gains.","If the multi-task design scales, the same approach could absorb downstream natural-language understanding, unifying the entire assistant loop in one model."],"forward_implications":["The virtual assistant input pipeline can be reduced from several specialised models to a single model, simplifying deployment and maintenance.","Joint training with ASR provides textual context that helps downstream VT and DDSD decisions, especially in noisy audio conditions.","LoRA adaptation keeps the trainable parameter count tiny, so the model can be switched between applications by swapping small adapter weights.","The concatenated pooled representation is a simple but effective alternative to learned aggregation like Q-Former, achieving comparable or better results without extra parameters.","Removing auxiliary tasks such as text-based DDSD or the standalone ASR task degrades VT detection or transcription quality, evidence that the multi-task design contributes to the results."],"supporting_citations":[{"why":"Supplies the Unified Acoustic Detector baseline, including the 0.33% VT EER against which SELMA's 64% relative improvement is measured.","marker":"[17]"},{"why":"Supplies the GPT2-XL and Qwen 7B multimodal DDSD baselines, with 10.00% and 10.78% EERs, the comparison behind SELMA's 22% relative DDSD improvement.","marker":"[9]"},{"why":"Supplies the text-based Out-of-Domain Language Detector baseline with a 12.32% DDSD EER, a lower-bound comparison for the single-model approach.","marker":"[1]"},{"why":"Qwen-Audio-Chat is the base architecture whose audio encoder and LLM SELMA adapts with LoRA.","marker":"[28]"},{"why":"LoRA provides the parameter-efficient adaptation method that makes joint audio-encoder and LLM fine-tuning feasible with only 5.5M trainable parameters.","marker":"[37]"},{"why":"Whisper large-v2 supplies the pre-trained audio encoder weights that SELMA's audio encoder is initialized from.","marker":"[34]"},{"why":"SALMONN's Q-Former serves as the learned aggregation alternative that mean pooling is compared against in the ablation study.","marker":"[27]"}],"fun_headline_variants":["SELMA: one speech LLM, three assistant tasks, better accuracy","Speech LLM with dual audio views cuts voice-trigger EER by 64%","Joint audio encoder and LLM training slashes voice-trigger errors","One model for trigger, direction, and ASR: SELMA beats dedicated nets","Feature pooling in speech LLM yields 64% better voice-trigger detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 64% and 22% improvements assume that the in-house test sets used for SELMA and the published baselines are comparable, since the UAD and ODLD baselines were not rerun on SELMA's exact test audio.","fun_headline_variants_meta":{"raw":{"variants":["SELMA: one speech LLM, three assistant tasks, better accuracy","Speech LLM with dual audio views cuts voice-trigger EER by 64%","Joint audio encoder and LLM training slashes voice-trigger errors","One model for trigger, direction, and ASR: SELMA beats dedicated nets","Feature pooling in speech LLM yields 64% better voice-trigger detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1390,"prompt_tokens":894,"completion_tokens":496,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":391}},"tokens_in":510,"tokens_out":496,"duration_ms":5232,"temperature":1.0,"reasoning_tokens":391,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T20:17:22.216179+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the UAD, ODLD, and [9] baselines on the exact VT and DDSD test sets used for SELMA; if their Equal-Error Rates fall to SELMA's levels on identical audio, the claimed relative gains would disappear.","supporting_citations":[{"cited_title":"Less is more: A unified architecture for device- directed speech detection with multiple invocation types,","cited_arxiv_id":null,"evidence_quote":"Supplies the Unified Acoustic Detector baseline, including the 0.33% VT EER against which SELMA's 64% relative improvement is measured."},{"cited_title":"A multimodal approach to device-directed speech detection with large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the GPT2-XL and Qwen 7B multimodal DDSD baselines, with 10.00% and 10.78% EERs, the comparison behind SELMA's 22% relative DDSD improvement."},{"cited_title":"V oice trigger system for Siri,","cited_arxiv_id":null,"evidence_quote":"Supplies the text-based Out-of-Domain Language Detector baseline with a 12.32% DDSD EER, a lower-bound comparison for the single-model approach."},{"cited_title":"LoRA: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"LoRA provides the parameter-efficient adaptation method that makes joint audio-encoder and LLM fine-tuning feasible with only 5.5M trainable parameters."},{"cited_title":"SALMONN: Towards generic hearing abilities for large language models,","cited_arxiv_id":null,"evidence_quote":"SALMONN's Q-Former serves as the learned aggregation alternative that mean pooling is compared against in the ablation study."}],"review_version":1}