{"id":"2e4cead2-0c3a-4c62-90aa-33f0e7ebc7f0","arxiv_id":"2509.04488","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Serialized CTC outputs from auxiliary branches, used as LLM prompts, improve LLM-based multi-talker ASR WER on Libri2Mix and Libri3Mix.","lead":"This paper proposes giving a large language model a rough first-pass transcript, serialized by speaker, as a prompt to improve multi-talker speech recognition. The prompt is generated by separate speech-to-text branches attached to the speech encoder, and the method cuts word error rates on LibriMix, especially when three speakers talk at once.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOP's contribution is confounded by extra Stage-3 LoRA adaptation; no-prompt control is missing.","rationale":"The reader's weakest assumption identified exactly the confound I consider most load-bearing: the main SOP comparison adds both the prompt and a new LoRA adaptation stage. My read of Sections III-B3 and IV-C and Table I confirms the absence of a Stage-3 no-prompt control. This is not an internal inconsistency—the method is coherent and the reported gains are plausible—but the central causal attribution is under-supported. A single well-chosen control would settle it. I also note the '– Mixed speech encoding' ablation does not serve as the needed control, and the novelty claim against reference [3] is overstated, but the confounded comparison is the primary concern. Since the reader's conditional verdict already captures this required control, my recommendation is to keep the verdict unchanged rather than adjust it.","tokens_in":11150,"tokens_out":3462,"duration_ms":42251,"concrete_test":"Run the exact 3B recipe with one change: in Stage 3, keep the decoder input as [Hp;Et] (no Esop) while training the same new LoRA adapters on the same data and loss (Eqn. 6); report Libri2Mix/Libri3Mix dev/eval WER alongside rows 5–8 of Table I. If this no-prompt Stage-3 control reaches or beats the row-7 WERs, the reported SOP gain is explained by the extra adaptation/training rather than by the prompt. To further isolate the prompt, also decode the Stage-2 SOT-CTC checkpoint with Esop appended at inference only; if the prompt alone carries the improvement, it should appear without Stage-3 adaptation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'SOP assists LLM decoding'—rests on comparing Table I row 6 (SOT-CTC, [Hp;Et], 3B, Libri3Mix eval WER 30.7) with row 7 (SOP-based, [Esop;Hp;Et], eval WER 28.1) and similarly row 5 vs. row 7. These comparisons differ in two ways at once: (i) the SOP token sequence Esop is added to the decoder input, and (ii) Stage 3 introduces a fresh set of LoRA adapters trained with Eqn. (6) while the encoder/separator/CTC are frozen (Section III-B3). Row 6 already underwent Stage 2, which the paper reports degrades performance (e.g., Libri2Mix eval from 9.8 to 11.1 for 3B); row 7 includes an additional adaptation stage that may simply recover that degradation or provide further tuning benefit independently of the prompt. The '– Mixed speech encoding' ablation (rows 8/4/12) removes Hp, not Esop, so it does not isolate the prompt; it only shows the model relies on Hp. No experiment trains a Stage-3 LoRA-adapted model with input [Hp;Et] and no Esop, which is the minimal control needed to attribute the WER gain to SOP. Additionally, Figs. 2–3 show CTC prompt errors in high-overlap and speaker-switching frames, so the assumed reliable-guidance mechanism is not independently validated. Separately, the abstract's claim of 'no prior work exploring prompt design' is questionable given reference [3].","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes SOP-MT-ASR, a three-stage training framework for LLM-based multi-talker ASR. A WavLM encoder is followed by a Separator and S serialized CTC branches; greedy decoding of these branches yields a Serialized Output Prompt (SOP) that is concatenated with the projected mixture encoding and text embedding as the LLM decoder input. Stage 1 fine-tunes the SOT baseline, Stage 2 trains the encoder/separator/CTC branches with a CTC+CE loss, and Stage 3 adapts a fresh LoRA module while freezing the rest. On Libri2Mix/Libri3Mix with LLaMA-3.2-1B/3B/LLaMA-3.1-8B, the reported Stage-3 system improves WER over the Stage-1 SOT baseline under both noisy and clean conditions. The paper concludes that SOP assists LLM decoding. The main empirical evidence is a comparison of Table I's Stage-1/2 rows with Stage-3 SOP rows.","tokens_in":11588,"tokens_out":8098,"duration_ms":91015,"significance":"If the effect were isolated, SOP would be a low-cost way to inject serialized acoustic-text hypotheses into an LLM decoder and could be useful for LLM-based MT-ASR. Strengths: systematic evaluation across three LLM sizes, two overlap conditions, and noisy/clean inputs; explicit architecture and loss equations; honest discussion of CTC branch errors. However, the central claim is currently underdetermined: the reported gain is not attributable to the SOP text because the Stage-3 condition also introduces new LoRA adaptation. The missing control is simple, and the paper would be much stronger with it. Reproducibility is also handicapped by absent training hyperparameters.","major_comments":[{"comment":"The central claim in Sec. IV.C ('SOP assists LLM decoding') is based on comparisons such as Table I row 6 (SOT-CTC, input [Hp;Et]) vs row 7 (SOP, input [Esop;Hp;Et]) for 3B Libri3Mix (eval 30.7 -> 28.1) and row 5 vs row 7. These conditions differ in two ways: the SOP token sequence Esop is added, and Stage 3 trains an additional set of LoRA adapters (Sec. III-B3). Since Stage 2 is reported to degrade several conditions (e.g., 3B clean Libri3Mix eval 22.0 -> 23.4), the Stage-3 improvement may reflect recovery from that degradation through additional adaptation, independent of prompt content. A minimal control is a Stage-3 LoRA-adapted model with input [Hp;Et] and no Esop. Without this control the improvement cannot be assigned to SOP.","section":"Section III-B3 / Table I"},{"comment":"The ablation labeled '– Mixed speech encoding' removes Hp, not Esop, so it changes the acoustic input and leaves the model with only text-like prompt embeddings; the catastrophic WER (e.g., row 8, Libri3Mix eval 154.0) is expected and says nothing about whether Esop contributes. To show the prompt content matters, the paper should compare identical Stage-3 adaptation with and without Esop and, ideally, with a mismatched or random SOP sequence. The current design cannot rule out that any extra token sequence plus adaptation yields the gain.","section":"Section IV.C, Table I rows 4/8/12"},{"comment":"The experimental section omits nearly all training hyperparameters (optimizer, learning rate, batch size, epochs, LoRA rank/alpha, number of CTC branches, and the value of alpha in Eq. (10)). The 'underlined p<0.05' results are not accompanied by a description of the significance test, the unit of analysis, or the number of utterances. The three-speaker offset files are withheld until after review. These omissions make the experiments non-reproducible and prevent assessment of the variability behind the claimed gains.","section":"Section IV.B / IV.C"}],"minor_comments":[{"comment":"The claim that there is 'no prior work exploring the design of prompts' is contradicted by reference [3], which uses versatile instructions. Recommend rephrasing to 'no prior work using decoded serialized output as a prompt'.","section":"Abstract / Introduction"},{"comment":"The number of talkers S is assumed known and fixed, since the separator and CTC branches are built for a predetermined S. This limitation should be stated explicitly, especially because multi-talker ASR is often evaluated with variable numbers of speakers.","section":"Section III.A"},{"comment":"Stage 1 says LoRA weights are merged into the LLM, but Stage 3 introduces another LoRA. Clarify whether the Stage-1 adapters remain merged and how the Stage-3 adapters are initialized and combined with the merged weights.","section":"Section III.B"},{"comment":"The sentence 'The training loss is defined as in Eqn. (10), which is applied not only to the CTC branch, but also to the LLM output' is ambiguous: L_CTC-EncSep uses H_s_sep and T_s, while L_SOT uses Te. Spell out which gradients flow through the LLM in Stage 2 and whether the LLM is frozen.","section":"Section III.B2"},{"comment":"The examples in Figs. 2 and 3 are only two cases; the statement that 'the overall output quality ... provides complete and well-aligned speech content' needs a quantitative CTC-branch WER or accuracy metric, especially because the shown examples contain errors in high-overlap frames.","section":"Section IV.C / Figs. 2-3"},{"comment":"On clean Libri3Mix, the proposed method remains worse than the non-LLM GEncSep baseline (3B eval 16.5 vs 13.1). The conclusion should carefully phrase the contribution as an improvement over the SOT-LLM baseline, not over prior MT-ASR systems.","section":"Table II"},{"comment":"Hcon is used for the concatenated separated embeddings in Eq. (11) and again for the decoder input in Eq. (22), which is actually Hsop; please avoid the overloaded notation. Also, 'Fig. 1–(Stage 3)' should be written as 'Fig. 1 (Stage 3)'.","section":"Notation / References"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The missing no-prompt Stage-3 control is the key issue. I recommend major revision, not rejection, because the architecture and the consistency of the reported improvements across model sizes and conditions make the paper potentially salvageable. If the control shows no WER gain, the contribution should be reframed as a training-strategy paper rather than a prompting paper. The withheld offset files and missing hyperparameters are a reproducibility concern, but I see no evidence of any problem with the data or the results as reported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something sensible: it takes the greedy outputs of the serialized CTC branches and feeds them to the LLM as a prompt. That specific mechanism is new as far as I can tell, and the experiments are extensive, with consistent WER improvements across 1B/3B/8B models and noisy/clean Libri2Mix/3Mix. The 3B clean Libri3Mix gain (22.0 to 16.5) is substantial, and the authors are honest about stage-2 degradation and about the terrible performance when the mixed speech encoding is removed. Credit where due: this is a practical idea that can be built on.\n\nThe soft spot is load-bearing. The comparison that supports the central claim (Table I rows 5 vs 7) differs in two ways at once: the input gets the SOP, and stage 3 introduces a fresh set of LoRA adapters. There is no stage-3 no-prompt control, so we cannot tell whether the gain comes from the prompt or just from the extra adaptation recovering the stage-2 degradation. The ablation removes Hp, not Esop, so it does not isolate the prompt either. The examples in Figs. 2 and 3 show the CTC prompt is quite noisy in high-overlap frames, which makes the \"reliable guidance\" story less intuitive. Also, the abstract's claim that no prior work explores prompt design is overstated given reference [3], which experiments with versatile instructions for the same task.\n\nThese are all fixable. The minimal control is to train the stage-3 LoRA with input [Hp;Et] and no SOP, under identical conditions. If that control shows the same gain, the method reduces to \"more adaptation helps\"; if not, the SOP is doing real work. I'd also want hyperparameters, code, and the three-speaker offset files before relying on the numbers.\n\nThis is a paper for researchers working on LLM-based multi-talker ASR, a small but active subfield. It deserves a serious referee: the method is plausible, the experiments are broad, and the gap is a clean experimental fix rather than a fundamental flaw. But as it stands, the central effect is under-supported, so I would not cite it as evidence for prompt-based gains until the control is done.","headline":"Useful and consistent gains, but the headline effect is confounded by an extra adaptation stage, so the prompt's own contribution is not actually isolated.","tokens_in":12037,"tokens_out":1882,"would_cite":false,"duration_ms":22632,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A rough per-speaker transcript cuts 3-talker speech errors by a quarter","keywords":["multi-talker ASR","serialized output training","connectionist temporal classification","prompting","large language models","LibriMix","speech recognition","LLM decoder adaptation"],"falsifier":"Run the same three-stage recipe but feed a fixed task prompt, or no prompt, during stage 3 while keeping the new LoRA adaptation; if WER matches the SOP result, the prompt content is not the cause. Alternatively, feed SOP into the stage-1 model without the extra LoRA adaptation and check whether WER still drops.","tokens_in":11097,"feed_emoji":"🎙️","tokens_out":8215,"duration_ms":88331,"temperature":0.7,"pith_summary":"This paper proposes a method called serialized output prompting (SOP) for large language model-based multi-talker speech recognition. Instead of giving the LLM decoder only a static task prompt, the system uses per-talker CTC decoders to write a rough serialized transcript of who said what, in speaking order, and feeds that text into the LLM as a guiding prompt. Three training stages are used: SOT fine-tuning, separator and CTC extraction, then SOP adaptation with a fresh LoRA adapter. On LibriMix, the full system improves word error rate over the SOT baseline under both two-talker and three-talker conditions, with the largest relative gains on three-talker clean speech. Prompting is a cheap and largely unexplored lever for making LLM decoders handle overlapping speech.","feed_headline":"A rough per-speaker transcript cuts 3-talker speech errors by a quarter","feed_subtitle":"Per-speaker acoustic decoders write the prompt, improving word error rate on two- and three-speaker LibriMix without retraining the encoder","key_machinery":"The central object is the serialized output prompt (SOP). An LSTM Separator and per-talker CTC heads are inserted after the speech encoder; each CTC branch is aligned to a talker in first-speaking-first-out order, and greedy decoding produces a rough per-talker transcript. These transcripts are concatenated with speaker-change tokens into SOP, embedded, and concatenated to the speech features to form the LLM input. The claimed mechanism is that SOP supplies explicit, acoustically grounded serialization cues, so the LLM does not have to infer overlap structure from acoustic features alone.","core_discovery":"This paper claims that an LLM-based multi-talker ASR system can be improved by prompting the decoder with a serialized output prompt (SOP): a rough text draft of each talker's speech in speaking order, produced by greedy decoding per-talker CTC branches. The authors argue that the serialized CTC outputs provide complete, well-aligned speech content for each talker (Figures 2 and 3), and that explicitly providing these guiding cues helps LLM decoding (Section IV-C). Their evidence is the WER comparison in Table I: on the 3B decoder model, SOP-MT-ASR reduces Libri3Mix clean evaluation WER from 22.0% to 16.5% and noisy evaluation WER from 31.7% to 28.1%, with smaller but consistent gains on Lib","pith_inferences":["I infer the SOP is a general prompting recipe: a lighter auxiliary ASR that writes a per-speaker draft could be bolted onto any promptable LLM decoder, not just this SOT-CTC architecture.","The paper leaves open whether the stage-3 LoRA adaptation alone, without the SOP text, would account for some of the gain; a no-prompt control would test this directly.","Because the paper shows errors concentrate in high-overlap and speaker-switching frames, improving the CTC draft in those frames should directly enlarge the prompt's benefit.","The 8B model's underperformance relative to 3B suggests data-limited adaptation; more SOT data or a different adapter scheme might let larger decoders win."],"forward_implications":["Three-talker recognition, the setting where a plain SOT-trained LLM degrades, is where SOP gives the largest relative gain: clean Libri3Mix evaluation WER drops from 22.0% to 16.5%.","SOP is cheap at inference: it is produced by greedy CTC decoding from a side branch, so the heavy LLM is simply being fed better context.","The three-stage recipe is load-bearing: training everything jointly (ID-0) or stopping after SOT-CTC produces worse WER than the SOP stage.","Removing the mixed speech encoding while keeping only the SOP text is not viable; the acoustic path stays necessary.","The improvement is consistent across 1B, 3B, and 8B LLM decoders, though the 3B model gives the best absolute results in these experiments."],"supporting_citations":[{"why":"supplies serialized output training labels and the speaker-change convention that the baseline and SOP both rely on","marker":"[19]"},{"why":"supplies the Separator and serialized CTC loss used to extract per-talker content for the SOP","marker":"[2]"},{"why":"defines the prior LLM-based MT-ASR paradigm with a simple task prompt, which this work extends","marker":"[3]"},{"why":"supplies the LoRA adapters used in the third training stage for LLM adaptation","marker":"[30]"},{"why":"provides the Libri2Mix and Libri3Mix datasets used for all experiments","marker":"[37]"},{"why":"provides the speech encoder whose pretraining includes multi-talker data","marker":"[31]"},{"why":"prior LLM-based multi-talker ASR work whose strong two-talker results motivate the SOT-LLM baseline","marker":"[1]"},{"why":"target-speaker extraction baseline that the LLM systems are compared against on Libri2Mix","marker":"[41]"}],"fun_headline_variants":["Rough speaker transcripts as prompts cut 3-talker errors","Serialized CTC prompts improve LLM multi-talker speech recognition","Prompting LLMs with rough transcripts reduces multi-talker WER","Self-generated serialized prompts slash 3-speaker ASR errors","Extracting serialized output prompts cuts multi-talker WER"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported gains are credited to the SOP prompt, but the main comparison adds both the prompt and a new LoRA adaptation stage at once; the paper never runs a stage-3 control with the adaptation but without the SOP to prove the prompt itself is the cause.","fun_headline_variants_meta":{"raw":{"variants":["Rough speaker transcripts as prompts cut 3-talker errors","Serialized CTC prompts improve LLM multi-talker speech recognition","Prompting LLMs with rough transcripts reduces multi-talker WER","Self-generated serialized prompts slash 3-speaker ASR errors","Extracting serialized output prompts cuts multi-talker WER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1426,"prompt_tokens":817,"completion_tokens":609,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":521}},"tokens_in":561,"tokens_out":609,"duration_ms":7103,"temperature":1.0,"reasoning_tokens":521,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:54:05.410157+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three-stage recipe but feed a fixed task prompt, or no prompt, during stage 3 while keeping the new LoRA adaptation; if WER matches the SOP result, the prompt content is not the cause. Alternatively, feed SOP into the stage-1 model without the extra LoRA adaptation and check whether WER still drops.","supporting_citations":[{"cited_title":"Serialized Output Training for End-to-End Overlapped Speech Recognition,","cited_arxiv_id":null,"evidence_quote":"supplies serialized output training labels and the speaker-change convention that the baseline and SOP both rely on"},{"cited_title":"Serialized speech informa- tion guidance with overlapped encoding separation for multi-speaker automatic speech recognition,","cited_arxiv_id":null,"evidence_quote":"supplies the Separator and serialized CTC loss used to extract per-talker content for the SOP"},{"cited_title":"Large language model can transcribe speech in multi-talker scenarios with versatile instructions,","cited_arxiv_id":null,"evidence_quote":"defines the prior LLM-based MT-ASR paradigm with a simple task prompt, which this work extends"},{"cited_title":"Wavlm: Large-scale self-supervised pre- training for full stack speech processing,","cited_arxiv_id":null,"evidence_quote":"provides the speech encoder whose pretraining includes multi-talker data"},{"cited_title":"Advancing multi-talker ASR performance with large lan- guage models,","cited_arxiv_id":null,"evidence_quote":"prior LLM-based multi-talker ASR work whose strong two-talker results motivate the SOT-LLM baseline"},{"cited_title":"Exploring time-frequency domain target speaker extraction for causal and non-causal processing,","cited_arxiv_id":null,"evidence_quote":"target-speaker extraction baseline that the LLM systems are compared against on Libri2Mix"}],"review_version":1}