{"id":"63f9b552-0c74-4f46-92ef-0bacbf4c7851","arxiv_id":"2506.01845","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A streaming, layer-reduced WavLM predicts discrete speech units with 50% fewer FLOPs, but the paper's reported low error rates for that configuration are contradicted by its own tables.","lead":"The paper shows that discrete speech units can be extracted with a smaller, streaming version of WavLM-large, cutting compute roughly in half at the cost of higher recognition error. The headline result is undermined by a mismatch between the text and the appendix tables.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 5's flagship Pareto point (6.3/14.0/24.4 at 0.96 TFLOPs) matches no appendix row; the actual 0.96-TFLOP row yields ~11.5/25.6/24.5, so the central efficiency claim is internally contradicted.","rationale":"I read the paper's central goal as showing that a lightweight, streaming DSU extractor can preserve downstream ASR performance. The abstract and Section 5 make this claim quantitative: about 50% FLOP reduction with only a small CER increase, and Section 5 extends it to WER with specific numbers. That quantitative claim is contradicted by the paper's own appendix. This is not a disagreement with prevailing consensus; it is an internal inconsistency between the prose and the tables. The 6.3/14.0/24.4 numbers are not a rounding artifact: at the stated 0.96 TFLOPs, the tables report roughly 11.5/25.6/24.5, a factor-of-two WER discrepancy. Because the contribution is framed around a specific Pareto point, this error is load-bearing. The experiments themselves appear carefully laid out, the appendix tables are detailed, and the code link is a positive; these make the inconsistency directly checkable. I do not see a separate fundamental flaw in the distillation or fine-tuning methodology. The frozen-U2T concern is partially mitigated by the U2T-FT condition, and even there the low-FLOP WER remains poor, so the decisive issue remains the numeric contradiction. The reader's verdict of REJECT is appropriate for the version as written; since my concern does not move that verdict, I mark the recommendation as UNCHANGED.","tokens_in":12932,"tokens_out":7248,"duration_ms":65658,"concrete_test":"Run the released code (github.com/Masao-Someki/StreamingDSU) for the 12-layer, l=r=64, Sym+WF+U2T FT configuration and verify both the computed TFLOPs and the resulting WER/CER. Then separately recompute the configuration whose reported scores match Section 5 (Layer 21, l=r=32) and record its true TFLOPs. If the 0.96-TFLOP row yields approximately 11.5 WER on test-clean and 25.6 on test-other, the paper must correct Section 5 and disclose the WER degradation next to the abstract's CER claim. If it instead yields 6.3/14.0/24.4, then the appendix tables or the FLOPs calculator need correction. Either outcome resolves whether the central Pareto claim is valid as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is an internal inconsistency, not a methodological dispute. Section 5 states: 'Especially with 0.96 TFLOPs, we achieve 6.3 WER (test-clean), 14.0 WER (test-other), and 24.4 CER (test-1h), close to the baseline performance of 5.0 WER, 9.2 WER, and 23.0 CER.' No row in the paper's own Appendix Tables 4 or 5 has that combination. The 0.96-TFLOPs rows are Layer 12, l=r=64: Table 4 (Sym+WF) gives 11.92/26.21/24.52, and Table 5 (Sym+WF+U2T FT) gives 11.50/25.63/24.50. The numbers 6.3/14.0/24.4 instead coincide with Table 4, Layer 21, l=r=32, whose TFLOPs is 1.412, not 0.96. So either the WER numbers or the TFLOPs number in Section 5 is wrong. If the appendix is right, the actual ~50%-FLOP-reduction operating point degrades test-clean WER from 5.0 to about 11.5 (a 130% relative increase) and test-other from 9.2 to about 25.6 (a 178% relative increase), while CER remains roughly flat at 24.5 versus 23.0. This is load-bearing because the paper's headline contribution is a Pareto-optimal efficiency/accuracy tradeoff; the published sentence changes the conclusion from 'near-baseline WER at half the compute' to 'roughly 2.3-2.8x worse WER at half the compute.' The reader's flagged frozen-U2T concern is secondary: the authors do include a U2T-fine-tuned condition, and the appendix's U2T-FT row at 0.96 TFLOPs still shows 11.5 WER.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether discrete speech units (DSUs) can be extracted in a streaming, on-device setting by reducing both the attention window and the number of layers of a WavLM-large based speech-to-unit (S2U) module. The lightweight S2U is trained to reproduce the DSUs of the full frozen model, and the resulting DSUs are evaluated with a discrete ASR system (unit-to-text, U2T) on LibriSpeech (WER) and ML-SUPERB 1h (CER). The authors systematically vary symmetric, past-only, and future-only attention windows; the number of layers; learnable weighted feature summation (WF); and U2T fine-tuning. They report a Pareto-optimal tradeoff curve and claim a 50% FLOPs reduction with only a 6.5% relative CER increase.","tokens_in":13353,"tokens_out":4405,"duration_ms":40676,"significance":"If the empirical results were consistent, the paper would provide a useful contribution to on-device and streaming DSU extraction, an area with practical relevance for speech transmission and LLM-based speech interfaces. The appendix tables give exact values for every plotted operating point, and the code is released, which aids reproducibility. The inclusion of a U2T-fine-tuned condition partly addresses the concern that the frozen baseline U2T may be unfair to the new DSU distribution. However, the central efficiency claim in Section 5 is contradicted by the paper's own appendix, which is a load-bearing issue: if the appendix numbers are correct, the 0.96 TFLOPs operating point has much larger WER degradation than the text states. The paper's contribution is therefore significantly weakened and must be reassessed after correcting the inconsistency.","major_comments":[{"comment":"The sentence \"Especially with 0.96 TFLOPs, we achieve 6.3 WER (test-clean), 14.0 WER (test-other), and 24.4 CER (test-1h)\" is not supported by the appendix. The only 0.96 TFLOPs configurations in the paper are Layer 12, l=r=64 in Table 4 (Sym.+WF: 11.92/26.21/24.52) and Table 5 (Sym.+WF+U2T FT: 11.50/25.63/24.50). The cited numbers 6.3/14.0/24.4 actually match Table 4, Layer 21, l=r=32, whose TFLOPs is 1.412, not 0.96. This is an internal inconsistency in the headline result. If the appendix is correct, the 0.96 TFLOPs operating point degrades test-clean WER from 5.0 to about 11.5 (a roughly 130% relative increase) and test-other WER from 9.2 to about 25.6 (a roughly 178% relative increase), while CER remains roughly flat at 24.5 versus 23.0. The claim \"close to the baseline performance\" is therefore false for WER, and the paper's central Pareto-optimal efficiency/accuracy claim must be corrected and re-evaluated.","section":"Section 5, \"Pareto optimal curve\""},{"comment":"The abstract highlights only the CER result (\"relative increase of 6.5% in CER\"), which is consistent with the 0.96 TFLOPs row (24.4/24.5 vs. 23.0 CER). However, the Section 5 sentence extends the near-baseline claim to WER as well. After the correction, the paper will report a 50% FLOPs reduction with near-flat CER but roughly 2.3–2.8× worse WER. This is a substantially different message and should be reflected in the abstract and conclusion. The authors should clearly state the WER degradation at the reduced-FLOP operating point rather than implying it is close to baseline on all metrics.","section":"Abstract and Section 5"},{"comment":"The term \"Pareto optimal\" is used without a formal definition or a demonstration that the plotted points form the true Pareto frontier. A set of configurations with varying layers and window sizes does not automatically constitute a Pareto-optimal curve unless all dominated points have been removed. Since the paper explicitly lists \"Pareto optimal trade-off curve\" as a contribution, the authors should either define the dominance criterion used, show that every plotted point is non-dominated, or use a more modest term such as \"trade-off curve.\" This is a presentation issue for the central claim, but the numeric inconsistency above is the more pressing problem.","section":"Section 5, \"Pareto optimal curve\" and Figure 1"}],"minor_comments":[{"comment":"The learning rate schedule is described as \"step learning rate decay with a rate of 0.9 every 1K steps,\" but it is unclear whether \"steps\" refers to optimizer steps or training steps within an epoch; please clarify.","section":"Section 2.4"},{"comment":"Reference [24] lists the author as \"A. Waswani\" but the correct name is \"A. Vaswani\" (Vaswani et al., \"Attention is all you need\").","section":"References"},{"comment":"The axis label in Figure 1 reads \"TFLOPs per 1m\" whereas Figures 2 and 4 use \"TFLOPs per 60s\"; please use consistent units (e.g., \"TFLOPs per 60s\" or \"TFLOPs per minute\").","section":"Figure 1 and Tables 1–5"},{"comment":"The paper claims to be the first to explore DSUs in the on-device streaming setting, but related work [38] already applies DSUs to streaming voice conversion; please qualify the novelty claim to avoid overstatement.","section":"Section 1"},{"comment":"In Table 1, the \"Baseline [6]\" row reports test-other WER of 9.1 while \"Baseline (Ours)\" reports 9.2 with the same configuration except beam size. This small difference is fine, but the caption could explicitly note that the beam size reduction explains it.","section":"Section 3, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The numerical inconsistency in Section 5 versus Tables 4–5 is serious enough that the paper should not be accepted in its current form. The authors must correct the text and abstract, and the revised claims must be consistent with the appendix tables. If the appendix numbers are correct, the paper's contribution is weaker than advertised but still potentially publishable after reframing. I also recommend that the editor ask the authors to verify all TFLOPs values and to provide the exact configuration used for the headline result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi. The headline is that the paper's central efficiency claim is internally contradicted by its own appendix. Section 5 says at 0.96 TFLOPs you get 6.3/14.0/24.4 on test-clean/test-other/test-1h, but Appendix Tables 4 and 5 show the only 0.96-TFLOP row (Layer 12, l=r=64) giving 11.5/25.6/24.5. Those WER numbers are about double what the text claims. The 6.3/14.0 numbers actually match a different row (Layer 21, l=r=32) that costs 1.412 TFLOPs, not 0.96. So either the text swapped the TFLOPs label or the WER numbers. This is a load-bearing error, not a methodological dispute.\n\nWhat's genuinely new: this is the first systematic look at streaming DSU extraction with time-restricted attention masks and layer pruning, and the WF and U2T-FT tricks make sense. The experimental setup is reasonable: they train a lightweight DSU predictor against the full model's DSUs, keep the U2T module frozen (though they also include a fine-tuned condition in the appendix), and measure FLOPs properly. The appendix is unusually thorough, with exact numbers for every configuration. That's real evidence.\n\nSoft spots, in proportion. The inconsistency is the big one, and it's not minor. The abstract also cherry-picks the 6.5% CER increase on ML-SUPERB and omits the much larger WER increases on LibriSpeech at the same operating point; that's misleading even if the numbers were right. The frozen-U2T concern the reader flagged is secondary: the paper does include a U2T-fine-tuned condition, and the appendix row at 0.96 TFLOPs with U2T FT still shows 11.5 WER. So that concern doesn't rescue the claim.\n\nAlso worth noting: the baseline uses beam size 5 instead of the challenge's 20, which they disclose. That's fine.\n\nWho's this for? People working on on-device speech-to-LLM pipelines and anyone deciding whether lightweight DSU extraction is plausible. The paper deserves a serious referee, but only with the understanding that the central claim must be fixed. I'd want the authors to either correct the text or correct the appendix, and to report the full set of numbers in the abstract honestly. If the discrepancy turns out to be a typo in the text, the paper is a solid empirical contribution; if it's the appendix that's wrong, the whole Pareto story collapses.\n\nMy recommendation: send to peer review, but flag the discrepancy to the editor before the first decision. It's a credible paper with a fixable but critical flaw.","headline":"The paper's headline efficiency claim is contradicted by its own appendix tables; the actual 0.96-TFLOP configuration gives roughly 11.5 WER, not 6.3, so the Pareto story as written doesn't hold.","tokens_in":13968,"tokens_out":1880,"would_cite":false,"duration_ms":17698,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Discrete speech units can be extracted by a streaming, half-size model that stays close to the full extractor's ASR accuracy.","keywords":["discrete speech units","streaming speech recognition","on-device ASR","WavLM","self-supervised speech models","attention window","FLOPs reduction","ML-SUPERB"],"falsifier":"Retrain or fully fine-tune the unit-to-text module on the DSU stream produced by the 12-layer, 64-frame-window model and measure WER and CER on the same three test sets; if the gap to the 5.0/9.2/23.0 baseline does not shrink, or instead widens beyond the reported 6.3/14.0/24.4, the near-parity claim fails. A complementary check is to measure real end-to-end latency on the target device to confirm that the TFLOPs reduction translates into streaming feasibility.","tokens_in":12655,"feed_emoji":"🎙️","tokens_out":4295,"duration_ms":37804,"temperature":0.7,"pith_summary":"This paper tries to show that discrete speech units (DSUs), the cluster tokens produced from self-supervised speech models, can be computed on the fly and on a resource-limited device at a fraction of the cost of the original full-context model. The authors build a lightweight speech-to-unit predictor that is trained to reproduce the DSUs of a frozen WavLM-large extractor, then reduce both the attention window and the number of layers. They report cutting floating-point operations by about 50% while keeping character error rate on the ML-SUPERB 1-hour benchmark within about 6.5% relative of the full model's CER. The payoff, if true, is that speech can be tokenized on the client and shipped to a server-side language model at a bitrate three or four orders of magnitude lower than raw audio.","feed_headline":"Streaming speech units at half the compute, near-equal accuracy","feed_subtitle":"A lightweight DSU extractor keeps ASR within about 1.5 WER points, enabling on-device speech tokenization for LLMs.","key_machinery":"The load-bearing object is the lightweight DSU predictor, a fine-tuned self-supervised model with a trainable fully-connected layer in place of k-means, trained to output the original model's DSU sequence as ground truth. Two mechanisms make it streaming: a time-restricted self-attention mask that limits how many past and future frames each position can see, and a reduced layer count. A third mechanism, a learnable weighted sum across self-supervised layers, recovers information that individual late layers lose, and fine-tuning the unit-to-text module adapts it to the noisier DSU predictions of small-window models.","core_discovery":"The paper's central claim is that the expensive parts of discrete speech unit extraction, full-length bidirectional attention and hundreds of millions of parameters, are not necessary. A predictor fine-tuned from WavLM-large, restricted to a symmetric time window of past and future frames and using fewer transformer layers, can mimic the original DSU stream closely enough that a frozen unit-to-text ASR module still transcribes well. With a 12-layer model and a 64-frame symmetric window (about 0.96 TFLOPs per minute of audio), the authors report 6.3 WER on LibriSpeech test-clean, 14.0 WER on test-other, and 24.4 CER on ML-SUPERB 1h, compared with the full-model baselines of 5.0 WER, 9.2 WER, and 23.0 CER. They further show that a learnable weighted sum of layer features and fine-tuning the unit-to-text module each close part of the remaining gap.","pith_inferences":["The reported numbers likely understate the achievable accuracy: the frozen unit-to-text module was trained on the original DSU distribution, and retraining it on the streaming model's predicted DSUs, as the paper's own fine-tuning experiments suggest, could close much of the remaining WER and CER gap.","The Pareto curve implies a deployment choice: for a given device budget, an operator could pick the window size and layer count that meet a target error rate rather than treating the full self-supervised model as a fixed cost.","Because DSUs compress one second of audio from roughly 512 kbps raw to about 0.6 kbps, a streaming extractor that runs on-device would let a phone or IoT device send token streams to a server-side LLM, a natural extension the architecture enables.","A testable extension is varying the k-means vocabulary size for the lightweight predictor; the current experiments fix the original challenge's clusters, and the tradeoff may shift if a smaller vocabulary suffices."],"forward_implications":["DSUs can be produced with a theoretical latency proportional to a small future window rather than full utterance length, enabling streaming ASR and speech-to-LLM pipelines.","Halving FLOPs from 1.94 to 0.96 TFLOPs per minute with WER and CER within a few points of baseline suggests on-device DSU extraction is feasible on edge hardware.","Symmetric windows outperform past-only or future-only windows at the same size, while full past with limited future matches symmetric performance beyond about 0.5 seconds, guiding latency and accuracy design.","Fine-tuning the unit-to-text module on predicted DSUs improves robustness to noisy predictions, particularly for small windows."],"supporting_citations":[{"why":"Supplies the challenge baseline and frozen WavLM-large 21st-layer DSU extraction setup that all experiments replace.","marker":"[6]"},{"why":"Supplies WavLM-large, the self-supervised model whose DSUs the lightweight predictor imitates and whose layer features are weighted.","marker":"[3]"},{"why":"Supplies the discrete ASR evaluation recipe, including the unit-to-text module, post-processing, and WER and CER metrics used to score predicted DSUs.","marker":"[5]"},{"why":"Motivates the small-window hypothesis by showing self-supervised speech models encode mainly phonetic information with limited higher-level content.","marker":"[13]"},{"why":"Motivates the learnable weighted feature summation by showing different layers encode different types of information.","marker":"[14]"},{"why":"Provides the time-restricted self-attention mechanism used to limit the attention window for streaming.","marker":"[26]"},{"why":"Supplies the ML-SUPERB 1-hour multilingual benchmark used for character error rate evaluation.","marker":"[21]"}],"fun_headline_variants":["Streaming DSUs: half the FLOPs, near-equal accuracy","Half the compute for on-device streaming speech units, slight error rise","Efficient DSU streaming: 50% fewer FLOPs, 6.5% more CER","On-device DSUs with 50% less compute, only 6.5% more errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the assumption that a unit-to-text module trained on the original clean discrete units is a fair judge of the streaming model's predicted units; if those predicted units differ in ways that module cannot absorb, the reported error rates would not reflect performance after adapting the recognizer to the new unit distribution.","fun_headline_variants_meta":{"raw":{"variants":["Streaming DSUs: half the FLOPs, near-equal accuracy","Half the compute for on-device streaming speech units, slight error rise","Efficient DSU streaming: 50% fewer FLOPs, 6.5% more CER","On-device DSUs with 50% less compute, only 6.5% more errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000639,"raw_usage":{"total_tokens":2919,"prompt_tokens":898,"completion_tokens":2021,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":1926}},"tokens_in":514,"tokens_out":2021,"duration_ms":15147,"temperature":1.0,"reasoning_tokens":1926,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:31:49.547076+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain or fully fine-tune the unit-to-text module on the DSU stream produced by the 12-layer, 64-frame-window model and measure WER and CER on the same three test sets; if the gap to the 5.0/9.2/23.0 baseline does not shrink, or instead widens beyond the reported 6.3/14.0/24.4, the near-parity claim fails. A complementary check is to measure real end-to-end latency on the target device to confirm that the TFLOPs reduction translates into streaming feasibility.","supporting_citations":[{"cited_title":"Knowledge distillation (KD) is of- ten used to reduce the size of S3Ms, such as DistilHuBERT","cited_arxiv_id":null,"evidence_quote":"Supplies the challenge baseline and frozen WavLM-large 21st-layer DSU extraction setup that all experiments replace."},{"cited_title":"However, this issue can be mitigated by limiting the future window size of the S2U module","cited_arxiv_id":null,"evidence_quote":"Supplies WavLM-large, the self-supervised model whose DSUs the lightweight predictor imitates and whose layer features are weighted."},{"cited_title":"Also, by applying such, we produce a Pareto optimal curve that represents the trade-off between the computational over- head and the downstream performance","cited_arxiv_id":null,"evidence_quote":"Supplies the discrete ASR evaluation recipe, including the unit-to-text module, post-processing, and WER and CER metrics used to score predicted DSUs."},{"cited_title":"Exploration of efficient end- to-end asr using discretized input from self-supervised learning,","cited_arxiv_id":null,"evidence_quote":"Motivates the small-window hypothesis by showing self-supervised speech models encode mainly phonetic information with limited higher-level content."},{"cited_title":"Exploring speech recognition, translation, and understanding with discrete speech units: A com- parative study,","cited_arxiv_id":null,"evidence_quote":"Motivates the learnable weighted feature summation by showing different layers encode different types of information."},{"cited_title":"HuBERT: Self- supervised speech representation learning by masked prediction of hidden units,","cited_arxiv_id":null,"evidence_quote":"Provides the time-restricted self-attention mechanism used to limit the attention window for streaming."},{"cited_title":"Anonymizing dysarthric speech: Investigating the effects of voice conversion on pathological information preservation,","cited_arxiv_id":null,"evidence_quote":"Supplies the ML-SUPERB 1-hour multilingual benchmark used for character error rate evaluation."}],"review_version":1}