{"id":"808abde5-ccd8-4266-bc0f-86df087dd880","arxiv_id":"2608.12082","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A unified comparison of six LM-based generative speech enhancement paradigms finds continuous non-autoregressive (CNAR) modeling best, and auxiliary-loss fine-tuning improves DNSMOS, NISQA, PESQ, and POLQA across all six.","lead":"This paper compares six language-model-based speech enhancement designs that operate on neural audio codec features, and finds that continuous, non-autoregressive modeling gives the best quality. It also shows that adding reconstruction and perceptual auxiliary losses during fine-tuning improves four speech quality metrics across all six designs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Discrete paradigms may be handicapped by fixed codebook-fusion and training settings, so the continuous-over-discrete conclusion is not yet established.","rationale":"The paper is internally consistent and the experiments are substantial, but the central claim is an empirical ranking without statistical support and with a potentially unfair comparison between continuous and discrete representations. The reader's weakest assumption correctly identifies the comparably-tuned assumption as the load-bearing issue. I agree that this concern is real and that the verdict should remain CONDITIONAL until the discrete-model architecture and tuning are varied and variance is reported. The proposed token-flattening control directly tests whether the discrete handicap is an artifact of architectural choices, which is the most direct way to settle the core claim. I do not see a reason to move to REJECT: the results are reproducible in principle, the code repository is referenced, and the fine-tuning improvements are transparently described, including the use of differentiable PESQ and STOI losses. The main missing evidence is a demonstration that the discrete paradigms were given a fair opportunity to perform at their best.","tokens_in":9332,"tokens_out":9243,"duration_ms":86645,"concrete_test":"Re-implement DAR, DNAR, and DDiff using the token-flattening (or two-stage codebook prediction) scheme from reference [6], with C times L sequence length and matched total parameters and compute budget (150K plus 50K steps, same hidden size), then re-run the Table 2 evaluation. If any discrete method reaches or exceeds the CNAR/CNAR-FT scores on DNSMOS or PESQ, the claim that continuous paradigms intrinsically excel discrete is falsified. As a secondary check, run at least three random seeds for the top three paradigms and require continuous-discrete margins and CNAR-vs-CAR margins to exceed two standard deviations before endorsing the ranking.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline ranking rests on an unvalidated assumption that the six paradigms are compared at comparable performance potential. Section 2.3(i) fixes a single discrete-token interface: all C=12 codebooks are embedded in parallel and summed into one H=1024 vector, explicitly rejecting the token-flattening strategy of [6] for efficiency. Sections 3.1 and 3.3 then fix identical 24-layer/H=1024 transformers, 150K steps, and one learning rate for every variant, tuning only inference settings such as beam size and N' on the validation set. Because discrete models are thereby restricted to a deliberately coarse codebook fusion and a fixed training budget, the consistent continuous-over-discrete gaps in Table 2 could reflect this architectural handicap rather than an intrinsic advantage of continuous latents. In addition, every entry in Table 2 is a single run with no variance estimate, so the claim that 'CNAR is top in all but one metric and second in the other' is not statistically grounded; the margin between CNAR and CAR is small on several metrics (e.g., DNSMOS 3.01 vs. 2.95).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript presents a unified decoder-only LM framework for six generative speech enhancement (SE) paradigms built on DAC latent features: discrete/continuous autoregressive (D/ CAR), discrete/continuous non-autoregressive (D/CNAR), discrete diffusion (DDiff), and continuous flow matching (CFM). Using the URGENT 2025 challenge data splits and a fixed backbone, the authors compare the paradigms on intrusive and non-intrusive metrics, propose an auxiliary-loss fine-tuning stage with straight-through estimation for discrete models, and report that continuous-domain paradigms outperform their discrete counterparts, with CNAR as the overall best method. The paper also includes an ablation on codec choice and a small inference-hyperparameter study. Code is made publicly available.","tokens_in":9585,"tokens_out":6088,"duration_ms":50610,"significance":"The paper provides a useful synopsis of six LM-based generative SE paradigms under a single experimental framework, which the community has lacked. The use of an official held-out test set, training all models from scratch, and the control of training non-fine-tuned models for the same extra 50K steps are good experimental practices. If the ranking is robust, the design rule — continuous NAC latents, non-autoregressive decoding, and auxiliary-loss fine-tuning — would be directly actionable. The main significance is conditional on two unresolved points: whether the six implementations are comparably tuned, and whether the reported margins are statistically meaningful despite coming from single runs.","major_comments":[{"comment":"The central claim that continuous-domain paradigms consistently outperform discrete-domain paradigms in Table 2 rests on an unvalidated assumption that the discrete implementations are comparably tuned. Section 2.3(i) deliberately replaces the token-flattening strategy of [6] with a parallel sum of all C=12 codebook embeddings, and Sections 3.1 and 3.3 fix the same 24-layer/H=1024 transformer, 150K training steps, and one learning rate for every variant, tuning only inference settings such as beam size and N' on the validation set. The continuous-over-discrete gaps in Table 2 could therefore reflect this fixed discrete token-fusion and training budget rather than an intrinsic advantage of continuous latents. Please add ablations (e.g., flattened token sequences, alternative codebook fusion, or longer training for discrete variants) or otherwise demonstrate that the discrete baselines are not handicapped by these choices.","section":"Sec. 2.3(i), Sec. 3.1, Sec. 3.3"},{"comment":"Every reported number appears to come from a single training run with no confidence intervals or significance tests. The claim that CNAR is top in all but one metric and second in the other relies on small margins in several cases (e.g., DNSMOS 3.01 vs. 2.95 for CAR; NISQA 3.39 vs. 3.46 for CFM). Likewise, some fine-tuning improvements are tiny (e.g., CFM-FT vs. CFM DNSMOS 2.96 vs. 2.95; CNAR-FT vs. CNAR DNSMOS 3.03 vs. 3.01), so the statement that fine-tuning 'consistently improves' DNSMOS, NISQA, PESQ, and POLQA is not statistically grounded. Please report variance across seeds or paired bootstrap confidence intervals, at least for the close comparisons.","section":"Table 2, Sec. 4.2"},{"comment":"The text defines the velocity estimator as v_theta(X1, X0, C, t), but Eq. (4) writes v_theta(X1, X0, Y_c, t) and Eq. (5) uses the same argument. This is an internal notational inconsistency: it is unclear whether the flow-matching model is conditioned on the WavLM features C or on the discrete noisy tokens Y_c. Please correct the notation and make the conditioning of the CFM model explicit, as this matters for reproducing the results.","section":"Eqs. (4)–(5), Sec. 2.3(vi)"}],"minor_comments":[{"comment":"The sentence 'with a significant margin, DAC is the clear winner' is only justified for the intrusive metrics PESQ and ESTOI; on DNSMOS, NISQA, and UTMOS, BiCodec and the X-Codec variants score higher. Please state explicitly that DAC was selected for its intrusive-metric fidelity rather than overall perceptual quality.","section":"Table 1, Sec. 4.1"},{"comment":"Because the fine-tuning loss includes a differentiable PESQ term (delta * J_PESQ) and PESQ is among the reported metrics, the PESQ improvements are partly by construction. The support for the fine-tuning claim is stronger on the non-optimized metrics (DNSMOS, NISQA, POLQA), all of which do improve in Table 2; the paper could make this point more explicitly.","section":"Sec. 2.4, Table 2"},{"comment":"The notation in the CAR description would benefit from distinguishing more clearly between the predicted unquantized features \\hX and the quantized/dequantized version \\h\\tildeX used as the next autoregressive step's input; the current text uses \\hX for both the training target and the inference-time prediction.","section":"Sec. 2.3(ii)"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the same-codebase, same-data head-to-head across the six LM-based SE paradigms, evaluated on an official held-out test set with both intrusive and non-intrusive metrics. That is a service to the field, and I believe the ranking is internally consistent with Table 2: continuous variants beat their discrete counterparts across the board, CNAR comes out best on all but one metric, and the auxiliary-loss fine-tuning improves the four claimed metrics in all six cases. The GitHub link and the use of the URGENT 2025 official splits are also real credit.\n\nThe soft spot is the one the stress-test flags. The comparison fixes one architecture, one training budget, and one discrete-token interface for every paradigm. The discrete models use the parallel-sum codebook fusion rather than the token-flattening strategy of [6], and they train for the same 150K steps at the same learning rate. That is controlled, but it is only 'fair' in the sense that everyone gets the same handicap, not in the sense that each method is near its potential. The continuous-over-discrete conclusion may well be true, but this setup does not prove it; it proves that under this particular unified framework, continuous wins. I would like to see a discrete variant with token flattening, or at least a per-paradigm hyperparameter search, before accepting the stronger claim that continuous latents are intrinsically better for LM-based SE.\n\nThe second soft spot is statistical. Every number in Table 2 comes from one run. No confidence intervals, no significance tests. Some of the margins are small (DNSMOS 3.01 vs 2.95 between CNAR and CAR). For a benchmark whose main message is a ranking, the absence of variance estimates matters. I would not treat the ranking as definitive until at least three seeds are run, or a paired significance test is reported.\n\nThe PESQ overlap in fine-tuning is a minor issue, not a fatal one; the consistent gains on non-optimized metrics like POLQA and NISQA partially address it.\n\nWho gets value from this? Anyone working on generative SE or codec-based LMs. It is a useful reference point and a practical fine-tuning recipe. But it deserves a serious referee, not because the ranking is wrong, but because the paper makes a practical design claim that others will build on. I would ask the authors for multi-run numbers and a fairer discrete baseline before publication.\n\nRecommendation: engage with it, send it to review, and push for the missing statistical grounding.","headline":"A useful six-way benchmark of LM-based SE paradigms in codec latent space, with a plausible but not yet bulletproof continuous-over-discrete ranking.","tokens_in":10082,"tokens_out":2961,"would_cite":true,"duration_ms":27887,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Language-model speech enhancement is better when it models continuous codec features rather than discrete tokens, and the best variant is a continuous non-autoregressive model.","keywords":["language-model speech enhancement","neural audio codec","continuous latent features","discrete tokens","non-autoregressive","flow matching","auxiliary loss fine-tuning","speech quality metrics"],"falsifier":"Train the discrete non-autoregressive model with an alternative token-flattening scheme that interleaves the 12 codebooks per frame, with a comparable parameter count and a doubled training budget; if it then matches or beats the continuous non-autoregressive model on PESQ and DNSMOS, the paper's central continuous-over-discrete conclusion would be falsified.","tokens_in":9136,"feed_emoji":"🎧","tokens_out":6755,"duration_ms":55878,"temperature":0.7,"pith_summary":"This paper argues that, for language-model-based generative speech enhancement, modeling the continuous latent features of a neural audio codec is better than modeling its discrete tokens. It unifies six popular paradigms—autoregressive and non-autoregressive, discrete and continuous, plus discrete diffusion and continuous flow matching—under one decoder-only transformer trained from scratch, and evaluates them on a common challenge corpus with both intrusive and non-intrusive quality metrics. The central result is that every continuous-domain variant outperforms its discrete counterpart, and the continuous non-autoregressive (CNAR) model is best on nearly every metric. The paper also claims that a fine-tuning stage with auxiliary losses on the reconstructed waveform improves DNSMOS, NISQA, PESQ, and POLQA consistently across all six paradigms.","feed_headline":"Continuous beats discrete in every LM speech-enhancement test","feed_subtitle":"A non-autoregressive model over continuous audio-codec features tops six paradigms; fine-tuning lifts all of them.","key_machinery":"The central object is a unified decoder-only language-model architecture over neural-audio-codec latent features. A frozen codec and a frozen self-supervised speech encoder supply the inputs: either discrete tokens from a 12-codebook residual vector quantizer, or continuous dequantized features with 1024 dimensions per frame, plus frame-aligned semantic condition features. A 24-layer transformer with hidden size 1024 is trained either with causal attention for autoregressive variants or full attention for non-autoregressive variants to predict clean latents from noisy latents. The six paradigms are instantiations of this same container: cross-entropy over the 12 codebooks for discrete methods, L1 regression over the continuous features for continuous methods, a masked-token generation schedule for discrete diffusion, and a flow-matching velocity field for continuous flow matching. This shared substrate is what makes the comparison fair.","core_discovery":"On the paper's own terms, the discovery is a ranking: when all six LM-based generative enhancement paradigms share the same codec, the same frozen condition encoder, the same transformer size, the same training budget, and the same inference-time search over only a few settings, continuous-domain methods consistently beat discrete-token methods. The winning paradigm, CNAR, predicts the 1024-dimensional unquantized codec features in a single parallel pass, then quantizes and decodes them; it is top-ranked in all but one metric among the six. The paper attributes the gap to fault tolerance: regressing continuous features is a softer target than classifying 12 discrete tokens per frame, and quantization at the output reduces error accumulation. A further claim is that fine-tuning the LM with auxiliary time- and frequency-domain losses on the decoded waveform improves both non-intrusive and intrusive quality scores in every paradigm.","pith_inferences":["A testable extension would vary the number of codebooks in the discrete branch: if the continuous advantage comes from fault tolerance, increasing codebook count or using a flattened token sequence should narrow the gap.","Because the fine-tuning objective includes differentiable PESQ and STOI losses, the same recipe could be applied to other codec latent domains, such as semantic tokens, where the auxiliary losses may matter even more.","The non-autoregressive advantage suggests that for long-form audio, continuous non-autoregressive models may scale better than autoregressive token predictors, since latency does not grow linearly with sequence length.","If the fault-tolerance explanation is right, continuous models should degrade gracefully under reduced codec bitrate, whereas discrete models should show step-like collapses; that is a direct, checkable consequence."],"forward_implications":["New LM-based speech enhancers should start from continuous codec latents with a non-autoregressive decoder, since this combination is the strongest and also the fastest, requiring only a single parallel forward pass.","Discrete-token approaches should be compared against a continuous regression baseline with matched compute and architecture before claiming gains.","Auxiliary-loss fine-tuning that leaves the codec frozen is a cheap and general recipe that transfers across all six paradigms, so it can likely be appended to future generative speech-enhancement models.","Intrusive and non-intrusive metrics can move together in this setting: PESQ and POLQA improve alongside DNSMOS and NISQA, contrary to the common worry that generative models only help perceptual scores."],"supporting_citations":[{"why":"Supplies the neural audio codec whose latent features all six paradigms share.","marker":"[14]"},{"why":"Supplies the frozen self-supervised encoder for frame-aligned condition features.","marker":"[18]"},{"why":"Provides the latent-space modeling strategy behind the continuous autoregressive variant.","marker":"[5]"},{"why":"Offers the alternative token-flattening scheme the paper's design deliberately avoids.","marker":"[6]"},{"why":"Supplies the masked-token generation schedule used by the discrete diffusion paradigm.","marker":"[19]"},{"why":"Provides the flow-matching objective for the continuous flow-matching paradigm.","marker":"[11]"},{"why":"Supplies the shared training, validation, and test data splits.","marker":"[28]"},{"why":"Provides the frequency-domain loss used in the auxiliary fine-tuning objective.","marker":"[20]"},{"why":"Provides the differentiable PESQ loss in the fine-tuning objective.","marker":"[22]"},{"why":"Provides the differentiable STOI loss in the fine-tuning objective.","marker":"[23]"}],"fun_headline_variants":["Continuous latents beat discrete in LM speech enhancement","CNAR tops six LM-based speech enhancement paradigms","Auxiliary-loss fine-tuning lifts all six LM enhancement models","Continuous codec features beat discrete tokens in LM enhancement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking assumes all six implementations are equally well tuned: same transformer size, same 150K training steps, same learning rate, and only inference settings tuned, so the result could be an artifact of settings that suit continuous paradigms more than discrete ones.","fun_headline_variants_meta":{"raw":{"variants":["Continuous latents beat discrete in LM speech enhancement","CNAR tops six LM-based speech enhancement paradigms","Auxiliary-loss fine-tuning lifts all six LM enhancement models","Continuous codec features beat discrete tokens in LM enhancement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000645,"raw_usage":{"total_tokens":2959,"prompt_tokens":934,"completion_tokens":2025,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":1962}},"tokens_in":550,"tokens_out":2025,"duration_ms":11852,"temperature":1.0,"reasoning_tokens":1962,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:17:02.935896+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the discrete non-autoregressive model with an alternative token-flattening scheme that interleaves the 12 codebooks per frame, with a comparable parameter count and a doubled training budget; if it then matches or beats the continuous non-autoregressive model on PESQ and DNSMOS, the paper's central continuous-over-discrete conclusion would be falsified.","supporting_citations":[{"cited_title":"A Consolidated View of Loss Func- tions for Supervised Deep Learning-Based Speech Enhance- ment,","cited_arxiv_id":null,"evidence_quote":"Supplies the shared training, validation, and test data splits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the neural audio codec whose latent features all six paradigms share."},{"cited_title":"DisContSE: Single-Step Diffusion Speech Enhancement Based on Joint Discrete and Continuous Embed- dings,","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen self-supervised encoder for frame-aligned condition features."},{"cited_title":"We are the first to de- liver a comprehensive and fair synopsis of the paradigms by employ- ing both non-intrusive and intrusive metrics","cited_arxiv_id":null,"evidence_quote":"Provides the latent-space modeling strategy behind the continuous autoregressive variant."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Offers the alternative token-flattening scheme the paper's design deliberately avoids."},{"cited_title":"Flow Matching for Generative Model- ing,","cited_arxiv_id":null,"evidence_quote":"Supplies the masked-token generation schedule used by the discrete diffusion paradigm."},{"cited_title":"LLaSE-G1: Incentivizing Generalization Capability for Llama-based Speech Enhancement,","cited_arxiv_id":null,"evidence_quote":"Provides the flow-matching objective for the continuous flow-matching paradigm."},{"cited_title":"FlowDec: A Flow-Based Full-Band General Audio Codec with High Perceptual Quality,","cited_arxiv_id":null,"evidence_quote":"Provides the frequency-domain loss used in the auxiliary fine-tuning objective."},{"cited_title":"High-Fidelity Audio Compression with Improved RVQGAN,","cited_arxiv_id":null,"evidence_quote":"Provides the differentiable PESQ loss in the fine-tuning objective."}],"review_version":1}