{"id":"b2ab2365-4242-481a-8d07-bb7186260e36","arxiv_id":"2506.06697","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A per-head learned linear bias on relative positions lets Transformer speech enhancement models trained on 1 or 2 second clips keep most of their quality on 5 to 20 second inputs.","lead":"This paper tests whether Transformer speech enhancement models trained on short clips can still perform on much longer audio, and proposes a simple positional encoding tweak: one learned number per attention head that scales the distance between time frames. It matters for real deployment, where models face utterances many times longer than their training clips and currently lose quality or must fall back to chunked processing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LearnLin's empirical length generalization is plausible, but the paper never reports the learned β_h values or checks that a linear distance bias stays well-scaled at ~1250-frame distances; the load-bearing extrapolation assumption remains untested.","rationale":"The paper is a serious empirical study: it evaluates eight positional encoding schemes, four training objectives, five metrics, causal/non-causal configurations, and chunk processing, and the headline ranking (APE collapses, RPE survives, LearnLin at least ties the best RPE) is consistent across the tables. The central claim—that LearnLin enables length generalization from 1s/2s to 5s-20s—is supported by held-out evaluation: LearnLin maintains or improves PESQ/ESTOI from 1s to 20s while No-Pos and APE degrade, and the chunk-processing control (Table X) shows that full-length processing is genuinely beneficial for LearnLin. I therefore do not think the central claim is false. The most load-bearing soft spot is the untested extrapolation of the single scalar β_h: Eq. (15) is a linear function of distance with no upper bound, and the paper neither reports the learned values nor analyzes whether the induced attention pattern remains sensible at the ~1250-frame distances reached at 20s. Because the content logits are O(1), a bias magnitude of just 0.01–0.1 per frame produces a 10–100× larger term at maximum distance, which would force near-deterministic attention to the farthest frame. That the model works implies β_h is small or negative, but the mechanism story ('different preferences on short- or long-term dependencies') requires positive and negative heads, and no evidence is given. A single concrete check—report β_h and the resulting attention distance distributions—would settle whether the mechanism as described is real or whether LearnLin's success is a learned locality prior. This check does not overturn the empirical result, but it is exactly the information needed to certify the extrapolation, so the paper remains conditional. I agree with the reader's weakest_assumption; my quantitative scaling argument strengthens it. No code or data artifacts currently exist, so requiring the β_h release is a reasonable condition. A secondary but non-fatal concern is that the test sets at different lengths use different underlying utterances, which could confound absolute length trends, but the relative method comparisons at each length are unaffected.","tokens_in":28195,"tokens_out":19101,"duration_ms":188552,"concrete_test":"Train the 1s IRM model for three seeds; extract the 8 learned β_h values. For a 20s input (L≈1250), compute the maximum distance bias max_h |β_h|·1249 and compare it to the 95th percentile of the content logits |QhKh^T|/√d_k. If max_h |β_h|·1249 > 5 for any head, that head's attention is deterministic at long range, contradicting the claimed 'different preferences' and showing the linear bias does not extrapolate stably. Also report the median attended distance per head on 20s inputs; if all heads attend within a short window (<100 frames), length generalization is achieved via a locality prior, not by learned long-range dependencies.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that one scalar β_h per head in Eq. (15) learns 'different preferences on short- or long-term dependencies' and thereby enables generalization from 1s/2s training to 5s-20s test—is supported empirically by Tables II–X, but the mechanism is unverified and the extrapolation is not demonstrated to be safe. With d_k = 32, the content logits QhKh^T/√d_k have entries of order 1. At a 20s input (L ≈ 1250 frames), the LearnLin bias reaches β_h·1249. If any head learned |β_h| > ~0.01, that bias would exceed 5–10, completely dominating softmax and forcing attention to a single farthest frame; if β_h < 0 at similar magnitude, attention collapses to a tiny local window. The model performing well implies |β_h| is small or negative, but the paper reports neither the learned values, nor their standard deviation across seeds, nor any attention-distance diagnostic. Consequently, the asserted mechanism is not established, and there is no evidence that the scalar would remain well-behaved on inputs longer than 20s, where the bias grows linearly without bound. This is the weakest load-bearing link: the empirical result is real for the tested grid, but the paper's interpretation and the robustness of the extrapolation rest entirely on an unexamined scalar.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether Transformer-based speech enhancement models trained on short utterances (1s or 2s) can generalize to longer inputs (5s, 10s, 15s, 20s) at test time. It compares seven existing positional encoding methods, including absolute (Sinusoidal, BERT-Pos) and relative (Gauss-Bias, T5-Bias, TISA, DA-Bias, KERPLE, RoPE) schemes, and proposes a new scheme called LearnLin, in which the attention bias for head h is P^h_{i,j} = β_h |i − j| with one learnable scalar per head shared across layers (Eq. 15). The evaluation covers four training objectives (MS, IRM, PSM, cIRM), five metrics (PESQ, ESTOI, CSIG, CBAK, COVL), causal and non-causal configurations, and chunk-based processing. The headline findings are that absolute positional encoding methods collapse on long utterances, relative positional encoding methods are more robust, and LearnLin is usually the best or tied with the best method.","tokens_in":28456,"tokens_out":8442,"duration_ms":87381,"significance":"If the empirical claims are taken at face value, this is a practically useful result: a single scalar per attention head added as a linear relative-distance bias is a very low-cost way to keep Transformer speech enhancement models functional on inputs longer than those seen in training. The paper's strengths are the breadth of the comparison (seven baseline positional encodings, four training objectives, five metrics, causal and chunk settings) and the clear demonstration that absolute positional encodings fail in the length-generalization regime. However, the absence of variance estimates or significance testing, the incomplete cIRM coverage, and the unverified mechanism attributed to LearnLin substantially temper the strength of the 'excellent length generalization' and 'superior' claims. The qualitative ranking visible in the tables is credible, but the quantitative comparative claims need additional support before the paper can be accepted as is.","major_comments":[{"comment":"The paper's comparative claims that LearnLin 'always performs best' and is 'superior' are supported only by point estimates. No number of independent runs, standard deviations, confidence intervals, or significance tests are reported. Many of the decisive gaps are within 0.01–0.03 in PESQ, CSIG, or COVL; for example, Table VI (5s row) shows LearnLin PESQ 2.74 vs. DA-Bias 2.73, and Table VIII (5s row) shows TISA PESQ 2.76 vs. LearnLin 2.75 and TISA CSIG 3.41 vs. LearnLin 3.40. With no variance measure, these margins cannot support the stated ranking. Please state the number of independent runs, report variance or confidence intervals, or explicitly discuss effect sizes relative to the uncertainty.","section":"§VII-B, Tables II–X"},{"comment":"The LearnLin mechanism is asserted but not verified. The paper never reports the learned β_h values, their distribution across heads, their variability across runs, or any diagnostic of attention as a function of relative distance. This matters because the bias is an unbounded linear function of |i − j|: at a 20s input (roughly 1,250 frames), a head with |β_h| ≥ 0.01 would add a bias above 12 to the content logits, potentially dominating the softmax. That the model performs well on 5–20s inputs is indirect evidence that the learned β_h values are benign, but the claimed 'preferences on short- or long-term dependencies' (Section V-B) and the safety of extrapolation are not established. Please report the learned β_h values and include a sensitivity/attention-distance analysis.","section":"§V-B, Eq. (15)"},{"comment":"The 'four training objectives' coverage is incomplete. For 1s training, Table V reports cIRM only for test lengths 1s, 10s, and 20s, omitting 5s and 15s; for 2s training, no cIRM table is given at all, since Tables VI–VIII contain only MS, IRM, and PSM. The conclusion that LearnLin is superior across four training objectives therefore goes beyond the reported evidence. The authors should supply the missing cIRM results or qualify the claim to the objectives and lengths actually evaluated.","section":"§VII-B, Tables II–V and VI–VIII"},{"comment":"The text states that aside from TISA showing 0.02 CSIG and 0.01 COVL gains over LearnLin at the 5s test length, LearnLin 'always performs best in all other test cases'. The table actually shows TISA also with higher PESQ (2.76 vs. 2.75) and DA-Bias with equal/higher CSIG (3.41 vs. 3.40), and the TISA CSIG gain is 0.01, not 0.02. Please correct this description and re-examine all similar 'always best' statements across Tables VI–VIII.","section":"Table VIII, 5s row, and surrounding text"}],"minor_comments":[{"comment":"The sentence 'Tables VI–IV report the comparison results' should read 'Tables VI–VIII'; the typo appears in the paragraph introducing the 2s training results.","section":"§VII-B"},{"comment":"Notation is inconsistent: the figure legends use 'TIAS' instead of 'TISA', and Table VII uses 'Gauss. Bias' and 'T5 Bias' while other tables use 'Gauss-Bias' and 'T5-Bias'. Please unify the naming.","section":"Figs. 5–7 and Table VII"},{"comment":"The non-causal variant of KERPLE is not described. Equation (6) gives the position bias, but the paper does not state how the original causal formulation is adapted to the non-causal speech enhancement setting. Please specify the exact modification.","section":"§III-B, Eq. (6)"},{"comment":"Table I reports the number of trainable parameters for BERT-Pos as L'·d_model, but the value of L' used in the experiments is never stated. Please report it, since the parameter count of that baseline depends on it.","section":"§VI-B, Table I"},{"comment":"No code or trained checkpoint release is mentioned. Given the density of the tables and the small performance margins, releasing code would substantially improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an extended version of the authors' INTERSPEECH 2024 paper on the same LearnLin approach. The additional experiments (more training objectives, causal setting, chunk processing, more baselines) are useful, but the incrementality should be clearly delineated in the paper. The missing cIRM tables and the absence of any variance reporting are the issues most likely to be probed by a careful reader; neither by itself invalidates the core empirical ranking, but together they require a major revision rather than acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: LearnLin is not new. It is the w|i-j| term inside DA-Bias, and the same head-wise linear bias already appeared in the authors' INTERSPEECH 2024 paper. What this version adds is a broad, mostly well-run empirical comparison: seven PE methods, four training objectives, five metrics, causal and non-causal settings, plus chunk-processing experiments. The tables tell a consistent story: absolute PE collapses on long utterances, relative PE methods survive, and LearnLin is usually best or tied. I checked the headline ranking against the tables and it holds up.\n\nCredit where due: the paper is honest about the relationship to DA-Bias, explicitly noting in Eq. (5) that LearnLin equals its w|i-j| term. The experimental grid is large, the test utterances are held out, and the chunk-processing comparison in Table X is a useful sanity check showing that full-length context matters.\n\nSoft spots, in rough order of severity:\n\n1. Novelty is thin. The central method and the central conclusion already appear in [17]. The systematic comparison is new and useful, but the proposal itself is not.\n\n2. No error bars, no multiple seeds, no significance tests. In Table II, LearnLin beats KERPLE by 0.01-0.03 PESQ at several lengths; with no variance estimates, it is hard to tell whether that is a real advantage or training noise.\n\n3. No code or data artifacts. For a purely empirical paper, that limits reproducibility.\n\n4. The cIRM results skip the 5s and 15s test lengths in Table V. That is selective reporting, even if the reason is compute.\n\n5. The learned beta_h values are never reported. The stress-test worry about a linear bias growing without bound at 1250-frame distances is reasonable: if any head had |beta_h| > 0.01, the bias would dominate the attention logits. The model clearly did not collapse, so the learned scales are probably small, but the paper never shows them. Reporting beta_h, attention-distance diagnostics, or a sensitivity analysis would close the gap. This is a missing analysis, not a reason to distrust the measured results at 5-20 seconds.\n\nBottom line: this is a useful engineering study for speech enhancement practitioners, but the core idea is already published. I would send it to a serious referee because the empirical sweep deserves scrutiny, and I would ask for code, multiple seeds with significance, and the learned beta_h values. Cite this version if you want the systematic comparison; cite [17] for the original LearnLin idea.","headline":"LearnLin itself is already in the authors' INTERSPEECH 2024 paper; this version's value is the systematic PE comparison, which is solid but lacks error bars, code, and the learned beta values.","tokens_in":29037,"tokens_out":3736,"would_cite":true,"duration_ms":40268,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Transformer speech enhancement models can be trained on 1- or 2-second clips and still generalize to 20-second noisy utterances, provided attention scores are biased by a learned linear function of frame distance.","keywords":["length generalization","speech enhancement","Transformer","relative position encoding","positional encoding","self-attention","LearnLin","utterance-length extrapolation"],"falsifier":"Train LearnLin on 1-second clips, record the learned $\\beta_h$ values for all heads, and inspect attention maps on 20- and 30-second inputs: if any head's attention saturates on distant frames, becomes nearly uniform, or if the model's PESQ and ESTOI scores on 30-second utterances fall below the no-position-encoding baseline, the claimed length generalization fails.","tokens_in":27978,"feed_emoji":"🎧","tokens_out":6685,"duration_ms":62477,"temperature":0.7,"pith_summary":"This paper asks whether Transformer-based speech enhancement models trained on short 1- or 2-second clips can still perform well on noisy speech that is 5 to 20 seconds long, and argues the answer is yes when positions are encoded as relative distances. Across four training objectives and five quality metrics, absolute position embeddings fail to generalize while relative position encodings mostly succeed. The paper's proposed scheme, LearnLin, adds to each attention head a bias equal to one learned scalar times the frame distance, and reports that it matches or beats every other positional encoding while using the fewest parameters. If this holds, the length gap is less a computational obstacle to engineer around than a position-encoding choice that a single learned slope per head can resolve.","feed_headline":"One learned scalar per attention head bridges 1s to 20s speech","feed_subtitle":"A learned distance bias lets speech enhancement train on 1-second clips and hold quality at 20 seconds","key_machinery":"The central object is LearnLin, a relative position bias $P^h_{i,j} = \\beta_h \\cdot |i - j|$ indexed by attention head $h$, with one learnable scalar $\\beta_h$ per head shared across all Transformer layers. It injects position information by addition before softmax in the scaled dot-product attention, so the bias directly shapes which time frames attend to which. This single linear distance penalty carries the argument: it is simpler than the sigmoidal DA-Bias, the kernelized KERPLE, the radial-basis TISA, the bucketed T5-Bias, or the Gaussian Gauss-Bias, yet the paper finds it damps the effect of utterance length more effectively across all four training objectives and both causal and non-causal configurations.","core_discovery":"The paper's central claim is that a head-wise learnable linear bias on relative frame distance enables Transformer speech enhancement models to extrapolate from short training utterances to much longer test utterances. Specifically, for each attention head $h$, the position bias is $P^h_{i,j} = \\beta_h \\cdot |i - j|$, where $\\beta_h$ is one trainable scalar, shared across layers, and the bias is added to the raw attention scores before softmax. A positive $\\beta_h$ makes the head favor long-range dependencies and a negative one favors short-range dependencies, so a few scalars let different heads specialize. In experiments on four training targets (magnitude spectrum, ideal ratio mask, phase-sensitive mask, and complex ideal ratio mask) and five metrics (PESQ, ESTOI, CSIG, CBAK, COVL), the paper reports that LearnLin gives the best or tied-best scores at 5s, 10s, 15s, and 20s test lengths in nearly all settings, and that its margin over other methods grows as the test utterance gets longer.","pith_inferences":["Editorial inference: the linear bias is an extrapolation gamble, because the paper does not report the learned $\\beta_h$ values; a natural next check is whether heads trained on 1-second clips keep their intended long-range or short-range profiles when evaluated at roughly 1,250 frames, since a large positive slope could make distant frames dominate attention in unintended ways.","Editorial inference: because speech spectrograms have strong local temporal structure, LearnLin's success suggests a monotonic distance penalty captures most of the position information needed for enhancement, a hypothesis that could be tested by ablating content-based attention and keeping only the distance bias.","Editorial inference: the same one-scalar-per-head recipe may transfer to other frame-based audio tasks such as speaker separation or speech recognition, where long-context extrapolation is also practically valuable, but the present study only demonstrates results for speech enhancement."],"forward_implications":["A model trained only on 1-second or 2-second noisy-clean pairs can enhance 20-second utterances with quality scores that match or exceed all seven compared positional encodings, across all four training objectives.","Absolute position embeddings, both sinusoidal and learned, collapse on long inputs and in some cases score below unprocessed noisy speech, while every relative position encoding generalizes better.","For LearnLin, longer inputs help rather than hurt: scores at 10s to 20s are generally higher than at the training length, because more context is available to each frame.","Chunk-based processing with 1-second segments loses context and underperforms full-length extrapolation, even with 50% overlap, so the length-generalization ability itself is what delivers the gains.","A single scalar per attention head is sufficient; the extra parameters and nonlinearities of TISA, KERPLE, and DA-Bias do not translate into better speech enhancement, and LearnLin's margin over them grows with test length."],"supporting_citations":[{"why":"Supplies the Transformer self-attention formulation and the sinusoidal position embedding used as the absolute-position baseline.","marker":"[18]"},{"why":"Introduces ALiBi's linear-distance bias for length extrapolation, the design that LearnLin makes learnable.","marker":"[49]"},{"why":"Introduces KERPLE, the strongest prior relative-position baseline against which LearnLin is compared.","marker":"[21]"},{"why":"Is the paper's earlier study on length generalization in speech enhancement, which this work extends to four training objectives, causal settings, and chunk processing.","marker":"[17]"},{"why":"Provides T5-Bias, a log-binned relative position encoding used as a comparison baseline.","marker":"[19]"},{"why":"Provides TISA, a translation-invariant relative position encoding using radial-basis kernels, used as a comparison baseline.","marker":"[20]"},{"why":"Provides DA-Bias, whose sigmoid distance bias is the closest prior to LearnLin and whose extra nonlinearity the paper finds redundant.","marker":"[58]"},{"why":"Supplies the Transformer backbone architecture and training schedule used in all the experiments.","marker":"[45]"},{"why":"Defines the length generalization problem and motivates the training-short-testing-long scenario.","marker":"[16]"}],"fun_headline_variants":["One scalar per head lets speech enhancement handle 20x longer audio","Train on 1-sec audio, test at 20 sec: one per-head scalar does it","Single learned bias per head lets transformers generalize to long speech","Length generalization in speech enhancement: a one-parameter fix"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single scalar $\\beta_h$ learned only from 1- or 2-second clips, and shared across all layers, continues to produce sensible attention weights at frame distances up to roughly 1,250 that were never seen during training, rather than over-amplifying or suppressing long-range attention.","fun_headline_variants_meta":{"raw":{"variants":["One scalar per head lets speech enhancement handle 20x longer audio","Train on 1-sec audio, test at 20 sec: one per-head scalar does it","Single learned bias per head lets transformers generalize to long speech","Length generalization in speech enhancement: a one-parameter fix"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000756,"raw_usage":{"total_tokens":3382,"prompt_tokens":990,"completion_tokens":2392,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":2314}},"tokens_in":606,"tokens_out":2392,"duration_ms":16623,"temperature":1.0,"reasoning_tokens":2314,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:52:30.978187+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train LearnLin on 1-second clips, record the learned $\\beta_h$ values for all heads, and inspect attention maps on 20- and 30-second inputs: if any head's attention saturates on distant frames, becomes nearly uniform, or if the model's PESQ and ESTOI scores on 30-second utterances fall below the no-position-encoding baseline, the claimed length generalization fails.","supporting_citations":[{"cited_title":"Attention is all you need,","cited_arxiv_id":null,"evidence_quote":"Supplies the Transformer self-attention formulation and the sinusoidal position embedding used as the absolute-position baseline."},{"cited_title":"Train short, test long: Attention with linear biases enables input length extrapolation,","cited_arxiv_id":null,"evidence_quote":"Introduces ALiBi's linear-distance bias for length extrapolation, the design that LearnLin makes learnable."},{"cited_title":"Kerple: Kernelized relative positional embedding for length extrapolation,","cited_arxiv_id":null,"evidence_quote":"Introduces KERPLE, the strongest prior relative-position baseline against which LearnLin is compared."},{"cited_title":"An exploration of length generalization in transformer-based speech enhancement,","cited_arxiv_id":null,"evidence_quote":"Is the paper's earlier study on length generalization in speech enhancement, which this work extends to four training objectives, causal settings, and chunk processing."},{"cited_title":"The case for translation-invariant self- attention in transformer-based language models,","cited_arxiv_id":null,"evidence_quote":"Provides TISA, a translation-invariant relative position encoding using radial-basis kernels, used as a comparison baseline."},{"cited_title":"Da-transformer: Distance-aware trans- former,","cited_arxiv_id":null,"evidence_quote":"Provides DA-Bias, whose sigmoid distance bias is the closest prior to LearnLin and whose extra nonlinearity the paper finds redundant."},{"cited_title":"A time-frequency attention module for neural speech enhancement,","cited_arxiv_id":null,"evidence_quote":"Supplies the Transformer backbone architecture and training schedule used in all the experiments."},{"cited_title":"Exploring length generalization in large language models,","cited_arxiv_id":null,"evidence_quote":"Defines the length generalization problem and motivates the training-short-testing-long scenario."}],"review_version":1}