{"id":"d7ca0a79-6b3c-4ef5-9528-f4e8d2276a1c","arxiv_id":"2509.21003","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"TF-Restormer restores degraded speech at arbitrary input-output sampling rates in a single model, using a heavy encoder and a lightweight query-based decoder to generate missing high-frequency bands.","lead":"A new speech-restoration model, TF-Restormer, cleans noisy, reverberant, or low-bandwidth audio and can output at a different sampling rate than the input without resampling. It uses a light decoder with learnable extension queries to synthesize missing high-frequency content, supporting denoising, bandwidth extension, and streaming from one model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'arbitrary input-output rates' claim is unsupported: extension queries are length-specific learned parameters and no unseen output rate is trained or evaluated.","rationale":"I focus on the arbitrary-rate claim because it is the core novelty of the paper and it fails at the architectural level, not only at the empirical level. The reader's weakest assumption correctly notes that only a discrete rate set is trained and tested, but the stronger issue is that the described extension-query mechanism is length-specific: q_ext is a learned tensor whose row count equals F_D-F_E, with no stated mechanism for unseen row counts. This makes the advertised 'arbitrary' behavior doubtful even for the rates listed in Sec. 3.1, such as 22.05 or 32 kHz. The metric-leakage concern (PESQ/UTMOS in the loss) is real but less central: fixing it would improve confidence in the perceptual comparisons without directly threatening the multi-rate flexibility claim. My verdict remains aligned with the reader's CONDITIONAL: the paper is a plausible engineering contribution for the evaluated rates, but the central universality claim needs either a scoped statement or dedicated evidence for unseen rates.","tokens_in":21304,"tokens_out":7671,"duration_ms":72867,"concrete_test":"Run the released TF-Restormer checkpoint at 8->32 kHz (and optionally 8->22.05 kHz), two rates listed as supported in Sec. 3.1 but absent from the f_D training set, under the same VCTK super-resolution evaluation protocol; compare LSD, MCD, and UTMOS against the reported 8->24 and 8->44.1 kHz rows. If the untrained-rate outputs match the trained-rate quality and show correct full-band energy, the concern is resolved; if quality collapses or the output is band-limited to roughly 16 kHz, the arbitrary-rate claim must be replaced by a scoped supported-rate claim, or a conditioning/interpolation mechanism must be added and validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"TF-Restormer's headline novelty is a single model that restores speech at arbitrary user-specified input/output rates via a query-based asymmetric decoder. The only rate-adaptive components described are the SFI-STFT front end and learnable extension queries q_ext in R^{(F_D-F_E)xTxC_D} (Sec. 3.2). As written, q_ext is a parameter tensor whose size changes with F_D-F_E; the paper never states how queries for lengths not seen in training are obtained (interpolation, conditioning, or a rate-conditional generator). Training samples f_D from {16,24,44.1,48} kHz with f_E in {8,16} kHz (Sec. 4), and every evaluation in Table 3 uses exactly these combinations. The rates 22.05 and 32 kHz are explicitly listed as supported in Sec. 3.1, but no experiment exercises them, and the extension length for 8->32 kHz (480 bins) is never encountered in training (training lengths are 160, 320, 562, 640, 722, 800). Moreover, f_D < f_E has no defined q_ext since its dimension would be negative and MHCA is only active when f_E < f_D. Thus the phrase 'arbitrary input-output rates' exceeds both the described mechanism and the evidence; the claim should at least be scoped to trained rates unless a rate-generalization mechanism is specified and tested. This is not merely a missing evaluation: the discrete, length-indexed query design makes transfer to unseen lengths non-trivial.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TF-Restormer, a single encoder-decoder model for speech restoration under decoupled input and output sampling rates (which the authors call the xSFI setting). The encoder uses a time-frequency dual-path Transformer to analyze the input bandwidth, while a lightweight decoder synthesizes the missing high-frequency band using learnable extension queries and cross-attention over encoder features. A shared sampling-frequency-independent STFT discriminator and a scaled log-spectral loss are introduced to support multi-rate training and perceptual quality. The model is trained on VCTK with input rates of 8 and 16 kHz and output rates of 16, 24, 44.1, and 48 kHz, and evaluated on UNIVERSE, VCTK-DEMAND, a VCTK super-resolution set, and real-recorded VoxCeleb data, including a streaming variant. The paper claims consistent improvements over prior, frequently larger baselines with lower computational cost for low input rates, and reports ablations for the loss, decoder design, and frequency projection.","tokens_in":21659,"tokens_out":5983,"duration_ms":47389,"significance":"If the empirical claims hold, TF-Restormer is a valuable contribution to speech restoration: a single non-diffusion model that handles denoising, dereverberation, bandwidth extension, and combined distortions across multiple sampling rates, with a streaming mode and openly available code and demos. The architectural asymmetry of a heavy encoder and a lightweight decoder with extension queries is well motivated, and the shared SFI-STFT discriminator is a sensible way to train a single generator across rates. The scaled log-spectral loss is clearly explained and ablated. The main reservations concern the breadth of the 'arbitrary rates' claim, the overlap between the training objective and the headline evaluation metrics, and the absence of uncertainty estimates in the reported tables.","major_comments":[{"comment":"The claim of 'arbitrary input-output sampling rates' is not supported. The extension query q_ext in Sec. 3.2 is a learnable parameter tensor with shape (F_D−F_E)×T×C_D, so its size is tied to the specific rate pair; no mechanism such as interpolation, conditioning, or generation is described for obtaining queries for extension lengths not seen in training. Training uses only f_E in {8,16} kHz and f_D in {16,24,44.1,48} kHz (Sec. 4), and Table 3 evaluates only these combinations; the rates 22.05 and 32 kHz, which Sec. 3.1 lists as supported, are never tested. Since an 8 to 32 kHz extension would require 480 extension bins and training only sees extension lengths of 160, 320, 562, 640, 722, and 800, generalization to arbitrary user-specified rates is nontrivial and remains unverified. Additionally, the case f_D < f_E is undefined because q_ext would have negative dimension. Please either scope the claim to the trained rate pairs or add a rate-generalization mechanism and evaluate it on unseen rates such as 22.05 and 32 kHz.","section":"Sec. 3.2, Sec. 4, Table 3"},{"comment":"The training objective in Eq. (6) includes L_hf = L_pesq + 10·L_utmos, while Tables 1-3 report PESQ and UTMOS as primary outcomes. Consequently, a non-trivial part of the reported advantage on these specific metrics is directly optimized during training, which is a form of evaluation leakage. The paper should disclose this overlap explicitly in the main text of Section 5 (it is currently not stated there) and provide results with the human-feedback loss ablated, or justify why the PESQ and UTMOS gains are not artifacts of this term. The independent metrics (WVMOS, DNSMOS, LSD, MCD, sBERT) mitigate the concern but do not eliminate it, because the headline tables foreground PESQ and UTMOS.","section":"Eq. (6), Tables 1-3"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any of the main quantitative results. Several comparisons hinge on small differences, for example Table 3 clean 8 to 16 kHz LSD is 0.89 for TF-Restormer versus 0.83 for NVSR, while the UTMOS advantage of 4.10 versus 3.64 is only one column; it is therefore difficult to assess whether 'consistently outperforms' is robust. Please add variability estimates across utterances or random seeds, or at least report the number of test utterances per cell and pairwise significance tests for the central claims.","section":"Tables 1-3"},{"comment":"The super-resolution test set is generated by a simulation pipeline that follows the same procedure used for training, which the paper acknowledges 'may provide a slight advantage to our model.' Since this is the only evaluation supporting the core multi-rate capability, the advantage is not necessarily slight: the model is tested on the distribution on which it was trained, while the baselines may be at a disadvantage if their training distributions differ. Please either include a cross-dataset or held-out evaluation, such as a public bandwidth-extension test set or native 44.1/48 kHz recordings, or temper the universality claims accordingly.","section":"Sec. 5.1, Appendix D, Table 3"}],"minor_comments":[{"comment":"The definition of q_ext as R^{(F_D−F_E)×T×C_D} conflicts with the statement that its values are 'shared across all the frames'; if the same query is shared, the tensor should be R^{(F_D−F_E)×1×C_D} and tiled, or the text should clarify the intended sharing mechanism.","section":"Sec. 3.2"},{"comment":"The shape notation is inconsistent: earlier the STFT features are R^{F×T×2}, but Sec. 3.3 writes 'feature with shape of R^{T×F×C}'. Since the modules treat time and frequency differently, please standardize the axis order.","section":"Sec. 3.3"},{"comment":"FINALLY is listed with '—' for PESQ, SDR, LSD, MCD, sBERT, and sTokDis; this incomplete comparison makes it impossible to verify the 'balanced gains' claim against the strongest perceptual baseline. Please add the missing metrics or indicate unavailability in the table.","section":"Table 1"},{"comment":"The entries for FINALLY contain typographical artifacts ('4.05±±±0.07', '3.98± ±±0.06', '3.31± ±±0.04'); please correct these.","section":"Table 8"},{"comment":"The x-axis label reads 'distrance d' instead of 'distance d', and the unit ×10^{-3} appears inconsistent with the text examples of w; please fix the label and clarify the units.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with a novel architectural idea and useful ablations, but the 'arbitrary rates' claim is overstated relative to what is trained and tested, and the metric-leakage issue with Eq. (6) requires a clear response. I would encourage the editor to seek a reviewer with experience in speech enhancement evaluation to assess the PESQ/UTMOS overlap. The authors should be given the opportunity to add unseen-rate experiments and error bars; if they cannot, the claims should be narrowed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Start with the punchline: this is a real step forward for speech restoration architecture, but the paper overclaims in one important place. The claim that one model handles arbitrary input and output sampling rates is not backed by the mechanism or the experiments; the model is tested only on trained rate pairs.\n\nWhat's genuinely new: the asymmetric encoder-decoder, where a heavy TF dual-path encoder analyzes only the input band and a lightweight decoder synthesizes missing high-frequency content via learnable extension queries with cross-self attention, is a good design. The SFI-STFT discriminator is a sensible way to share adversarial supervision across rates. The scaled log-spectral loss is well motivated, and the gradient analysis in the appendix makes the behavior clear. The authors also report streaming results and compute complexity honestly, and the citation pattern is clean—they build on MAE, TF-Locoformer, and FINALLY without ignoring prior work.\n\nThe soft spots are real but manageable. First, PESQ and UTMOS appear in the training loss (Eq. 6) and then as headline metrics in Tables 1–3. That is partial circularity. They mostly mitigate it by also reporting LSD, MCD, sBERT, DNSMOS, and WVMOS, which tell the same broad story, but the overlap should be acknowledged more explicitly, or those metrics dropped from the loss. Second, no error bars in the main tables. They do compute them for the VoxCeleb real-data appendix, so this is a choice, not an oversight. Third, the 'arbitrary rates' claim. Section 3.1 lists 22.05 and 32 kHz as supported, but training uses only f_D in {16,24,44.1,48} and f_E in {8,16}, and all evaluations use these combos. The extension query q_ext is a parameter tensor of dimension (F_D−F_E); nothing says how an unseen length would get a query. If the intended answer is interpolation or rate conditioning, it needs to be stated and tested. Otherwise the claim should be scoped to the trained rates. The super-res test uses the same simulation pipeline as training, which the authors openly note; this is a minor-to-moderate concern, softened by the real-data appendix.\n\nWho gets value from this? Anyone working on universal restoration, sampling-rate-agnostic audio modeling, or adversarial training across resolutions. It deserves a serious referee; the architecture and losses are worth building on. My recommendation: send it out, but require either an unseen-rate experiment or a clearly scoped claim, and ask the authors to address the metric overlap and add error bars to the main tables.","headline":"A genuinely useful architecture and a solid empirical study, but the 'arbitrary rates' claim overreaches and the PESQ/UTMOS training-evaluation overlap needs attention.","tokens_in":22213,"tokens_out":4324,"would_cite":true,"duration_ms":37147,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One asymmetric model restores speech across decoupled input-output sampling rates without resampling.","keywords":["speech restoration","sampling-frequency-independent","bandwidth extension","asymmetric encoder-decoder","extension queries","time-frequency transformer","adversarial training","streaming speech enhancement"],"falsifier":"Evaluate the trained model on input-output rate pairs absent from training, for example 8→22.05 kHz or 32→48 kHz, and compare LSD, MCD, and UTMOS against the same model's results on trained rate pairs; a clear quality collapse on unseen rate ratios would refute the arbitrary-rate claim.","tokens_in":21056,"feed_emoji":"🎙️","tokens_out":8505,"duration_ms":69202,"temperature":0.7,"pith_summary":"This paper claims that speech restoration does not need to be tied to a single input-output sampling rate. It proposes TF-Restormer, a model that encodes only the observed frequency band of a degraded signal and then synthesizes the missing high-frequency band through learned extension queries, so one trained model can denoise, dereverberate, extend bandwidth, and undo combined distortions at multiple input and output rates without resampling. A shared sampling-frequency-independent STFT discriminator and a scaled log-spectral loss allow the model to be trained adversarially across rates while preserving fine spectral detail. If the claim holds, a single non-diffusion model can replace a collection of rate-specific restoration systems, and a streaming variant can do the same in real time.","feed_headline":"One model restores speech across sampling rates","feed_subtitle":"TF-Restormer analyzes only the input band and synthesizes missing highs, beating per-rate baselines.","key_machinery":"The load-bearing object is the decoupled SFI-STFT formulation, in which a fixed 40 ms window with 20 ms hop makes the number of frequency bins scale with the sampling rate, so input and output rates obey $f_E:f_D = (F_E-1):(F_D-1)$. On top of this sits the asymmetric encoder-decoder: a time-frequency dual-path encoder analyzes only the $F_E$ input bins, while a lightweight decoder pads the frequency axis to $F_D$ with learnable extension queries shared across frames and refines them with a frequency cross-self module whose cross-attention uses encoder features as key and value, a band-partitioned cross-attention mechanism. The same frequency projection layer is shared across frequency modules to inject a spectral-structure inductive bias, and the SFI-STFT discriminator applies adversarial supervision with STFT windows of fixed physical duration across all rates. This combination is what lets one model synthesize missing high-frequency content at user-chosen output rates without resampling.","core_discovery":"TF-Restormer's central claim is that a single asymmetric encoder-decoder operating on a sampling-frequency-independent short-time Fourier transform can restore speech at decoupled input-output rates, with no external resampling and no systematic loss of fidelity or perceptual quality relative to rate-specific systems. The encoder performs heavy time-frequency analysis only on the observed input band, while the decoder reconstructs the unobserved high-frequency band from learnable extension queries whose cross-attention keys and values come from the encoder; because the STFT uses a constant frame duration, frequency-bin counts scale with the sampling rates, making the internal representation rate-agnostic. Trained with a perceptual loss, a scaled log-spectral loss, and a shared SFI-STFT discriminator, the same model handles denoising, dereverberation, bandwidth extension, and combined distortions, and its causal time module yields a streaming variant. The reported evaluations show it matching or beating dedicated per-rate baselines on these tasks while keeping compute proportional to the input band rather than the output rate.","pith_inferences":["The 'arbitrary rate' claim is only as strong as the interpolation: the model is trained with output rates {16,24,44.1,48} kHz and inputs {8,16} kHz, so rates such as 22.05 or 32 kHz are an inference. A test on unseen rates would settle whether the ratio identity and shared extension queries generalize.","The same architecture could be pointed at arbitrary spectral holes, not just a high-frequency band: because extension queries are learnable and cross-attend to the observed band, one could mask arbitrary frequency regions and let the decoder inpaint them, turning the model into a general spectral inpainter.","The asymmetric encoder-decoder suggests a separable capacity budget: an application that needs better high-frequency detail for a fixed input rate could enlarge the decoder (or its number of extension queries) without recomputing the encoder, which the ablations hint at by showing decoder design matters mainly when rates differ."],"forward_implications":["A single TF-Restormer model covers the trained rate pairs (8/16 kHz inputs to 16/24/44.1/48 kHz outputs), so a deployment can drop per-rate models and the associated resampling pipeline.","Compute scales with the input band: 8→16 kHz uses roughly half the MACs of 16→16 kHz, because heavy analysis is never applied to bins the input does not contain.","The streaming variant, built by replacing the time module with a causal Mamba block, keeps near-offline quality with roughly 80 ms latency, making real-time universal restoration feasible.","Combined distortions such as noise, reverberation, clipping, codecs, and band limitation can be repaired in one pass at the chosen output rate, instead of chaining a denoiser and a separate bandwidth-extension model.","The shared SFI-STFT discriminator allows adversarial training to be unified across rates, so adding a new supported output rate does not require training a new discriminator."],"supporting_citations":[{"why":"Defines the sampling-frequency-independent STFT formulation that this work extends to decoupled input-output rates.","marker":"(Paulus & Torcoli, 2022)"},{"why":"Supplies the TF dual-path block design (time self-attention with RoPE, frequency self-attention) that TF-Restormer adapts.","marker":"(Saijo et al., 2024)"},{"why":"MAE-style division of labor motivates concentrating heavy processing in the encoder and keeping the decoder lightweight.","marker":"(He et al., 2022)"},{"why":"Cross-self attention mechanism that the frequency decoder module uses to condition extension queries on encoder features.","marker":"(Gupta et al., 2023)"},{"why":"Supplies the perceptual loss, human-feedback loss, and the FINALLY vocoder baseline that the method is compared against.","marker":"(Babaev et al., 2024)"},{"why":"Provides the UNIVERSE benchmark and diffusion-based baseline for general speech restoration evaluation.","marker":"(Serrà et al., 2022)"},{"why":"Provides the VCTK corpus used for training and for the super-resolution evaluation data.","marker":"(Yamagishi et al., 2019)"},{"why":"Supplies the noise and room-impulse-response samples used to simulate degraded training and test inputs.","marker":"(Reddy et al., 2020)"},{"why":"The multi-scale STFT discriminator design that is made sampling-frequency-independent for unified adversarial training.","marker":"(Défossez et al., 2023)"}],"fun_headline_variants":["One model, no resampling, any input-output rate","TF-Restormer: decoupled rates without resampling","Asymmetric encoder-decoder: no resampling, all rates","Query-based model restores speech at any rate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that extension queries trained only on 8/16 kHz inputs and 16/24/44.1/48 kHz outputs will still synthesize correctly at any other user-chosen rate, because the SFI formulation alone does not guarantee that unseen frequency-bin counts and alignments are handled well.","fun_headline_variants_meta":{"raw":{"variants":["One model, no resampling, any input-output rate","TF-Restormer: decoupled rates without resampling","Asymmetric encoder-decoder: no resampling, all rates","Query-based model restores speech at any rate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000831,"raw_usage":{"total_tokens":3622,"prompt_tokens":934,"completion_tokens":2688,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":2621}},"tokens_in":550,"tokens_out":2688,"duration_ms":18536,"temperature":1.0,"reasoning_tokens":2621,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:47:11.764632+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the trained model on input-output rate pairs absent from training, for example 8→22.05 kHz or 32→48 kHz, and compare LSD, MCD, and UTMOS against the same model's results on trained rate pairs; a clear quality collapse on unseen rate ratios would refute the arbitrary-rate claim.","supporting_citations":[{"cited_title":"Germain, Zexu Pan, and Jonathan Le Roux","cited_arxiv_id":null,"evidence_quote":"Supplies the TF dual-path block design (time self-attention with RoPE, frequency self-attention) that TF-Restormer adapts."},{"cited_title":"Siamese masked autoencoders","cited_arxiv_id":null,"evidence_quote":"Cross-self attention mechanism that the frequency decoder module uses to condition extension queries on encoder features."},{"cited_title":"FINALLY : fast and universal speech enhancement with studio-like quality","cited_arxiv_id":null,"evidence_quote":"Supplies the perceptual loss, human-feedback loss, and the FINALLY vocoder baseline that the method is compared against."}],"review_version":1}