REVIEW 3 major objections 6 minor 23 references
Splitting speaker from speech lets neural codecs stream in 660ms blocks
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · glm-5.2
2026-07-07 21:37 UTC pith:S4QF7OEL
load-bearing objection Probing analysis of TiCodec's TIRE module is solid and useful; Dual-TIRE is a reasonable extension with modest gains; but the streaming evaluation in Table 5 has metric values that are internally inconsistent with the rest of the paper and cannot be interpreted as written. the 3 major comments →
Streaming Neural Speech Codecs through Time-Invariant Representations
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central finding is that TiCodec's time-invariant representation, extracted by the TIRE module, primarily encodes acoustic-scene and paralinguistic information while retaining little linguistic content, and that different encoder layers capture complementary aspects of this invariant information. Exploiting this complementarity through Dual-TIRE (two TIRE modules at layers 2 and 3) improves speaker similarity on out-of-domain data from an average of 0.681 to 0.701. Separately, the codec can process audio as contiguous 660ms blocks in a streaming fashion with near-identical reconstruction quality to offline processing (MOS 0.618 vs. 0.621), because the invariant representation provides a稳定
What carries the argument
The TIRE (Time-Invariant Representation Extraction) module extracts a compact global representation from an encoder layer, which is then quantized and replicated along the time axis to condition the decoder. Dual-TIRE extends this by connecting two independent TIRE modules to encoder layers 2 and 3, each quantized separately and reinjected at the corresponding decoder layer. The consistency loss during training pushes TIRE to produce similar representations for two segments from the same source. The streaming mechanism simply processes contiguous 660ms blocks independently and concatenates outputs.
Load-bearing premise
The streaming evaluation assumes that processing contiguous 660ms blocks with no overlap or smoothing is a sufficient test of streaming capability, but since the model was trained on 660ms segments, this test may simply confirm that inference matches the training distribution rather than demonstrating generalization to real streaming conditions with shorter or variable block sizes.
What would settle it
If TIRE representations were shown to encode significant linguistic content (e.g., high accuracy on keyword spotting or language identification), the core premise that factorization cleanly separates invariant from time-varying information would be undermined. Additionally, if streaming inference at block lengths shorter than the training segment produced substantial quality degradation, the streaming capability claim would be weakened.
If this is right
- If factorized codecs can stream at 660ms latency, codec-based speech generation systems (where a language model predicts codec tokens) could operate in real-time conversational settings without buffering entire utterances.
- Separating invariant from time-varying information could reduce the number of tokens a language model must predict per second, since global attributes need not be regenerated at every frame.
- The probing methodology (frozen representations fed to diagnostic classifiers across five tasks) provides a template for auditing what information other neural codec intermediate representations actually encode.
- Layer-dependent training strategies, where different TIRE branches use different segment sampling policies, suggest that multi-level invariant extraction can be tuned for specific attributes like speaker similarity.
Where Pith is reading between the lines
- The streaming test may be too easy: the model was trained on 660ms segments, so block-by-block inference at the same length may simply match the training distribution rather than demonstrate robust generalization to streaming conditions. A stronger test would use block lengths shorter than the training segment or evaluate on continuous audio with varying durations.
- The near-identical streaming and offline metrics (MOS 0.618 vs. 0.621) could indicate that the offline mode gains little from full-utterance context, which would mean either the invariant representation is already sufficient or that the model is not exploiting long-range dependencies.
- The absence of overlap or smoothing at block boundaries means any artifacts would appear as discontinuities; the fact that metrics hold suggests the invariant representation provides enough global conditioning to maintain coherence across blocks, but perceptual evaluation of boundary artifacts is not reported.
- Dual-TIRE's improvement on speaker similarity but degradation on PESQ across all out-of-domain datasets suggests the two TIRE branches may be competing for representational capacity, and a more principled fusion mechanism (e.g., attention-weighted combination) might resolve the trade-off.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper investigates the time-invariant representations (TIRE) of TiCodec through probing tasks, proposes a Dual-TIRE architecture that extracts invariant representations from two encoder layers, and evaluates TiCodec in a streaming inference setting using 660ms processing blocks. The probing analysis shows that TIRE captures acoustic and paralinguistic information but little linguistic content. Dual-TIRE is shown to improve speaker similarity on out-of-domain data. The streaming evaluation claims that block-by-block decoding does not significantly degrade reconstruction quality.
Significance. The probing analysis of TIRE representations across multiple encoder layers and information types is a useful diagnostic contribution. The Dual-TIRE architecture is a reasonable, low-parameter-cost extension. The exploration of segment selection strategies for TIRE training is a novel and practical investigation. However, the streaming evaluation, which is the paper's titular contribution, has verifiability issues that undermine its significance (see major comments).
major comments (3)
- Table 5 (streaming vs. offline, LibriTTS validation) reports metric values that are inconsistent with Table 1 (LibriTTS test-clean) for the same model family. Specifically, PESQ in Table 1 ranges 2.75–2.96, while Table 5 reports PESQ 0.728–0.767 — a ~4× discrepancy. STOI moves in the opposite direction (0.94 in Table 1 vs. 0.99 in Table 5). Additionally, MOS of 0.618 on a conventional 1–5 scale would indicate essentially unusable quality, and SI-SDR of 0.75 dB is near the silence baseline (0 dB). The paper provides no explanation for these discrepancies. If the Table 5 metrics are computed with a different PESQ variant, normalized to [0,1], or otherwise rescaled, the central streaming claim ('no significant degradation') is not verifiable because both offline and streaming modes could score equally poorly on a miscomputed metric. The authors must clarify how each metric in Table 5 was计算,
- Table 4, EMILIA-DE row (TIRE system): the values ViSQOL=4.382, PESQ=2.956, STOI=0.944 are identical to Table 1, Layer 2 row (ViSQOL=4.382, PESQ=2.956, STOI=0.944). This exact match across different datasets (LibriTTS test-clean vs. EMILIA-DE) is implausible and suggests a data-entry or copy-paste error. Since Table 4 supports the claim that Dual-TIRE improves out-of-domain generalization (average Sim 0.701 vs. 0.681), any incorrect row affects the averaged results and the cross-corpus comparison. The authors should verify all values in Table 4.
- Section 5: The streaming evaluation uses 660ms blocks, which is the same segment length used during training. The paper acknowledges this ('Each block lasts 660ms, the same duration as during the training process') but does not discuss whether this constitutes a meaningful test of streaming generalization. Since the model was trained on 660ms segments, block-by-block inference at the same length may simply confirm that inference matches training conditions. The streaming claim would be substantially strengthened by testing with different block sizes, overlapping windows, or smoothing at block boundaries, and by comparing against at least one other codec in streaming mode.
minor comments (6)
- Table 3: The paper states Dual-TIRE 'improves speech reconstruction quality and speaker similarity' (abstract), but Table 3 shows PESQ decreases for both Dual-TIRE variants compared to single-TIRE (2.931 and 2.923 vs. 2.956). The abstract should be revised to reflect the trade-off rather than a uniform improvement.
- Section 4.1: No statistical significance tests are reported for any comparison. Given that many differences in Tables 1–4 are small (e.g., ViSQOL 4.382 vs. 4.387), confidence intervals or significance tests would strengthen the claims.
- Table 4: MCD values for EMILIA subsets (1.5–1.9) are much higher than for VCTK (0.62–0.63) and LibriTTS (0.73). A brief discussion of why MCD differs so dramatically across corpora would help interpretation.
- Figure 3: The y-axis scale and task labels are difficult to read. Consider enlarging or using a table format for the probing results.
- Section 2.2: The choice of layers 2 and 3 for Dual-TIRE is motivated by Table 1, but the paper does not report results for other layer combinations (e.g., layers 1+2, 1+3, 3+4). A brief justification for not testing other combinations would be helpful.
- The paper does not report bitrate information for the codec configurations evaluated, which is relevant for comparing reconstruction quality across systems.
Simulated Author's Rebuttal
We thank the referee for a careful and constructive report. The referee raises three major comments concerning (1) apparent metric inconsistencies between Tables 1 and 5, (2) a suspected copy-paste error in Table 4, and (3) the validity of the streaming evaluation given that the block size matches the training segment length. We address each below. Two of the three comments identify genuine errors that require correction; the third motivates additional experiments that we will incorporate.
read point-by-point responses
-
Referee: Table 5 (streaming vs. offline, LibriTTS validation) reports metric values inconsistent with Table 1 (LibriTTS test-clean). PESQ in Table 1 ranges 2.75–2.96, while Table 5 reports PESQ 0.728–0.767. STOI moves in the opposite direction (0.94 in Table 1 vs. 0.99 in Table 5). MOS of 0.618 on a 1–5 scale would indicate unusable quality, and SI-SDR of 0.75 dB is near silence baseline. The authors must clarify how each metric in Table 5 was computed.
Authors: The referee is correct that the metrics in Table 5 are not directly comparable to those in Table 1, and we acknowledge that the manuscript fails to explain this. After reviewing our experimental code, we can confirm the following: (1) PESQ in Table 5 was computed using the wide-band PESQ implementation from a different library than the one used for Table 1, and the raw scores were normalized to [0,1] — this was not stated in the paper and we will correct it. (2) STOI values near 0.99 in Table 5 reflect the fact that STOI was computed on a per-block basis on 660ms segments, where high values are expected for clean reconstructed speech at short durations; this differs from the utterance-level STOI in Table 1. (3) The MOS values were not obtained from human listeners but are NISQA-style predicted MOS scores; the label 'Mean Opinion Score' without this qualification is misleading and will be corrected. (4) SI-SDR values near 0 dB are unexpectedly low and we are re-examining the computation pipeline for a possible implementation issue. We agree that without clarification of these methodological differences, the streaming claim is not verifiable. In the revision, we will: (a) recompute all Table 5 metrics using the same metric implementations and scales as Table 1 so that direct comparison is possible, (b) clearly state the metric implementations used, and (c) report both offline and streaming results on the same dataset partition with consistent computation. If any metric cannot be recomputed in a compatible manner, we will remove it rather than present an unverifiable comparison. revision: yes
-
Referee: Table 4, EMILIA-DE row (TIRE system): ViSQOL=4.382, PESQ=2.956, STOI=0.944 are identical to Table 1, Layer 2 row. This exact match across different datasets is implausible and suggests a copy-paste error. The authors should verify all values in Table 4.
Authors: The referee is correct. The EMILIA-DE row in the TIRE section of Table 4 contains values that are identical to the Layer 2 row in Table 1, which is indeed implausible across different datasets. This is a data-entry error: the LibriTTS test-clean results were inadvertently copied into the EMILIA-DE row during table preparation. We have re-examined our raw results files and confirmed that the correct EMILIA-DE values for the TIRE system differ from what is reported. We will correct the EMILIA-DE row with the actual values and will re-verify every entry in Table 4 against our experiment logs. We will also recompute the TIRE average row and confirm that the qualitative conclusion (Dual-TIRE improves average speaker similarity) still holds with the corrected values. If the correction changes any reported average or conclusion, we will update the text accordingly. revision: yes
-
Referee: The streaming evaluation uses 660ms blocks, the same segment length used during training. The paper acknowledges this but does not discuss whether this constitutes a meaningful test of streaming generalization. The streaming claim would be strengthened by testing with different block sizes, overlapping windows, or smoothing at block boundaries, and by comparing against at least one other codec in streaming mode.
Authors: We agree that testing only at the training block size is a limitation of the current evaluation. The referee's suggestion is well-taken: demonstrating robustness across different block sizes, particularly sizes not seen during training, would substantially strengthen the streaming claim. In the revision, we will add experiments with at least two additional block sizes (e.g., 330ms and 990ms) to test whether block-by-block decoding degrades when the block size differs from training. We will also add an overlapping-window configuration with simple overlap-add smoothing at block boundaries and report the resulting metrics. Regarding comparison with another codec in streaming mode, we agree this would be informative; we will include EnCodec evaluated under the same block-by-block streaming protocol as a baseline, subject to computational feasibility within the revision period. If a full comparison is not feasible, we will at minimum report EnCodec offline results on the same data to contextualize the absolute quality level of our system. revision: partial
Circularity Check
No circularity found: derivation is self-contained against external benchmarks
full rationale
The paper presents empirical contributions built on TiCodec [9], which is cited as prior work by different authors (Ren et al., ICASSP 2024). The TIRE module, Dual-TIRE architecture, and streaming evaluation are independent extensions. The probing tasks use external datasets (VoxCeleb, TAU, MELD, Common Voice, Speech Commands) and the evaluation corpora include out-of-domain data (VCTK, EMILIA). No step in the derivation chain reduces to its own inputs by construction. The probing classifiers are trained on frozen TIRE representations against external labels, the segment selection strategies are ablated against held-out test sets, and the streaming comparison (Table 5) measures offline vs. streaming on the same model without fitting parameters to the streaming condition. The metric inconsistencies between Tables 1 and 5 (PESQ 2.9 vs 0.73, MOS 0.618) are a correctness risk, not a circularity issue — the streaming claim is not forced by definition or by a self-citation chain. No self-citation load-bearing pattern is present: the central premise rests on externally verifiable experiments, not on a uniqueness theorem or ansatz from the authors' own prior work.
Axiom & Free-Parameter Ledger
free parameters (2)
- TIRE connection layer index =
Layers 2 and 3 for Dual-TIRE
- Segment selection strategy per layer =
Layer 2: random unconstrained; Layer 3: cross-file same speaker same book
axioms (3)
- domain assumption Time-invariant speech information (speaker, environment) can be separated from time-varying content (phonetics) at the encoder level.
- standard math Probing task performance indicates the presence of information in representations.
- ad hoc to paper 660ms blocks are sufficient context for estimating time-invariant representations in streaming mode.
read the original abstract
Neural speech codecs are increasingly used as intermediate representations in codec-based speech generation systems. TiCodec introduces a factorized representation that separates time-varying speech content from time-invariant information through a Time-Invariant Representation Extraction (TIRE) module, potentially reducing the amount of information that must be modeled at the frame-level. In this work, we investigate the nature of the information captured by TIRE representations and their suitability for low-latency speech processing. Using a series of probing tasks, we analyze the influence of the encoder layer and show that intermediate layers capture complementary speaker- and environment-related information while containing little linguistic content. We further study several segment selection strategies for TIRE training and demonstrate that cross-file sampling improves the robustness of invariant representations. Based on these findings, we propose Dual-TIRE, a multi-level architecture that exploits the complementarity of different encoder layers and improves speech reconstruction quality and speaker similarity. Finally, we evaluate TiCodec in a streaming inference setting using successive 660ms processing blocks. Results show that streaming operation can be achieved without significant degradation in reconstruction performance, highlighting the potential of factorized neural codec representations for future low-latency speech generation systems.
Figures
Reference graph
Works this paper leans on
-
[1]
Sound- stream: An end-to-end neural audio codec,
N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Sound- stream: An end-to-end neural audio codec,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 495–507, 2021
work page 2021
-
[2]
High Fidelity Neural Audio Compression
A. D´ efossez, J. Copet, G. Synnaeve, and Y. Adi, “High fidelity neural audio compression,” 2022. [Online]. Available: https://arxiv.org/abs/2210.13438
work page internal anchor Pith review Pith/arXiv arXiv 2022
-
[3]
High-fidelity audio compression with improved rvqgan,
R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,”Advances in Neural Information Pro- cessing Systems, vol. 36, pp. 27 980–27 993, 2023
work page 2023
-
[4]
Audiolm: a language modeling approach to audio generation,
Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Rob- lek, O. Teboul, D. Grangier, M. Tagliasacchiet al., “Audiolm: a language modeling approach to audio generation,”IEEE/ACM transactions on audio, speech, and lan- guage processing, vol. 31, pp. 2523–2533, 2023
work page 2023
-
[5]
Text-free prosody-aware generative spoken language modeling,
E. Kharitonov, A. Lee, A. Polyak, Y. Adi, J. Copet, K. Lakhotia, T.-A. Nguyen, M. Riviere, A. Mohamed, E. Dupouxet al., “Text-free prosody-aware generative spoken language modeling,” inACL, 2022. Streaming Neural Speech Codecs through Time-Invariant Representations 15
work page 2022
-
[6]
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Liet al., “Neural codec language models are zero-shot text to speech synthesiz- ers,”arXiv preprint arXiv:2301.02111, 2023
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[7]
Voicebox: Text-guided multilingual universal speech generation at scale,
M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokaret al., “Voicebox: Text-guided multilingual universal speech generation at scale,”Advances in neural information processing systems, vol. 36, pp. 14 005–14 034, 2023
work page 2023
-
[8]
Naturalspeech 3: zero-shot speech synthesis with factorized codec and diffusion models,
Z. Ju, Y. Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y. Liu, Y. Leng, K. Song, S. Tanget al., “Naturalspeech 3: zero-shot speech synthesis with factorized codec and diffusion models,” inProceedings of the 41st International Conference on Ma- chine Learning, 2024, pp. 22 605–22 623
work page 2024
-
[9]
Fewer-token neural speech codec with time-invariant codes,
Y. Ren, T. Wang, J. Yi, L. Xu, J. Tao, C. Y. Zhang, and J. Zhou, “Fewer-token neural speech codec with time-invariant codes,” inICASSP. IEEE, 2024, pp. 12 737–12 741
work page 2024
-
[10]
Voxceleb: Large-scale speaker verification in the wild,
A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, “Voxceleb: Large-scale speaker verification in the wild,”Computer Speech & Language, vol. 60, p. 101027, 2020
work page 2020
-
[11]
A multi-device dataset for urban acoustic scene classification
A. Mesaros, T. Heittola, and T. Virtanen, “A multi-device dataset for urban acous- tic scene classification,”arXiv preprint arXiv:1807.09840, 2018
work page internal anchor Pith review Pith/arXiv arXiv 2018
-
[12]
Meld: A multimodal multi-party dataset for emotion recognition in conversations,
S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea, “Meld: A multimodal multi-party dataset for emotion recognition in conversations,” in ACL, 2019
work page 2019
-
[13]
Common voice: A massively-multilingual speech corpus,
R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” inProceedings of the twelfth language resources and evaluation conference, 2020, pp. 4218–4222
work page 2020
-
[14]
Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
P. Warden, “Speech commands: A dataset for limited-vocabulary speech recogni- tion,”arXiv preprint arXiv:1804.03209, 2018
work page internal anchor Pith review Pith/arXiv arXiv 2018
-
[15]
Libritts: A corpus derived from librispeech for text-to-speech,
H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu, “Libritts: A corpus derived from librispeech for text-to-speech,”Interspeech, 2019
work page 2019
-
[16]
Librispeech: an asr corpus based on public domain audio books,
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210
work page 2015
-
[17]
Cstr vctk corpus: English multi- speaker corpus for cstr voice cloning toolkit,
C. Veaux, J. Yamagishi, K. MacDonaldet al., “Cstr vctk corpus: English multi- speaker corpus for cstr voice cloning toolkit,”University of Edinburgh. The Centre for Speech Technology Research (CSTR), vol. 6, p. 15, 2017
work page 2017
-
[18]
Emilia: An extensive, multilingual, and diverse speech dataset for large- scale speech generation,
H. He, Z. Shang, C. Wang, X. Li, Y. Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shi et al., “Emilia: An extensive, multilingual, and diverse speech dataset for large- scale speech generation,” inSpoken Language Technology Workshop (SLT). IEEE, 2024, pp. 885–890
work page 2024
-
[19]
Visqol v3: An open source production ready objective speech and audio metric,
M. Chinen, F. S. C. Lim, J. Skoglund, N. Gureev, F. O’Gorman, and A. Hines, “Visqol v3: An open source production ready objective speech and audio metric,”
-
[20]
ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric
[Online]. Available: https://arxiv.org/abs/2004.09584
work page internal anchor Pith review Pith/arXiv arXiv 2004
-
[21]
A. Rix, J. Beerends, M. Hollier, and A. Hekstra, “Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,” inICASSP, 2001
work page 2001
-
[22]
An algorithm for intel- ligibility prediction of time–frequency weighted noisy speech,
C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “An algorithm for intel- ligibility prediction of time–frequency weighted noisy speech,”IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 7, pp. 2125–2136, 2011
work page 2011
-
[23]
Mel-cepstral distance measure for objective speech quality assess- ment,
R. Kubichek, “Mel-cepstral distance measure for objective speech quality assess- ment,” inProceedings of IEEE Pacific Rim Conference on Communications Com- puters and Signal Processing, vol. 1, 1993, pp. 125–128 vol.1
work page 1993
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.