Pith. sign in

REVIEW 4 major objections 5 minor 37 references

HASRD: Hierarchical Acoustic and Semantic Representation Disentanglement

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Speech SSL representations separate into semantic and acoustic token streams, preserving ASR while halving bitrate.

desk verdict Reconstruction results are credible, but the headline 44% WER gain comes from an asymmetric comparison and does not survive on quantized tokens. read the letter →

arxiv 2506.00843 v1 pith:UYSY7OEG submitted 2025-06-01 eess.AS cs.SD

Amir Hussein , Sameer Khurana , Gordon Wichern , Francois G. Germain , Jonathan Le Roux This is my paper
classification eess.AScs.SD
keywords self-supervisedlearningspeechtokenizationresidualvectorquantizationspokenlanguagemodelsautomaticrecognitionneuralaudiocodecsrepresentationdisentanglement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HASRD proposes that self-supervised speech representations can be factorized into two complementary discrete streams: a semantic stream carried by the first residual quantizer codebook, and an acoustic stream carried by the remaining codebooks. The paper claims this separation preserves automatic speech recognition accuracy while delivering high-quality waveform reconstruction at roughly half the bitrate of SpeechTokenizer (3.1 vs 6.0 kbps), and that a more efficient CNN pre-encoder improves both speed and WER. Compared with SpeechTokenizer, HASRD reports a 44% relative WER improvement on continuous encoder features (12.0% vs 21.5%) while also improving mel-spectrogram distance from 0.76 to 0.64. If correct, a single tokenizer could serve both spoken-language modeling, which wants semantic tokens, and synthesis, which wants acoustic detail, removing the need for separate codecs.

What carries the argument

The central mechanism is residual vector quantization over SSL latents with a semantic first codebook: a convex combination of SSL layer representations is learned, the best semantic layer (layer 8) is k-means quantized as $\widetilde{\mathbf{H}}_s$, the acoustic residual is computed as $\mathbf{H}_a = \mathbf{H}\mathbf{W} - \widetilde{\mathbf{H}}_s$ with a learnable projection $\mathbf{W}$, and the final quantized representation is $\widehat{\mathbf{H}} = \widetilde{\mathbf{H}}_s + \sum_{m=1}^{M-1} \widetilde{\mathbf{H}}_{a,m}$. The efficient front-end uses fbank input, depthwise separable convolutions, and random-projection (BestRQ) pre-training, while reconstruction relies on the DAC-style discriminators and losses.

What would settle it

Reconstruct speech using only the acoustic RVQ codebooks and decode it with a strong ASR model; if the residual is semantic-free the WER should be near chance, whereas the paper's own acoustic-only WER of 70.2% suggests a stronger test would reveal how much semantic leakage remains.

Watch

Extended reading notes

Core claim

The paper claims that SSL latent layers contain separable semantic and acoustic information, and that this separation can be made operational by quantizing one chosen layer with k-means to form the first RVQ codebook, then quantizing the residual $\mathbf{H}_a = \mathbf{H}\mathbf{W} - \widetilde{\mathbf{H}}_s$ with subsequent codebooks. On LibriSpeech test-clean, semantic-only tokens reach 21.0% WER, better than SpeechTokenizer's quantized 23.0%, while the full codebook stack reconstructs speech with mel distance 0.64 at 3.1 kbps versus SpeechTokenizer's 0.76 at 6.0 kbps. Speaker-similarity measurements support the intended split: semantic-only tokens give low speaker similarity (0.15) and good ASR, while acoustic-only tokens give high speaker similarity (0.67) and poor ASR, indicating that speaker identity lives in the acoustic codebooks and linguistic content in the first codebook.

Load-bearing premise

The factorization is clean only if subtracting the k-means quantized semantic layer from the projected SSL representation leaves a residual that contains no semantic content; if the residual retains word or phone information, the two codebooks are not truly disentangled.

Editorial extensions

If this is right

  • A single tokenizer can serve both semantic tasks and high-quality synthesis: use the first codebook for ASR or language modeling and the full codebook stack for waveform reconstruction.
  • At 3.1 kbps with better reconstruction than SpeechTokenizer at 6.0 kbps, storage and transmission costs for spoken-language-model training drop roughly twofold.
  • The learned layer weights reveal which SSL layers encode semantic versus acoustic content, making SSL representations more interpretable and guiding layer selection for downstream tasks.
  • Because HASRD with a HuBERT teacher retains its teacher's WER rather than degrading during distillation, the approach may generalize to stronger SSL teachers without losing semantic performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, if the acoustic codebooks truly carry speaker identity, swapping them between two utterances while keeping the same semantic codebook should perform one-shot voice conversion; the speaker-similarity results are consistent with this, but the paper does not test it.
  • The paper's own Table 4 shows that full tokens (semantic plus acoustic) give worse ASR WER than semantic-only tokens (24.2 vs 21.0), suggesting that downstream speech models should not blindly consume all codebooks for recognition tasks; this design rule is an editorial consequence, not stated by the authors.
  • A direct probe of the residual is testable: reconstruct speech from acoustic-only codebooks and run ASR; their 70.2% WER is far above chance but not maximal, so a stronger recognizer would quantify whether any semantic leakage remains in the acoustic codebooks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HASRD, a two-stage framework that factorizes SSL speech representations into discrete semantic and acoustic tokens. In the first stage, a CNN-Conformer encoder is pre-trained with a BestRQ-style masked language modeling objective using random-projection quantization; in the second stage, the frozen encoder's layer representations are combined with learned weights, a k-means quantized semantic layer becomes the first RVQ codebook, and subsequent RVQ codebooks encode the acoustic residual via a reconstruction objective. The authors report improved computational efficiency of the encoder, better reconstruction quality than SpeechTokenizer at lower bitrate, and improved ASR WER, along with an analysis of acoustic/semantic disentanglement using speaker similarity and ASR.

Significance. If fully substantiated, the method would be a valuable contribution to spoken language model tokenization: it aims to preserve semantic performance while adding acoustic fidelity, with interpretable codebook roles and a 2x bitrate reduction over SpeechTokenizer. The paper's strengths include a systematic set of ablations on LibriSpeech (100h and 960h), evaluation with external ASR, ViSQOL, and speaker-verification metrics, and a clear visual and quantitative analysis of layer-wise contributions to acoustic versus semantic objectives. The efficiency improvements to the DAC-style encoder are believable and potentially useful independently. However, the headline quantitative claim about WER improvement is not supported by the numbers as reported for discrete tokens, and the full token sequence behaves differently than the semantic-only token sequence, which is central to the paper's premise for SLM use.

major comments (4)
  1. [Abstract / Table 3] The abstract's claim of a 44% relative WER improvement over SpeechTokenizer is not supported by the quantized token comparison. In Table 3, the quantized rows show HASRD (BestRQ+) at 21.0 WER versus SpeechTokenizer at 23.0, which is a 8.7% relative improvement; the 44% figure matches only the continuous latent row (12.0 vs. 21.5). Since spoken language models consume discrete tokens, the abstract should report the quantized comparison or explicitly state that the 44% figure refers to continuous latent features.
  2. [Table 4 / Section 4.3] Table 4 reports that the combined semantic+acoustic token sequence yields WER 24.2, which is worse than both HASRD's semantic-only WER of 21.0 and SpeechTokenizer's full-token WER of 23.0. This directly weakens the claim that HASRD improves ASR while adding acoustic fidelity for the token set an SLM would actually consume. Please report ASR on the exact token set used by an SLM (e.g., all codebooks) and analyze why adding acoustic codebooks degrades WER relative to semantic-only tokens.
  3. [Section 2.2, Eq. (2)] The acoustic residual Ha = HW - H_s assumes that subtracting the k-means quantized semantic term removes essentially all semantic content. The results in Table 4, where acoustic-only tokens achieve 70.2% WER, indicate that substantial semantic information remains in the acoustic tokens, so the factorization is not clean. Provide a quantitative leakage analysis, for example ASR on acoustic-only tokens with a controlled training setup or a probing task, and discuss the consequence for the disentanglement claim.
  4. [Table 3 and Section 3 (Evaluation)] The evaluation protocol is not fully symmetric across models. The Table 3 caption states that for HASRD only the encoder is used for ASR, but it does not specify which token set is used for the quantized rows of SpeechTokenizer or for HASRD. If the HASRD quantized row uses only the first codebook while SpeechTokenizer is evaluated on its full RVQ output, the comparison is asymmetric and the reported improvement is not meaningful for downstream SLM use. Please specify the token set used for every row and, if necessary, retrain the ASR model on matching token sets.
minor comments (5)
  1. [Section 4.2, text after Table 3] The text says 'Increasing the attention dimension from 516 to 768' but the encoder description in Section 3 states an attention dimension of 512; also 'HASARD' appears to be a typo for 'HASRD'.
  2. [Table 1] The table reports inference speedups of 2x and 3x and a GPU memory reduction of 25%, but the text only says 'inference speed also improves by 2x on CPU'; please clarify the hardware and measurement conditions for all speedup numbers.
  3. [Section 2.2, Eq. (2)] The notation for the quantized semantic representation is inconsistent: H_s is used for the k-means output in the text, while Eq. (2) uses a hat notation (H^s); please unify the notation.
  4. [Section 3, evaluation setup] The statement that the semantic layer index n_s = 8 was selected as the best performing layer for ASR is a legitimate design choice, but it should be clearly acknowledged in the text as a free parameter selected on the dev set (which the paper does later); please make this explicit in Section 3 as well.
  5. [Table 2] The comparison with SpeechTokenizer is at different bitrates (3.1 vs. 6.0 kbps); while this is favorable to HASRD, it would strengthen the paper to include a same-bitrate comparison or to discuss the rate-distortion tradeoff more explicitly.

Circularity Check

1 steps flagged · score 1.0 of 10

No load-bearing circularity; the central HASRD claims are grounded in external ASR, ViSQOL, and speaker-similarity benchmarks. The only mild self-definitional element is that the 'semantic' layer is chosen by ASR performance and then ASR is used to confirm semantic quality.

  1. self definitional [Section 2.2 (Eq. 2) and Section 4.3 (Table 4)]
    "We identify the index ns ∈ {1, . . . , N} of the best performing representation for the ASR task, and define the corresponding representation Hs = Hns as the semantic representation. ... semantic tokens yield lower speaker similarity (0.15) but better ASR performance (21% WER). This confirms that our approach effectively disentangles semantic content (first codebook) from speaker information (acoustic codebooks)."

    The semantic codebook is defined as the SSL layer with the best ASR WER, so the finding that semantic tokens perform well on ASR is partly a restatement of the selection criterion rather than an independent confirmation. The acoustic residual is defined as the remainder after subtracting this ASR-selected codebook, so the semantic/acoustic ASR contrast in Table 4 is shaped by construction. This is a minor labeling tautology: it does not drive the headline comparisons against SpeechTokenizer, which are external benchmarks, and ns was selected on dev-clean while final WER is reported on test-clean.

full rationale

The paper's core comparisons (quantized and latent WER against SpeechTokenizer/BestRQ, mel/STFT/ViSQOL reconstruction metrics, bitrate, speaker similarity) are empirical evaluations against external baselines and are not fitted inputs renamed as predictions. No load-bearing self-citations appear: the cited prior work (BestRQ, DAC, SpeechTokenizer, HuBERT) is external, and no uniqueness theorem or author-previous-work premise is used to force the architecture. The reconstruction objective and Eq. (2)-(3) define a factorization, but the paper does not claim to derive ASR quality from that factorization; ASR is measured downstream. The only mildly circular element is the labeling of Hs as 'semantic' by ASR selection and then using ASR as evidence of semantic content, which is a feature-selection tautology rather than a statistically forced prediction. The skeptic's concern about the '44% relative WER improvement' figure is a benchmarking/reporting issue (continuous-latent vs quantized-token rows), not a circularity, so it does not raise the circularity score. Overall the derivation chain is self-contained against external benchmarks; score 1 reflects the minor definitional labeling issue only.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities beyond learned codebooks and a residual representation; these are standard components of RVQ codecs and SSL feature extraction.

free parameters (4)
  • Semantic layer index n_s = 8
    Selected as the layer with best ASR performance on dev-clean; the entire semantic branch and the headline WER results depend on this choice.
  • Number of RVQ stages = 9 (1 k-means semantic + 8 acoustic)
    Chosen by the authors; this sets the 3.1 kbps bitrate and directly affects both reconstruction quality and ASR results.
  • Learned projection W and convex weights alpha_n = learned during training
    These define the acoustic residual in Eq. (2); the disentanglement claim depends on their learned values.
  • k-means codebook size = not stated
    The offline semantic codebook is trained on 10% of train-clean-100, but its vocabulary size is never given, which affects bitrate and ASR.
assumptions (3)
  • domain assumption SSL representations can be factorized into semantic and acoustic components by linear combination and residual subtraction.
    Section 2.2 states this hypothesis and Eq. (2) relies on it; if the linear projection cannot separate the two, the disentanglement claim fails.
  • domain assumption k-means quantization of the selected SSL layer preserves semantic information well enough for ASR.
    The first codebook is formed by k-means on layer 8; the paper provides only experimental evidence on LibriSpeech, not a guarantee.
  • ad hoc to paper Frozen SSL encoder features remain valid for reconstruction while the decoder and RVQ codebooks are trained.
    The encoder is frozen during continued training; if SSL features lack acoustic detail, reconstruction quality would degrade, yet the method assumes they contain enough information.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HASRD: Hierarchical Acoustic and Semantic Representation Disentanglement." pith.science (2026). https://pith.science/paper/UYSY7OEG

@misc{pith2026250600843,
  author       = {Pith},
  title        = {Pith review of: HASRD: Hierarchical Acoustic and Semantic Representation Disentanglement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UYSY7OEG}},
  note         = {Machine review of arXiv:2506.00843}
}
read the original abstract

Effective speech representations for spoken language models must balance semantic relevance with acoustic fidelity for high-quality reconstruction. However, existing approaches struggle to achieve both simultaneously. To address this, we introduce Hierarchical Acoustic and Semantic Representation Disentanglement (HASRD, pronounced `hazard'), a framework that factorizes self-supervised learning representations into discrete semantic and acoustic tokens. HASRD assigns the semantic representation to the first codebook, while encoding acoustic residuals in subsequent codebooks. This preserves ASR performance while achieving high-quality reconstruction. Additionally, we enhance HASRD's encoder efficiency, improving ASR performance without compromising reconstruction quality. Compared to SpeechTokenizer, HASRD achieves a 44% relative WER improvement, superior reconstruction quality, and 2x lower bitrate, demonstrating its effectiveness in disentangling acoustic and semantic information.

Figures

Figures reproduced from arXiv: 2506.00843 by the authors.

Figure 1
Figure 1. Proposed HASRD framework for hierarchical disentangling of acoustic and semantic representations objective and 2) continued training with residual vector quan￾tization (RVQ) and reconstruction objective. This framework enables good word error rate (WER) performance, strong com￾pression, and high-quality reconstruction. 2.1. Semantic Pre-Training We adopt SSL pre-training with an MLM objective. Our CNN encoder design… view at source ↗
Figure 2
Figure 2. Weights of latent representations of SSL in acoustic (reconstruction) and semantic (ASR) tasks. effective, yielding a WER of 92%. Following the methodol￾ogy in [24], we first reproduce the BestRQ model and introduce key improvements. Specifically, we replace the 2D-CNN pre￾encoder with our optimized 1D-CNN pre-encoder from Sec￾tion 4.1, with modifications on the encoder from Section 3. These modifications result in … view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 27 canonical work pages

  1. [1]

    Introduction The remarkable performance of large language models has in- spired the development of spoken language models (SLMs) that leverage discrete speech representations [1–3]. SLM use cases range from cross-modal conversational abilities, enabling them to understand and generate multimodal content, to vari- ous speech-related tasks such as automatic...

    work page 2025

  2. [2]

    Proposed Approach For discrete speech representations to be effective for SLMs, they must capture semantic information while preserving acous- tic details to enable high-quality speech synthesis. However, these objectives often conflict, as enhancing one can degrade the other, making it challenging to learn a single representation that balances both, as i...

    work page

  3. [3]

    This framework enables good word error rate (WER) performance, strong com- pression, and high-quality reconstruction

    SSL pretraining with a masked language modeling (MLM) arXiv:2506.00843v1 [eess.AS] 1 Jun 2025 Speech Fbank ConformerMasking Codebook CE lossProjection CNN Encoder (a) BestRQ pretraining Speech Fbank Conformer Discriminator Frozen SSL W CNN Encoder CNN Decoder K- Means R VQ Hierarchical Disentanglement - (b) Continued training with reconstruction objective...

    arXiv 2025

  4. [4]

    For acoustic training, we adopt the DAC framework [16], extracting random 5-second segments (vs

    Experimental Setup Data & Pre-processing: We train our models on the Lib- riSpeech dataset [27], following [5]. For acoustic training, we adopt the DAC framework [16], extracting random 5-second segments (vs. 0.38 s in DAC) from each utterance, zero-padding shorter ones for uniform batch sizes. This extended duration captures richer context and improves s...

    work page

  5. [5]

    Results 4.1. Improved Computational Efficiency of DAC We conducted ablation studies on 100 hours of LibriSpeech to evaluate the impact of our enhanced CNN encoder on re- construction quality and efficiency using fbank inputs. Unlike previous DAC models trained on 0.38-second audio segments, we found that increasing the window size improves contextual mode...

    work page

  6. [6]

    Our approach effectively preserves semantic performance for ASR while achieving reconstruction quality comparable to state-of-the-art neural audio codecs like DAC

    Conclusion In this paper, we proposed a novel framework for learning hi- erarchical disentangled acoustic and semantic representations. Our approach effectively preserves semantic performance for ASR while achieving reconstruction quality comparable to state-of-the-art neural audio codecs like DAC. Additionally, we introduced an efficient CNN pre-encoder ...

    work page

  7. [7]

    Comparing dis- crete and continuous space LLMs for speech recognition,

    Y . Xu, S.-X. Zhang, J. Yu, Z. Wu, and D. Yu, “Comparing dis- crete and continuous space LLMs for speech recognition,”arXiv preprint arXiv:2409.00800, 2024

    arXiv 2024

  8. [8]

    AudioLM: A language modeling approach to audio generation,

    Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasac- chi, et al., “AudioLM: A language modeling approach to audio generation,” IEEE/ACM Trans. Audio, Speech, Lang. Process. , vol. 31, pp. 2523–2533, 2023

    work page 2023

Show all 37 references
  1. [9]

    A survey on speech large language models,

    J. Peng, Y . Wang, Y . Xi, X. Li, X. Zhang, and K. Yu, “A survey on speech large language models,” arXiv preprint arXiv:2410.18908, 2024

    2024

  2. [10]

    SpeechGPT: Empowering large language models with in- trinsic cross-modal conversational abilities,

    D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y . Zhou, and X. Qiu, “SpeechGPT: Empowering large language models with in- trinsic cross-modal conversational abilities,” in Proc. EMNLP, 2023, pp. 15 757–15 773

    2023

  3. [11]

    SpeechTok- enizer: Unified speech tokenizer for speech large language mod- els,

    X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “SpeechTok- enizer: Unified speech tokenizer for speech large language mod- els,” arXiv preprint arXiv:2308.16692, 2023

    2023 arXiv

  4. [12]

    Exploring speech recognition, translation, and understanding with discrete speech units: A comparative study,

    X. Chang, B. Yan, K. Choi, J.-W. Jung, Y . Lu, S. Maiti, R. Sharma, J. Shi, J. Tian, S. Watanabe, et al., “Exploring speech recognition, translation, and understanding with discrete speech units: A comparative study,” inProc. ICASSP, 2024, pp. 11 481– 11 485

    2024

  5. [13]

    HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhut- dinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Trans. Audio, Speech, Lang. Process. , vol. 29, pp. 3451–3460, 2021

    2021

  6. [14]

    WavLM: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, et al., “WavLM: Large-scale self- supervised pre-training for full stack speech processing,” IEEE J. Sel. Top. Signal Process., vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  7. [15]

    W2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre- training,

    Y .-A. Chung, Y . Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y . Wu, “W2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre- training,” in Proc. ASRU, 2021, pp. 244–250

    2021

  8. [16]

    Ex- ploration of efficient end-to-end ASR using discretized input from self-supervised learning,

    X. Chang, B. Yan, Y . Fujita, T. Maekaku, and S. Watanabe, “Ex- ploration of efficient end-to-end ASR using discretized input from self-supervised learning,” in Proc. Interspeech, vol. 2023, 2023, pp. 1399–1403

    2023

  9. [17]

    Speech resynthesis from discrete disentangled self-supervised representations,

    A. Polyak, Y . Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux, “Speech resynthesis from discrete disentangled self-supervised representations,” in Proc. Interspeech, 2021, pp. 3615–3619

    2021

  10. [18]

    Ex- presso: A benchmark and analysis of discrete expressive speech resynthesis,

    T. A. Nguyen, W.-N. Hsu, A. d’Avirro, B. Shi, I. Gat, M. Fazel- Zarani, T. Remez, J. Copet, G. Synnaeve, M. Hassid,et al., “Ex- presso: A benchmark and analysis of discrete expressive speech resynthesis,” in Proc. Interspeech, 2023, pp. 4823–4827

    2023

  11. [19]

    SoundStream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “SoundStream: An end-to-end neural audio codec,” IEEE/ACM Trans. Audio, Speech, Lang. Process. , vol. 30, pp. 495–507, 2021

    2021

  12. [20]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438 , 2022

    2022 arXiv

  13. [21]

    Moshi: A speech-text foundation model for real-time dialogue,

    A. D ´efossez, L. Mazar´e, M. Orsini, A. Royer, P. P´erez, H. J´egou, E. Grave, and N. Zeghidour, “Moshi: A speech-text foundation model for real-time dialogue,”arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  14. [22]

    High-fidelity audio compression with improved rvqgan,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Ku- mar, “High-fidelity audio compression with improved rvqgan,” in Proc. NeurIPS, 2023, pp. 27 980–27 993

    2023

  15. [23]

    Codec does matter: Exploring the semantic shortcoming of codec for audio language model,

    Z. Ye, P. Sun, J. Lei, H. Lin, X. Tan, Z. Dai, Q. Kong, J. Chen, J. Pan, Q. Liu, et al., “Codec does matter: Exploring the semantic shortcoming of codec for audio language model,”arXiv preprint arXiv:2408.17175, 2024

    2024 arXiv

  16. [24]

    SKQVC: One-shot voice con- version by k-means quantization with self-supervised speech representations,

    Y . Sim, J. Yoon, and Y .-J. Suh, “SKQVC: One-shot voice con- version by k-means quantization with self-supervised speech representations,” arXiv preprint arXiv:2411.16147, 2024

    2024 arXiv

  17. [25]

    MMM: Multi-layer multi-residual multi-stream discrete speech repre- sentation from self-supervised learning model,

    J. Shi, X. Ma, H. Inaguma, A. Sun, and S. Watanabe, “MMM: Multi-layer multi-residual multi-stream discrete speech repre- sentation from self-supervised learning model,” in Proc. Inter- speech, 2024, pp. 2569–2573

    2024

  18. [26]

    Towards universal speech discrete tokens: A case study for ASR and TTS,

    Y . Yang, F. Shen, C. Du, Z. Ma, K. Yu, D. Povey, and X. Chen, “Towards universal speech discrete tokens: A case study for ASR and TTS,” in Proc. ICASSP, 2024, pp. 10 401–10 405

    2024

  19. [27]

    ReVISE: Self-supervised speech resynthesis with visual input for univer- sal and generalized speech regeneration,

    W.-N. Hsu, T. Remez, B. Shi, J. Donley, and Y . Adi, “ReVISE: Self-supervised speech resynthesis with visual input for univer- sal and generalized speech regeneration,” in Proc. CVPR, 2023, pp. 18 796–18 806

    2023

  20. [28]

    Self-supervised disentan- gled representation learning for robust target speech extraction,

    Z. Mu, X. Yang, S. Sun, and Q. Yang, “Self-supervised disentan- gled representation learning for robust target speech extraction,” in Proc. AAAI, vol. 38, 2024, pp. 18 815–18 823

    2024

  21. [29]

    A ConvNet for the 2020s,

    Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” in Proc. CVPR , 2022, pp. 11 976–11 986

    2022

  22. [30]

    Self-supervised learning with random-projection quantizer for speech recogni- tion,

    C.-C. Chiu, J. Qin, Y . Zhang, J. Yu, and Y . Wu, “Self-supervised learning with random-projection quantizer for speech recogni- tion,” in Proc. ICML, 2022, pp. 3915–3924

    2022

  23. [31]

    Mel- GAN: Generative adversarial networks for conditional wave- form synthesis,

    K. Kumar, R. Kumar, T. De Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo, A. De Brebisson, Y . Bengio, and A. C. Courville, “Mel- GAN: Generative adversarial networks for conditional wave- form synthesis,” Proc. NeurIPS, vol. 32, 2019

    2019

  24. [32]

    HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,” Proc. NeurIPS, vol. 33, pp. 17 022–17 033, 2020

    2020

  25. [33]

    Lib- riSpeech: An ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- riSpeech: An ASR corpus based on public domain audio books,” in Proc. ICASSP, 2015, pp. 5206–5210

    2015

  26. [34]

    Lhotse: A speech data representation library for the modern deep learning ecosystem,

    P. ˙Zelasko, D. Povey, J. Trmal, S. Khudanpur, et al., “Lhotse: A speech data representation library for the modern deep learning ecosystem,” in NeurIPS Data-Centric AI Workshop, 2021

    2021

  27. [35]

    Open implementation and study of BEST-RQ for speech processing,

    R. Whetten, T. Parcollet, M. Dinarelli, and Y . Est `eve, “Open implementation and study of BEST-RQ for speech processing,” arXiv preprint arXiv:2405.04296, 2024

    2024 arXiv

  28. [36]

    Con- nectionist temporal classification: Labelling unsegmented se- quence data with recurrent neural networks,

    A. Graves, S. Fern ´andez, F. Gomez, and J. Schmidhuber, “Con- nectionist temporal classification: Labelling unsegmented se- quence data with recurrent neural networks,” in Proceedings of the 23rd international conference on Machine learning , 2006, pp. 369–376

    2006

  29. [37]

    ECAPA- TDNN: Emphasized channel attention, propagation and aggre- gation in TDNN based speaker verification,

    B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA- TDNN: Emphasized channel attention, propagation and aggre- gation in TDNN based speaker verification,” in Proc. Inter- speech, 2020, pp. 3830–3834

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.