Pith. sign in

REVIEW 3 major objections 5 minor 65 references

AuriStream-1B, trained only to predict the next cochlear token, reports state-of-the-art lexical semantic similarity (12.52 sSIMI) on the ZeroSpeech benchmark.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Autoregressive prediction over discrete cochlear tokens yields a speech representation that beats prior self-supervised models on lexical-semantic similarity and is competitive on SUPERB tasks.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A genuinely novel two-stage speech representation design with a solid autoregressive objective, but the 'state-of-the-art lexical semantics' claim is statistically unsupported—send to peer review with requests for error bars and dev-based tuning. the 3 major comments →

arxiv 2508.11598 v1 pith:EWVULQQL submitted 2025-08-15 cs.CL cs.SDeess.AS

Representing Speech Through Autoregressive Prediction of Cochlear Tokens

Greta Tuckute , Klemen Kotar , Evelina Fedorenko , Daniel L.K. Yamins This is my paper
classification cs.CL cs.SDeess.AS
keywords AuriStreamWavCochcochlear tokensautoregressive predictionlexical semanticssSIMISUPERBspeech representation learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AuriStream asks whether a single, simple objective—predicting the next discrete token in a sequence derived from a model of the human cochlea—can produce speech representations rich enough for phoneme, word, and meaning-level tasks. The paper argues yes: a 1-billion-parameter Transformer trained only to predict upcoming cochlear tokens outperforms all compared models on the ZeroSpeech lexical-semantic similarity benchmark (12.52 on LibriSpeech audio, 10.64 on synthetic), and is competitive with much more hand-engineered models on the SUPERB suite. The claim matters because it suggests that the kind of prediction the auditory system itself might perform, over a biologically inspired time-frequency code, is sufficient to bootstrap word meanings without transcript supervision, contrastive sampling, or masked-prediction heuristics.

Core claim

The paper's central discovery is that autoregressive next-token prediction over cochlear tokens yields state-of-the-art lexical semantic representations. WavCoch, a causal encoder-decoder, converts a 16 kHz waveform into a cochleagram (211 frequency bins at 5 ms resolution) and compresses its internal embedding through a 13-bit lookup-free quantization bottleneck, producing one of 8,192 cochlear tokens per 5 ms step—about 200 tokens per second. AuriStream-1B, a 48-layer GPT-style Transformer, is then trained to predict the next cochlear token on 60k hours of unlabeled speech. From its embeddings, linear probes decode phonemes at 0.88 weighted accuracy and words at 0.65; on the sSIMI lexical

What carries the argument

The load-bearing machinery is the two-stage pipeline: WavCoch's 'Transformation Imitation'—training a network to transform the waveform into a different, biologically motivated representation (the cochleagram) rather than reconstructing its input—through a 13-bit LFQ bottleneck that discretizes the stream into 8,192 cochlear tokens; and AuriStream, a GPT-style autoregressive Transformer trained to predict the next token in that stream. The discrete tokens are what couple the two stages: they are compact enough (about 200 tokens per second) to fit a 4,096-token context window (roughly 20 seconds of speech), yet their cochleagram origin lets predictions be decoded back into audio.

Load-bearing premise

If WavCoch's 13-bit discrete cochlear tokens discard the phonetic or lexical information needed to tell words apart, then AuriStream's autoregressive training cannot recover it, and the word-semantics results would not follow.

What would settle it

Train an identically sized AuriStream on tokens from a mel-spectrogram-based WavCoch (the paper's own Appendix 7.3 variant) or on shuffled cochlear tokens, keeping data and compute fixed; if either reaches a comparable sSIMI score on the LibriSpeech audio subset, the claim that predicting cochlear tokens specifically drives lexical semantics is refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the claim holds, transcript-free speech encoders do not need masked prediction or contrastive learning: plain causal next-token prediction over a cochlear code is enough to reach top lexical-semantic scores.
  • AuriStream-1B's sSIMI lead over HuBERT-xl and WavLM-large on both natural and synthetic audio suggests that no global clustering or noise-augmentation heuristic is necessary for word-meaning representations.
  • The same frozen backbone supports competitive performance on SUPERB tasks spanning content, speaker, semantics, paralinguistics, and generation, so one representation can serve diverse downstream uses.
  • Since performance rises with model scale (100M to 1B) across phoneme decoding, word decoding, and sSIMI, further scaling is a direct path to stronger representations under this objective.
  • Because cochlear tokens can be inverted into audio, AuriStream doubles as a generator; its long-horizon continuations degrade, but short completions of phoneme prompts are plausible, offering an interpretable window into what the model predicts.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the result is driven by the cochlear token bottleneck rather than by scale, then the same two-stage recipe—biologically motivated discrete targets plus autoregressive prediction—could transfer to other sensory modalities, e.g., predicting tokens derived from retinal or cortical representations.
  • The English-only, read-speech training regime leaves open whether the sSIMI advantage comes from language-general pressures or from English-specific statistics; testing on non-English or code-switched data would separate those.
  • The mel-spectrogram variant in Appendix 7.3 is a natural control: if a mel-target WavCoch under the same quantization and training yields comparable sSIMI, then the cochleagram choice is a convenient metaphor rather than the active ingredient.
  • The paper's own observation that long completions drift suggests that a single autoregressive objective may need explicit long-range memory or recurrence before it can serve as a full speech language model, even though short-range word structure emerges.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces AuriStream, a two-stage self-supervised speech representation model. The first stage, WavCoch, converts raw audio into a cochleagram and then into discrete 'cochlear tokens' via a 13-bit LFQ bottleneck; the second stage is a GPT-style Transformer trained to autoregressively predict the next cochlear token. The authors evaluate linear phoneme and word decoding on TIMIT, lexical semantic similarity on the ZeroSpeech 2021 sSIMI benchmark, and six SUPERB tasks. They report state-of-the-art sSIMI scores for AuriStream-1B (12.52 natural, 10.64 synthetic) and competitive SUPERB performance, together with qualitative demonstrations of cochleagram-based audio continuation and inversion.

Significance. If the empirical claims hold, this is a useful contribution: it shows that a simple autoregressive prediction objective over a biologically inspired, discrete auditory representation can yield strong lexical semantic and downstream-task representations without reconstruction, contrastive, or masked-prediction objectives. The release of model weights and the transparent discussion of limitations (English-only, read speech, degradation over long generations) are strengths. The 'Transformation Imitation' framing for WavCoch is a clear conceptual distinction from standard neural codecs. However, the headline 'state-of-the-art lexical semantics' claim is currently supported by small numeric margins without statistical quantification, and the evaluation pipeline contains test-set-dependent hyperparameter choices. These issues are fixable but must be addressed before the central claim is reliable.

major comments (3)
  1. [Section 2.3.3] The claim of state-of-the-art lexical semantics is not statistically supported. No confidence intervals, bootstrap intervals, or significance tests are reported for any Table 2 comparison. On the natural subset the lead over WavLM-large is about 2 sSIMI points (12.52 vs 10.50), and on the synthetic subset the lead is only 0.27 points (10.64 vs 10.37). Since sSIMI is a Spearman correlation computed over a few hundred word pairs and then multiplied by 100, these gaps are well within the sampling noise of the metric. The synthetic subset, which is the cleaner isolated-word test, shows AuriStream essentially tied with HuBERT-xl and WavLM-large. The same lack of uncertainty affects Table 3, where differences such as KS 95.25 vs WavLM-base 96.79 may also be within decoder training noise. The paper should report bootstrap CIs and/or paired significance tests for the AuriStream-vs-baseline diffe
  2. [Appendix 7.2] The best probing layer and pooling operation for phoneme/word decoding are selected by evaluating on 'a subset of the TIMIT test set (the top 10 phonemes/words in the TIMIT test set)'. This is test-set contamination: the reported Table 1 numbers are then measured on the full TIMIT test set after the selection step. The subset is not an independent validation set, and the multiple-comparisons optimism from trying multiple layers and pooling functions is not accounted for. For sSIMI the paper says the best layer is chosen on an independent dev set, but the full protocol (how many layers/pooling variants, which layer was ultimately used) is not reported. The authors should restrict all selection to a proper validation split (e.g., a portion of TIMIT train or a separate dev set), document the number of choices tried, and apply the same selection protocol to all baselines.
  3. [Table 3 / Section 3.2] The WavCoch vocabulary size (13-bit / 8,192 tokens) is selected by ablating 12-, 13-, and 14-bit codes and evaluating reconstruction error and phoneme cluster purity on the TIMIT test set. Because WavCoch is the tokenizer feeding AuriStream, this hyperparameter choice is effectively fitted to an evaluation corpus (TIMIT). This biases the TIMIT probing results reported in Table 1, and it may indirectly affect the sSIMI and SUPERB results through the tokenizer. The ablation should be performed on a held-out validation set that is disjoint from all test sets used in the paper, and the transfer of the chosen vocabulary to the sSIMI and SUPERB benchmarks should be documented.
minor comments (5)
  1. [Section 2.3.3] The text refers to 'Figure 7.1A' in multiple places; this should be 'Figure 1A'. The appendix similarly references 'Figure 7.1A'.
  2. [Section 3.3] The phrase 'the top 10 phonemes/words in the TIMIT test set' is ambiguous. Does this mean the 10 most frequent classes? Please define the subset precisely and report its size. This also connects to the major concern about test-set selection.
  3. [Appendix 7.6] The qualitative statements that completions are 'consistently' produced and 'often sound very plausible' are based on visual/auditory inspection. A small listening study or a quantitative metric (e.g., ASR-based intelligibility of continuations) would strengthen these claims, or the text should explicitly label them as anecdotal.
  4. [Throughout] The cochleagram inversion is described as an optimization procedure but no evaluation of inversion fidelity is provided. Reporting a simple reconstruction error (e.g., L2 or SI-SNR) would help readers calibrate the quality of the generated audio.
  5. [Section 2.1] Several references have typographical spacing issues (e.g., 'Y . Zhang', 'Y .-A.') and inconsistent figure labels (Figure I vs Fig. I). These are minor and do not affect content.

Circularity Check

2 steps flagged

No circular derivation of the central sSIMI/SUPERB claims; two minor test-set-based model selections make the TIMIT probing numbers mildly self-referential.

specific steps
  1. fitted input called prediction [Section 2.3.3 and Table 1]
    "To select the best layer for decoding, we evaluate the phoneme/word probing performance on a subset of the TIMIT set (the top 10 phonemes/words in the TIMIT test set)."

    The layer used for the reported TIMIT phoneme/word accuracies is selected using labels from the same TIMIT test set that Table 1 reports as the result. The reported best-layer accuracy is therefore not an independent out-of-sample number: the choice of layer is fitted to the evaluation set. This is a mild self-referential fit. It does not affect the sSIMI result, where the paper explicitly selects the layer on the independent dev set, nor the SUPERB results, which use frozen representations with separately trained decoders.

  2. fitted input called prediction [Section 2.1 and Appendix 7.2]
    "For each of these models, we evaluated the cochleagram reconstruction L2 error and phoneme cluster purity on an out-of-distribution test set (TIMIT test set [43])."

    The WavCoch vocabulary size is chosen on the TIMIT test set using phoneme cluster purity, a direct proxy for the phoneme-decoding task, and the same TIMIT test set is then used to report phoneme decoding in Table 1. This makes the tokenizer hyperparameter partially fitted to the TIMIT evaluation. The effect is indirect and does not force the sSIMI or SUPERB outcomes, which are external benchmarks.

full rationale

The paper's central claims—state-of-the-art sSIMI lexical semantics and competitive SUPERB performance—are not circular. AuriStream is trained with a plain autoregressive next-token objective on unlabeled LibriLight speech; WavCoch is trained independently to predict a cochleagram from waveforms. Neither training objective uses the sSIMI human judgments, the SUPERB labels, or the TIMIT labels. The sSIMI layer selection is done on the independent dev set, and SUPERB uses frozen features with standard downstream decoders. The self-citations in the paper ([54], [56]) appear only in the limitations/NeuroAI framing and are not load-bearing for any reported result. The only genuine self-referential elements are the use of a subset of the TIMIT test set to choose the probing layer and the use of TIMIT test phoneme purity to choose the WavCoch vocabulary size; both mildly inflate the TIMIT phoneme/word probing numbers but do not affect the main external-benchmark claims. Accordingly, the paper is not circular in its derivation; the appropriate score is low.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

The central claim rests on a standard empirical ML pipeline. The main entities are the discrete cochlear tokens, which are internal to the model and not independently validated. The free parameters are the tokenizer vocabulary size and probe layer, both of which are tuned using the evaluation test set. The key domain assumptions are that the cochleagram target is a useful auditory representation and that next-token prediction plus linear probing reveals semantic structure.

free parameters (2)
  • WavCoch vocabulary size = 8192 (13-bit)
    Selected by ablating 12-, 13-, and 14-bit codebooks and evaluating L2 reconstruction error and phoneme cluster purity on the TIMIT test set (Appendix 7.2).
  • Probe layer for phoneme/word decoding = Not specified, best layer per task
    Best layer chosen on a subset of the TIMIT test set for phonemes and words, and on the sSIMI dev set for semantics (Section 2.3.3).
axioms (4)
  • domain assumption The cochleagram is a faithful model of human auditory peripheral processing (Feather et al., 2023)
    Used to define the WavCoch prediction target and to motivate the biologically inspired framing (Sections 1.2, 2.1, Appendix 7.3).
  • domain assumption Autoregressive next-token prediction over discrete tokens yields useful speech representations
    Core training objective; the paper motivates this by analogy to language modeling but does not prove it (Section 1.2).
  • domain assumption Linear probing of frozen embeddings is a valid measure of representation informativeness
    Standard protocol used for all phoneme, word, and semantic quality claims (Section 2.3.1).
  • domain assumption The sSIMI benchmark captures lexical semantics
    The state-of-the-art lexical semantics claim rests entirely on this benchmark from the ZeroSpeech 2021 challenge (Section 2.3.2).
invented entities (1)
  • Cochlear tokens no independent evidence
    purpose: Discrete input units for AuriStream, derived from an LFQ bottleneck in WavCoch
    Internal discrete codes introduced by the paper. There is no independent biological or behavioral evidence that these tokens correspond to actual cochlear codes; they are a modeling construct.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Representing Speech Through Autoregressive Prediction of Cochlear Tokens." pith.science (2026). https://pith.science/paper/EWVULQQL

@misc{pith2026250811598,
  author       = {Pith},
  title        = {Pith review of: Representing Speech Through Autoregressive Prediction of Cochlear Tokens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EWVULQQL}},
  note         = {Machine review of arXiv:2508.11598}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We introduce AuriStream, a biologically inspired model for encoding speech via a two-stage framework inspired by the human auditory processing hierarchy. The first stage transforms raw audio into a time-frequency representation based on the human cochlea, from which we extract discrete \textbf{cochlear tokens}. The second stage applies an autoregressive sequence model over the cochlear tokens. AuriStream learns meaningful phoneme and word representations, and state-of-the-art lexical semantics. AuriStream shows competitive performance on diverse downstream SUPERB speech tasks. Complementing AuriStream's strong representational capabilities, it generates continuations of audio which can be visualized in a spectrogram space and decoded back into audio, providing insights into the model's predictions. In summary, we present a two-stage framework for speech representation learning to advance the development of more human-like models that efficiently handle a range of speech-based tasks.

Figures

Figures reproduced from arXiv: 2508.11598 by Daniel L.K. Yamins, Evelina Fedorenko, Greta Tuckute, Klemen Kotar.

Figure 1
Figure 1. Figure 1: Schematic of speech representation framework. De￾scription of the steps of the framework can be found in the In￾troduction (1.2) and Methods (2). erates in two stages: The first stage is WavCoch, a model that transforms the raw audio into a time-frequency representation based on the human cochlea (Figure 1A). This approach bears some resem￾blance to neural audio codecs; however, instead of reconstruct￾ing … view at source ↗
Figure 2
Figure 2. Figure 2: Cochleagram predictions by AuriStream-1B. A. AuriStream-1B is prompted with the first phoneme of the word “she” (left of red vertical line) and predicts the word comple￾tion (right of red line) across three seeds. The ground-truth (GT) cochleagram is shown in the first column. B. AuriStream is prompted with the first phoneme of the words “wash” and “water”. C. AuriStream is prompted with the first 2.5 seco… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 41 canonical work pages · 4 internal anchors

  1. [1]

    Representing Speech Through Autoregressive Prediction of Cochlear Tokens

    Introduction Humans possess a remarkable ability to perform a wide range of tasks on speech inputs, from recognizing words in noise to sep- arating speakers’ voices and interpreting emotional tone. These processes are carried out by the human ear and networks of bio- logical neurons. However, developing artificial neural networks that mirror the human abi...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    water” and “river

    Methods 2.1. Input Tokens: WavCoch We proposeWavCoch, a model that efficiently tokenizes audio by transforming waveforms into cochleagrams, loosely mim- icking the function of the human cochlea [31, 34]. The purpose of WavCoch is to extract discrete tokens from continuous au- dio signals to serve as the input to AuriStream. WavCoch is a causal encoder-dec...

    work page 2021

  3. [3]

    er” was often confused with “r

    Results 3.1. AuriStream Embeddings Contain Information about Phoneme Identity, Word Identity, and Lexical Semantics To first assess whether AuriStream representations contain in- formation about phoneme and word identity, we trained linear classifiers on the phonemes and words from the TIMIT train set [43] and evaluated the classifiers on the test set wit...

    work page 2021

  4. [4]

    Transformation Imitation

    Conclusion We introduced AuriStream, a self-supervised speech represen- tation model that achieves competitive phoneme and word de- coding, state-of-the-art lexical semantic representations, and serves as a strong representational backbone for various audio tasks. A key strength of our framework is the use of cochlear tokens: a biologically inspired and h...

    work page

  5. [5]

    acknowledges support from The K

    Acknowledgements G.T. acknowledges support from The K. Lisa Yang ICoN Cen- ter and McGovern Institute for Brain Research. E.F. acknowl- edges support from McGovern Institute for Brain Research, the Department of Brain and Cognitive Sciences, MIT’s Quest for Intelligence, and the Simons Foundation. K.K. and D.L.K.Y . acknowledge support from the Simons Fou...

    work page

  6. [6]

    Superb: Speech processing universal performance benchmark,

    S.-w. Yang, P.-H. Chi, Y .-S. Chuang, C.-I. J. Lai, K. Lakho- tia, Y . Y . Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Linet al., “Superb: Speech processing universal performance benchmark,” arXiv preprint arXiv:2105.01051, 2021

    Pith/arXiv arXiv 2021

  7. [7]

    Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec models

    H. Wu, X. Chen, Y .-C. Lin, K. Chang, J. Du, K.-H. Lu, A. H. Liu, H.-L. Chung, Y .-K. Wu, D. Yanget al., “Codec-superb@ slt 2024: A lightweight benchmark for neural audio codec models,”arXiv preprint arXiv:2409.14085, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    A review of deep learning techniques for speech processing,

    A. Mehrish, N. Majumder, R. Bhardwaj, R. Mihalcea, and S. Poria, “A review of deep learning techniques for speech processing,” 2023. [Online]. Available: https://arxiv.org/abs/ 2305.00359

    Pith/arXiv arXiv 2023

  9. [9]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,”IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, vol. 30, pp. 495–507, 2021

    2021

  10. [10]

    High fidelity neural audio compression,

    A. D’efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,”ArXiv, vol. abs/2210.13438, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID: 253097788

    Pith/arXiv arXiv 2022

  11. [11]

    Hifi- codec: Group-residual vector quantization for high fidelity audio codec,

    D. Yang, S. Liu, R. Huang, J. Tian, C. Weng, and Y . Zou, “Hifi- codec: Group-residual vector quantization for high fidelity audio codec,”arXiv preprint arXiv:2305.02765, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    High-fidelity audio compression with improved rvqgan,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,”Ad- vances in Neural Information Processing Systems, vol. 36, 2024

    work page 2024

  13. [13]

    Language-codec: Reducing the gaps between discrete codec representation and speech language models,

    S. Ji, M. Fang, Z. Jiang, R. Huang, J. Zuo, S. Wang, and Z. Zhao, “Language-codec: Reducing the gaps between discrete codec representation and speech language models,”arXiv preprint arXiv:2402.12208, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Speechtok- enizer: Unified speech tokenizer for speech language models,

    X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “Speechtok- enizer: Unified speech tokenizer for speech language models,” inThe Twelfth International Conference on Learning Representa- tions, 2024

    work page 2024

  15. [15]

    Clam-tts: Improving neural codec language model for zero-shot text-to-speech,

    J. Kim, K. Lee, S. Chung, and J. Cho, “Clam-tts: Improving neural codec language model for zero-shot text-to-speech,”arXiv preprint arXiv:2404.02781, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Spectral codecs: Spectrogram-based audio codecs for high quality speech synthesis,

    R. Langman, A. Juki ´c, K. Dhawan, N. R. Koluguri, and B. Gins- burg, “Spectral codecs: Spectrogram-based audio codecs for high quality speech synthesis,”arXiv preprint arXiv:2406.05298, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Neural codec language models are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z.-H. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei, “Neural codec language models are zero-shot text to speech synthesizers,”ArXiv, vol. abs/2301.02111, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Hierarchical organization of hu- man auditory cortex: evidence from acoustic invariance in the re- sponse to intelligible speech,

    K. Okada, F. Rong, J. Venezia, W. Matchin, I.-H. Hsieh, K. Saberi, J. T. Serences, and G. Hickok, “Hierarchical organization of hu- man auditory cortex: evidence from acoustic invariance in the re- sponse to intelligible speech,”Cerebral Cortex (New York, N.Y.: 1991), vol. 20, no. 10, pp. 2486–2495, Oct. 2010

    work page 1991

  19. [19]

    Intonational speech prosody encoding in the human auditory cortex,

    C. Tang, L. Hamilton, and E. Chang, “Intonational speech prosody encoding in the human auditory cortex,”Science, vol. 357, no. 6353, pp. 797–801, 2017

    work page 2017

  20. [20]

    Hubert: Self-supervised speech rep- resentation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhut- dinov, and A. Mohamed, “Hubert: Self-supervised speech rep- resentation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 29, pp. 3451–3460, 2021

    2021

  21. [21]

    W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training,

    Y .-A. Chung, Y . Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y . Wu, “W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training,” Sep. 2021, arXiv:2108.06209 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2108.06209

    Pith/arXiv arXiv 2021

  22. [22]

    Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  23. [23]

    Blind phoneme segmentation with temporal prediction errors

    P. Michel, O. Räsänen, R. Thiolliere, and E. Dupoux, “Blind phoneme segmentation with temporal prediction errors,”arXiv preprint arXiv:1608.00508, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  24. [24]

    An unsupervised autoregressive model for speech representation learning,

    Y .-A. Chung, W.-N. Hsu, H. Tang, and J. Glass, “An unsupervised autoregressive model for speech representation learning,”arXiv preprint arXiv:1904.03240, 2019

    Pith/arXiv arXiv 1904

  25. [25]

    Acquiring language from speech by learning to remember and predict,

    C. Shain and M. Elsner, “Acquiring language from speech by learning to remember and predict,” inProceedings of the 24th Conference on Computational Natural Language Learning, 2020, pp. 195–214

    work page 2020

  26. [26]

    Vector-quantized autoregres- sive predictive coding,

    Y .-A. Chung, H. Tang, and J. Glass, “Vector-quantized autoregres- sive predictive coding,”arXiv preprint arXiv:2005.08392, 2020

    Pith/arXiv arXiv 2005

  27. [27]

    Audio albert: A lite bert for self-supervised learning of audio representation,

    P.-H. Chi, P.-H. Chung, T.-H. Wu, C.-C. Hsieh, Y .-H. Chen, S.- W. Li, and H.-y. Lee, “Audio albert: A lite bert for self-supervised learning of audio representation,” in2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 344–350

    work page 2021

  28. [28]

    On generative spoken language modeling from raw au- dio,

    K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y . Adi, A. Polyak, B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski, A. Mohamed et al., “On generative spoken language modeling from raw au- dio,”Transactions of the Association for Computational Linguis- tics, vol. 9, pp. 1336–1354, 2021

    work page 2021

  29. [29]

    Audiolm: a language modeling approach to audio gener- ation,

    Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi et al., “Audiolm: a language modeling approach to audio gener- ation,”IEEE/ACM transactions on audio, speech, and language processing, vol. 31, pp. 2523–2533, 2023

    2023

  30. [30]

    Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Target

    G.-W. Wu, G.-T. Lin, S.-W. Li, and H.-y. Lee, “Improving textless spoken language understanding with discrete units as intermediate target,”arXiv preprint arXiv:2305.18096, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  31. [31]

    BERT: Pre- training of Deep Bidirectional Transformers for Language Under- standing,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of Deep Bidirectional Transformers for Language Under- standing,” inProceedings of NAACL-HLT 2019, Jun. 2019

    work page 2019

  32. [32]

    Representation learning with contrastive predictive coding,

    A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,”arXiv preprint arXiv:1807.03748, 2018

    Pith/arXiv arXiv 2018

  33. [33]

    Wav2vec 2.0: a framework for self-supervised learning of speech representa- tions,

    A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “Wav2vec 2.0: a framework for self-supervised learning of speech representa- tions,” inAdvances in Neural Information Processing Systems 33 (NeurIPS 2020), Dec. 2020

    work page 2020

  34. [34]

    Contrastive learning of general-purpose audio representations,

    A. Saeed, D. Grangier, and N. Zeghidour, “Contrastive learning of general-purpose audio representations,”ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3875–3879, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:224814216

    work page 2021

  35. [35]

    Unsupervised speech segmentation and variable rate repre- sentation learning using segmental contrastive predictive coding,

    S. Bhati, J. Villalba, P. ˙Zelasko, L. Moro-Velazquez, and N. De- hak, “Unsupervised speech segmentation and variable rate repre- sentation learning using segmental contrastive predictive coding,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 30, pp. 2002–2014, 2022

    work page 2002

  36. [36]

    Self-normalization and noise- robustness in early auditory representations,

    K. Wang and S. Shamma, “Self-normalization and noise- robustness in early auditory representations,”IEEE transactions on speech and audio processing, vol. 2, no. 3, pp. 421–435, 1994

    work page 1994

  37. [37]

    K. N. Ochsner and S. Kosslyn,The Oxford Handbook of Cognitive Neuroscience, Volume 1: Core Topics. Oxford University Press, 12 2013. [Online]. Available: https://doi.org/10. 1093/oxfordhb/9780199988693.001.0001

    arXiv 2013

  38. [38]

    Multiresolution spectrotem- poral analysis of complex sounds,

    T. Chi, P. Ru, and S. A. Shamma, “Multiresolution spectrotem- poral analysis of complex sounds,”The Journal of the Acoustical Society of America, vol. 118, no. 2, pp. 887–906, Aug. 2005

    work page 2005

  39. [39]

    Model metamers reveal divergent invariances between biological and ar- tificial neural networks,

    J. Feather, G. Leclerc, A. M ˛ adry, and J. H. McDermott, “Model metamers reveal divergent invariances between biological and ar- tificial neural networks,”Nature Neuroscience, vol. 26, no. 11, pp. 2017–2034, 2023

    work page 2017

  40. [40]

    Derivation of auditory filter shapes from notched-noise data,

    B. R. Glasberg and B. C. J. Moore, “Derivation of auditory filter shapes from notched-noise data,”Hearing Research, vol. 47, no. 1, pp. 103–138, Aug. 1990. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/037859559090170T

    work page arXiv 1990

  41. [41]

    Sound texture perception via statistics of the auditory periphery: evidence from sound syn- thesis,

    J. H. McDermott and E. P. Simoncelli, “Sound texture perception via statistics of the auditory periphery: evidence from sound syn- thesis,”Neuron, vol. 71, no. 5, pp. 926–940, Sep. 2011

    work page 2011

  42. [42]

    Language model beats diffusion – tokenizer is key to visual generation,

    L. Yu, J. Lezama, N. B. Gundavarapu, L. Versari, K. Sohn, D. Minnen, Y . Cheng, V . Birodkar, A. Gupta, X. Gu, A. G. Hauptmann, B. Gong, M.-H. Yang, I. Essa, D. A. Ross, and L. Jiang, “Language model beats diffusion – tokenizer is key to visual generation,”ArXiv, 2023, iCLR 2024. [Online]. Available: https://arxiv.org/abs/2310.05737

    Pith/arXiv arXiv 2023

  43. [43]

    Lib- rispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

    2015

  44. [44]

    Improving language understanding by generative pre-training,

    A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” 2018, technical report

    work page 2018

  45. [45]

    Gaussian error linear units (gelus),

    D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),”ArXiv, 2016. [Online]. Available: https://arxiv.org/abs/ 1606.08415

    Pith/arXiv arXiv 2016

  46. [46]

    Root mean square layer nor- malization,

    B. Zhang and R. Sennrich, “Root mean square layer nor- malization,”ArXiv, 2019, neurIPS 2019. [Online]. Available: https://arxiv.org/abs/1910.07467

    Pith/arXiv arXiv 2019

  47. [47]

    Libri-light: A benchmark for asr with limited or no supervision,

    J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazar’e, J. Karadayi, V . Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. rahman Mohamed, and E. Dupoux, “Libri-light: A benchmark for asr with limited or no supervision,”ICASSP 2020 - 2020 IEEE International Confer- ence on Acoustics, Speech and Signal Processing...

    work page 2020

  48. [48]

    Timit acoustic phonetic continuous speech cor- pus,

    J. S. Garofolo, “Timit acoustic phonetic continuous speech cor- pus,”Linguistic Data Consortium, 1993, 1993

    work page 1993

  49. [49]

    Speaker-independent phone recogni- tion using hidden markov models,

    K.-F. Lee and H.-W. Hon, “Speaker-independent phone recogni- tion using hidden markov models,”IEEE Transactions on Acous- tics, Speech, and Signal Processing, vol. 37, no. 11, pp. 1641– 1648, 1989

    work page 1989

  50. [50]

    Scikit-learn: Machine Learning in Python,

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, A. Müller, J. Nothman, G. Louppe, P. Pret- tenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cour- napeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine Learning in Python,”Journal of Machine Learning Re- search, vol. 12, pp. 2825–2830, 2011

    work page 2011

  51. [51]

    The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling,

    T. A. Nguyen, M. de Seyssel, P. Rozé, M. Rivière, E. Kharitonov, A. Baevski, E. Dunbar, and E. Dupoux, “The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling,”arXiv preprint arXiv:2011.11588, 2020

    Pith/arXiv arXiv 2021

  52. [52]

    Over-reliance on english hinders cognitive science,

    D. E. Blasi, J. Henrich, E. Adamou, D. Kemmerer, and A. Majid, “Over-reliance on english hinders cognitive science,”Trends in cognitive sciences, vol. 26, no. 12, pp. 1153–1170, 2022

    work page 2022

  53. [53]

    To- wards inclusive automatic speech recognition,

    S. Feng, B. M. Halpern, O. Kudina, and O. Scharenborg, “To- wards inclusive automatic speech recognition,”Computer Speech & Language, vol. 84, p. 101567, 2024

    work page 2024

  54. [54]

    Say- cam: A large, longitudinal audiovisual dataset recorded from the infant’s perspective,

    J. Sullivan, M. Mei, A. Perfors, E. Wojcik, and M. C. Frank, “Say- cam: A large, longitudinal audiovisual dataset recorded from the infant’s perspective,”Open mind, vol. 5, pp. 20–29, 2021

    work page 2021

  55. [55]

    Call for Papers – The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus,

    A. Warstadt, L. Choshen, A. Mueller, A. Williams, E. Wilcox, and C. Zhuang, “Call for Papers – The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus,”Arxiv, 2023, publisher: arXiv Version Number: 1. [Online]. Available: https://arxiv.org/abs/2301.11796

    Pith/arXiv arXiv 2023

  56. [56]

    A task-optimized neural network repli- cates human auditory behavior, predicts brain responses, and re- veals a cortical processing hierarchy,

    A. J. E. Kell, D. L. Yamins, E. N. Shook, S. V . Norman-Haignere, and J. H. McDermott, “A task-optimized neural network repli- cates human auditory behavior, predicts brain responses, and re- veals a cortical processing hierarchy,”Neuron, vol. 98, no. 3, pp. 630–644.e16, May 2018

    work page 2018

  57. [57]

    Toward a realistic model of speech processing in the brain with self-supervised learning,

    J. Millet, C. Caucheteux, P. Orhan, Y . Boubenec, A. Gramfort, E. Dunbar, C. Pallier, and J.-R. King, “Toward a realistic model of speech processing in the brain with self-supervised learning,” in Advances in Neural Information Processing Systems 35 (NeurIPS 2022), Jun. 2022

    work page 2022

  58. [58]

    Dissecting neural computations of the human auditory pathway using deep neural networks for speech,

    Y . Li, G. K. Anumanchipalli, A. Mohamed, J. Lu, J. Wu, and E. F. Chang, “Dissecting neural computations of the human auditory pathway using deep neural networks for speech,”bioRxiv, 2022, publisher: Cold Spring Harbor Laboratory

    work page 2022

  59. [59]

    Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions,

    G. Tuckute, J. Feather, D. Boebinger, and J. H. McDermott, “Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions,”Plos Biology, vol. 21, no. 12, p. e3002366, 2023

    work page 2023

  60. [60]

    Speech taskonomy: Which speech tasks are the most predictive of fmri brain activity?

    S. R. Oota, V . Agarwal, M. Marreddy, M. Gupta, and R. S. Bapi, “Speech taskonomy: Which speech tasks are the most predictive of fmri brain activity?” inINTERSPEECH 2023-24th INTER- SPEECH Conference, 2023, pp. 5167–5171

    work page 2023

  61. [61]

    Language in brains, minds, and machines,

    G. Tuckute, N. Kanwisher, and E. Fedorenko, “Language in brains, minds, and machines,”Annual Review of Neuroscience, vol. 47, no. 2024, pp. 277–301, 2024

    work page 2024

  62. [62]

    Brain-tuned speech models bet- ter reflect speech processing stages in the brain,

    O. Moussa and M. Toneva, “Brain-tuned speech models bet- ter reflect speech processing stages in the brain,”arXiv preprint arXiv:2506.03832, 2025

    Pith/arXiv arXiv 2025

  63. [63]

    An algorithm for the machine calculation of complex fourier series,

    J. W. Cooley and J. W. Tukey, “An algorithm for the machine calculation of complex fourier series,”Mathematics of Computation, vol. 19, pp. 297–301, 1965. [Online]. Available: https://api.semanticscholar.org/CorpusID:121744946

    work page 1965

  64. [64]

    Lib- rispeech: An ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: An ASR corpus based on public domain audio books,” in2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Apr. 2015, pp. 5206–5210, iSSN: 2379-190X

    work page 2015

  65. [65]

    codebook usage

    Appendix 7.1. WavCoch Architecture Details As shown in Figure 1A, the raw waveform (shape: 1×80,000 for 5s of mono audio sampled at 16kHz) is first transformed into the time–frequency domain via a fixed-kernel discrete Fourier transform implemented as a bank of 1D convolutional filters (window size 1,001 samples, hop length 80 samples). The fil- ter weigh...

    work page

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.