Pith. sign in

REVIEW 3 major objections 5 minor 48 references

HARP trains neural audio codecs to put bass in early stages and treble in later ones, improving quality with no extra inference cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 20:18 UTC pith:NYO4L3TX

load-bearing objection Training-only spectral hierarchy for RVQ is a real idea; the evidence is suggestive but the band-locality assumption and missing controls make the strongest claims premature. the 3 major comments →

arxiv 2607.16657 v1 pith:NYO4L3TX submitted 2026-07-18 cs.SD

HARP: Harmonic-Aware Residual Partitioning for Neural Audio Codecs

Qiaoyu Yang , Lixing He , Binyue Deng , Weifeng Zhao This is my paper
classification cs.SD
keywords neural audio codecresidual vector quantizationspectral entanglementfrequency hierarchyharmonic coherencesubband supervisiontraining-only methodbitrate scalability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to show that the residual vector quantization stages of a neural audio codec can be made spectrally specialized by changing only the training loss, not the network architecture. Standard RVQ codebooks capture an unpredictable mix of frequencies, so truncating stages for low bitrates removes bass or treble erratically. HARP partitions the nine RVQ stages into four frequency-ordered groups, decodes each group's contribution cumulatively over all lower groups, and supervises each contribution with a soft, learnable mel-band weight. The paper reports that this yields better objective and listening-test scores than both standard RVQ and a parallel band-split codec, with largest gains at low bitrate, and that inference stays identical to standard RVQ.

Core claim

On the paper's own terms, the central discovery is that spectral hierarchy can be imposed on RVQ through the loss only: by supervising each stage group's isolated waveform increment (Eq. 12) with a Gaussian-weighted mel loss, the learned codebooks organize themselves so that early groups carry low-frequency content and later groups add high-frequency detail, while the decoder always receives the cumulative lower-frequency latent as context. This preserves harmonic coherence that parallel band decomposition loses, because overtones are reconstructed with knowledge of their fundamentals. The paper claims consistent SI-SDR and kernel-audio-distance gains over the standard RVQ codec it builds on

What carries the argument

The load-bearing mechanism is the waveform increment defined in Eq. (12): x_k = G(sg[z_prev] + z_k) - sg[x_prev], which subtracts the stopped-gradient output of prior groups from a decoder pass that keeps the prior cumulative latent as context. This isolates, in the gradient path, the spectral contribution of group k while still giving the decoder access to lower-frequency content, so that mid and treble reconstruction is supervised in the presence of its harmonic foundation. Two auxiliary pieces support it: subband contribution supervision (the band loss applied to x_k rather than to the cumulative output) and soft band weighting, a learnable Gaussian over mel bins with a fixed floor, which

Load-bearing premise

The load-bearing assumption is that the subtraction in Eq. (12) really isolates group k's spectral contribution; because the decoder is a nonlinear convolutional network, the difference of two decoded waveforms is not guaranteed to be band-local, and the paper provides no direct diagnostic that x_k is confined to its target band.

What would settle it

Compute the actual spectrum of x_k for each group on held-out audio; if group 1's waveform increment retains substantial energy below 1 kHz beyond the floor weight, or if removing the band loss changes the learned centers and the claimed gains vanish, the isolation assumption fails. A simpler check: if the learned centers of groups 0 and 1 remain identical at 0.08 after longer training with a stronger band weight, the frequency-ordered hierarchy is not being achieved.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, any RVQ-based codec can gain frequency-prioritized bit allocation without architectural change, so low-bitrate truncation becomes predictable: bass survives, treble is dropped first.
  • The single unified token stream is preserved, unlike parallel band-split codecs, so HARP can be dropped into existing codec language-model pipelines unchanged.
  • The gap over standard RVQ grows at low bitrates (average +0.6 dB at 2.6 kbps vs +0.5 dB at 7.7 kbps), meaning the benefit is largest exactly where bitrate is scarce.
  • Harmonic coherence is measurably better than parallel decomposition: on aligned-phase synthetic tones HARP reaches phase coherence 0.988 vs the parallel codec's 0.914 with high variance, directly supporting the cross-band-context claim.
  • The method is orthogonal to the quantizer variant, so the same loss can be applied to other RVQ-style codecs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's Table 3 shows that groups 0 and 1 both converge to the same learned center (~228 Hz), so the five bass/low-mid codebooks are not splitting at the 1 kHz boundary as initialized. A likely extension is to test whether reallocating stage counts or increasing the band loss weight forces a cleaner split.
  • Because the band parameters are learned, the hierarchy adapts to data: on datasets with little low-frequency energy, groups may re-organize. An open extension is to measure how the learned centers shift across domains.
  • The stop-gradient in Eq. (12) means codebook entries are updated only through commitment/codebook losses, not through the band loss. A testable variant is to allow a small band-loss gradient into the codebook entries of the current group, which may improve band localization without breaking isolation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes HARP, a training-only modification of RVQ-based neural audio codecs in which the nine RVQ stages are partitioned into four ordered groups (3-2-2-2 allocation) associated with low-to-high frequency bands. For each group, the decoded waveform increment is computed via Eq. (12) and supervised with a learnable mel-band-weighted L1 loss (Eq. 15); cumulative decoding retains lower-stage latents as context. At inference the architecture is identical to standard RVQ. Experiments on music, speech, and general audio compare HARP against DAC and a parameter-matched BSCodec, reporting SI-SDR, KAD, PESQ, STOI, MUSHRA, synthetic harmonic-coherence tests, and ablations. The central claim is that HARP outperforms both standard RVQ and parallel band decomposition while preserving harmonic coherence and requiring no architectural change.

Significance. If the central claim holds, HARP is a useful and economical contribution: it imposes a frequency hierarchy on RVQ entirely through the training loss, leaves inference and the token stream unchanged, and offers more graceful bitrate truncation than standard RVQ. The paper also includes a synthetic harmonic-coherence experiment that directly targets the proposed mechanism, an ablation isolating the main design choices, and public code/models. The main risk is that the spectral-locality assumption underpinning Eq. (12) is not demonstrated and is partially contradicted by Table 3, and the headline objective and subjective claims are reported without variance or significance testing. With additional diagnostics and more careful statistical reporting, the contribution would be solid.

major comments (3)
  1. [§4.4, §4.5, §6.2 (Eq. 12, Eq. 15, Table 3)] The central mechanism treats x_k = G(sg[z_prev] + z_k) - sg[x_prev] as group k's isolated spectral contribution. Because G is a nonlinear convolutional decoder, the subtraction does not guarantee band-local content; energy from x_k can spread across the spectrum. Eq. (15) compares x_k to the full-band waveform x, not to a band-filtered residual, so a group can be rewarded for reconstructing lower-frequency content even if its intended band is not respected. Table 3 gives direct evidence that the intended frequency ordering only partially emerges: the learned centers for groups 0 and 1 both converge to 0.08 (~228 Hz), so the low-mid group consolidated with the bass group instead of splitting at the initialized 0-1 kHz / 1-4 kHz boundary. The paper needs a quantitative band-energy-concentration diagnostic for each x_k (e.g., fraction of energy in the target band) and either revised claims
  2. [§6.1, §6.4, §6.5 (Tables 2, 5, 7, 6)] The headline comparisons are point estimates without variance, confidence intervals, or significance tests. Table 2 reports SI-SDR and KAD on a single test set per domain; a +0.25 dB music SI-SDR gain over DAC may be within run-to-run or test-set noise. Table 5 spans bitrates but has no error bars, and Table 7 reports a PESQ gain of +0.09 while stating STOI is 'not significant' without showing any test. Table 6's MUSHRA medians/IQRs are not accompanied by an inferential statistic, yet the text claims HARP scores 'significantly above' DAC. The authors should provide standard errors, per-seed or per-item variability, and appropriate paired significance tests for the main objective and subjective claims.
  3. [§6.5 (Table 6)] The MUSHRA protocol is incompletely reported. A MUSHRA test normally includes an explicit low anchor; no anchor scores are given, and the conditions listed omit BSCodec entirely, so the subjective comparison cannot assess the parallel-decomposition baseline. The claim of perceptual improvement rests on a single trained model, 12 participants, and median/IQR summaries. The authors should report the anchor condition, participant screening details, per-item scores, and a paired test (e.g., Wilcoxon signed-rank) for the DAC-HARP difference.
minor comments (5)
  1. [§4.4-4.5] Notation is inconsistent: Eq. (12) defines \hat{x}_k, but Eq. (15) and later text use x_k; the paper should use one symbol consistently.
  2. [§6.2 (Table 3)] Table 3 compares HARP's learned centers \mu_k with DAC's empirical mel centroids from 'tier-diff contributions' on n=50 test items, with no variance for either column. These are different quantities; please clarify what is being compared and report dispersion.
  3. [§4.7, Algorithm 1] The text says band losses are computed 'regardless of n', but Algorithm 1 computes them only for active groups k=0,...,n-1. Clarify whether inactive groups receive band losses when group dropout is active.
  4. [§5.3] The BSCodec baseline is described as 'scaled down' to match parameter count, but no details are given on quantizer capacity, bitrate matching, or training hyperparameters. A short specification would strengthen the fairness argument.
  5. [§5.4] STOI is described as a 'direct indicator of speech intelligibility' but is computed here on reconstructed speech at 7.7 kbps; the reader would benefit from a sentence noting that STOI saturates at high quality, which likely explains the small difference.

Circularity Check

1 steps flagged

Fitted band centers used as evidence of specialization; central quality gains are externally validated and not circular.

specific steps
  1. fitted input called prediction [Section 6.2, Table 3; cf. Section 4.5, Eqs. (13)–(15)]
    "Table 3: Spectral centroid of each stage group, normalized to [0,1] in mel space. HARP reports learned band centers µ_k after training; DAC reports empirical mel centroids from tier-diff contributions on MUSDB18-HQ test set (n=50). ... HARP's learned centers for groups 0 and 1 both converge to 0.08 (∼228 Hz), indicating the model consolidated its five bass–low-mid codebooks in the fundamental frequency range of most instruments rather than splitting at the initialized boundary."

    The HARP values in Table 3 are the learned parameters µ_k of Eq. (13), which define the band weights w_k in Eq. (14) and directly control the band loss in Eq. (15). Eq. (15) is minimized by moving µ_k toward whatever mel region lets each group's increment reduce L_band most, so reporting the final µ_k values as 'spectral specialization' is an in-loop description of the fitted objective, not an independent test of the intended 0–1 kHz / 1–4 kHz split. The DAC column is independent, but the HARP specialization evidence is partly self-referential. This affects a supporting analysis table, not the main held-out SI-SDR/KAD/PESQ/MUSHRA comparisons.

full rationale

The core derivation chain is: define cumulative latent (Eq. 10), define the group waveform increment via stop-gradient subtraction (Eq. 12), define soft learnable band weights (Eqs. 13–14), apply the band loss (Eq. 15), combine into the full training objective (Eq. 16), and then evaluate on held-out MUSDB18-HQ, LibriTTS, FSD50K and MUSHRA listening tests. Those evaluations are external to the training objective, so the central claims—HARP improves SI-SDR, KAD, PESQ, and MUSHRA over DAC and BSCodec at 2.6–7.7 kbps while leaving inference identical to standard RVQ—are not circular. There is no load-bearing self-citation: the only co-authored reference in the bibliography (WavTokenizer) appears in related work and is not used to justify any load-bearing premise. The one in-loop evidence item is Table 3, where HARP's learned centers µ_k are fitted parameters of the same band loss used to create the specialization; reporting them as evidence that the groups specialize is partly by construction. The paper's own Table 3 also shows groups 0 and 1 both converging to 0.08 (~228 Hz), which weakens the 'frequency-ordered partition' claim, but that is a correctness/robustness concern rather than a circularity. The held-out mel-spectrogram inspection of x_k and the objective results give the central claim independent content, so the paper should be scored low. Score 2.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The central claim rests on the band-locality of Eq. (12), the perceptual adequacy of mel-magnitude supervision, and fair baselines. Six hand-chosen or learned parameters control the mechanism; no new physical entities are introduced.

free parameters (6)
  • learnable mel centers mu_k (k=0..3) = 0.08, 0.08, 0.65, 0.81 (Table 3)
    Optimized during training to steer each group's band weight; groups 0 and 1 collapsed to the same value, so the hierarchy is coarser than the intended four bands.
  • learnable bandwidths sigma_k = not reported; initialized to M/(3K) ≈ 6.7 mel bins
    Optimized during training; final values are not disclosed, so their contribution to specialization is untestable from the paper.
  • band loss weight lambda_band = 5
    Hand-chosen; no sensitivity analysis is provided.
  • Gaussian floor beta = 0.3
    Hand-chosen; ablation only tests beta=0 (rectangular), not intermediate values.
  • group dropout probability p_drop = 0.5
    Hand-chosen; because the DAC baseline is described as identical hyperparameters but without band supervision, it is unclear whether dropout was included in the baseline, creating a potential confound in low-bitrate comparisons.
  • stage allocation K=4, 3-2-2-2 = groups: stages 1-3, 4-5, 6-7, 8-9
    Hand-chosen to give bass more codebooks; no ablation of alternative allocations is reported.
axioms (6)
  • standard math The straight-through estimator provides usable gradients through the argmin quantization
    Invoked in Section 3.4 for L_vq; standard practice but not a proven guarantee for band-specialized objectives.
  • domain assumption Mel-spectrogram magnitude loss is an adequate perceptual proxy for spectral quality
    Eq. (15) supervises groups using log-mel magnitudes; phase coherence is claimed to emerge despite magnitude-only supervision.
  • ad hoc to paper The decoder is sufficiently band-local that subtracting the prior decoded output in Eq. (12) isolates group k's spectral contribution
    This is the core mechanism of HARP; no proof or diagnostic is given, and Table 3 partially contradicts it for groups 0 and 1.
  • domain assumption Cumulative access to lower-frequency latents improves higher-band reconstruction
    R2 in Section 4.1; plausible but not independently established beyond the HARP vs BSCodec comparison.
  • domain assumption The parameter-matched BSCodec baseline remains representative of parallel band decomposition
    Section 5.3 scales BSCodec down to match parameter count; this may handicap its architecture-specific design.
  • domain assumption Group dropout during training does not alter which mechanism causes low-bitrate gains
    Section 4.7 adds dropout with p=0.5; the DAC baseline is not explicitly reported to include the same dropout, so the isolation claim is unverified.

pith-pipeline@v1.3.0-alltime-deepseek · 13792 in / 15324 out tokens · 147364 ms · 2026-08-01T20:18:47.189687+00:00 · methodology

0 comments
read the original abstract

Neural audio codecs with residual vector quantization (RVQ) normally treat all frequencies uniformly, so their codebooks become spectrally entangled. Truncating stages then removes an unpredictable mix of frequencies. Parallel band decomposition addresses this by splitting audio into independent bands, but fragments the latent space and loses cross-frequency coherence. We introduce HARP (Harmonic-Aware Residual Partitioning), a training strategy that partitions RVQ stages into frequency-ordered groups where each group refines its target band while the decoder retains access to all lower frequencies. Overtones are reconstructed in the context of their fundamentals, preserving coherence that parallel methods lose. HARP requires no architectural changes; it only modifies the training loss, leaving inference identical to standard RVQ. On speech, music, and general audio, HARP outperforms both standard RVQ and parallel decomposition. MUSHRA listening tests also show perceptual improvements.

Figures

Figures reproduced from arXiv: 2607.16657 by Binyue Deng, Lixing He, Qiaoyu Yang, Weifeng Zhao.

Figure 1
Figure 1. Figure 1: Overview of the HARP pipeline. Nine RVQ stages are partitioned into four groups targeting progressively higher frequency bands. At each group boundary, the cumulative latent (zˆ≤k) is decoded and its mel-spectrogram contribution is supervised against a band-weighted reference (−1 denotes subtraction of the previous group’s output). At inference, only a single encoder–decoder forward pass is needed, identic… view at source ↗
Figure 2
Figure 2. Figure 2: MUSHRA score distributions at 4.3 kbps (Test A) and 7.7 kbps (Test B). Boxes show median and IQR; whiskers extend to 1.5×IQR; outliers shown as points. 6.3. Harmonic Coherence Two stimulus conditions are used ( [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 5 linked inside Pith

  1. [1]

    Beyond compression, they pro- duce discrete tokens that bridge continuous audio and language models, enabling speech synthesis [4], music generation [5, 6], and audio understanding

    Introduction Neural audio codecs [1, 2, 3] compress speech, music, or en- vironmental sound directly from data, matching or exceeding classical codecs at low bitrates. Beyond compression, they pro- duce discrete tokens that bridge continuous audio and language models, enabling speech synthesis [4], music generation [5, 6], and audio understanding. Most mo...

    Pith/arXiv arXiv 2026

  2. [2]

    Neural Audio Codecs End-to-end neural audio compression pairs a learned encoder– decoder with a discrete bottleneck, trained jointly with recon- struction and adversarial losses

    Related Work 2.1. Neural Audio Codecs End-to-end neural audio compression pairs a learned encoder– decoder with a discrete bottleneck, trained jointly with recon- struction and adversarial losses. SoundStream [1] established the dominant recipe, a convolutional encoder–decoder with residual vector quantization (RVQ) and multi-scale discrimina- tors. Encod...

  3. [3]

    Neural Audio Codec Architecture A neural audio codec comprises an encoderE, a quantizerQ, and a decoderG

    Background 3.1. Neural Audio Codec Architecture A neural audio codec comprises an encoderE, a quantizerQ, and a decoderG. Given a waveformx∈R T , the encoder pro- duces a continuous latent representationz=E(x)∈R D×T ′ , whereT ′ =T /Sis determined by the cumulative strideSof the convolutional backbone (typicallyS∈ {320,512}at 24– 44.1 kHz). The decoder re...

  4. [4]

    Architecture and inference are identical to standard RVQ, incurring no additional cost at deployment

    Method HARP modifies only the training objective. Architecture and inference are identical to standard RVQ, incurring no additional cost at deployment. 4.1. Design Motivation We seek to partition RVQ stages by frequency while preserv- ing harmonic coherence across bands. A natural first attempt— applying bandpass filters directly to the supervision target...

  5. [5]

    Architecture We build on DAC [3]

    Experiments 5.1. Architecture We build on DAC [3]. The encoder uses a 1D convolution (ker- nel 7, 64 channels) followed by four downsampling blocks with stride factors [2, 4, 8, 8] (512×total). Each block doubles chan- nels (64→128→256→512→1024) and contains three dilated residual units (dilations 1, 3, 9), all with Snake activa- tions [8]. The decoder mi...

  6. [6]

    Reconstruction Quality Table 2 shows full-bitrate reconstruction

    Results 6.1. Reconstruction Quality Table 2 shows full-bitrate reconstruction. HARP achieves the best SI-SDR across all domains. The SI-SDR gain over DAC is +0.25 dB on music, +0.96 dB on speech, and +0.33 dB on gen- eral audio—modest in absolute terms, but note that DAC and HARP share the exact same architecture and parameter count. Table 2:Reconstructio...

  7. [7]

    Conclusion HARP demonstrates that spectral structure can be imposed on RVQ through the training loss alone, with no architectural changes. Partitioning stages into frequency-ordered groups and supervising each group’s contribution with soft, learnable band weights produces codebooks that specialize by frequency, while cumulative decoding ensures that high...

  8. [8]

    All technical content, experimental design, results, and conclusions are the work of the authors

    Generative AI Use Disclosure Large language models were used during the preparation of this manuscript for editing and polishing purposes: revising prose for clarity and academic register, checking mathematical nota- tion for consistency, and proofreading L ATEX formatting. All technical content, experimental design, results, and conclusions are the work ...

  9. [9]

    SoundStream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “SoundStream: An end-to-end neural audio codec,”IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, vol. 30, pp. 495–507, 2022

    2022

  10. [10]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,”Transactions on Machine Learning Research, 2023

    2023

  11. [11]

    High-fidelity audio compression with improved RVQGAN,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved RVQGAN,” in Proc. NeurIPS, 2023

    2023

  12. [12]

    Neural codec language models are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei, “Neural codec language models are zero-shot text to speech synthesizers,”arXiv preprint arXiv:2301.02111, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Simple and controllable music gen- eration,

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. D ´efossez, “Simple and controllable music gen- eration,” inProc. NeurIPS, 2023

    2023

  14. [14]

    AudioLM: A language modeling approach to audio generation,

    Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour, “AudioLM: A language modeling approach to audio generation,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 2523–2533, 2023

    2023

  15. [15]

    BSCodec: A band-split neural codec for high-quality universal audio reconstruction,

    H. Wang, J. Shi, J. Tian, B. Li, K. Yu, and S. Watanabe, “BSCodec: A band-split neural codec for high-quality universal audio reconstruction,” 2025, arXiv preprint arXiv:2511.06150

    arXiv 2025

  16. [16]

    Neural networks fail to learn periodic functions and how to fix it,

    L. Ziyin, T. Hartwig, and M. Ueda, “Neural networks fail to learn periodic functions and how to fix it,” inProc. NeurIPS, 2020

    2020

  17. [17]

    HiFi- Codec: Group-residual vector quantization for high fidelity audio codec,

    D. Yang, S. Liu, R. Huang, J. Tian, C. Weng, and Y . Zou, “HiFi- Codec: Group-residual vector quantization for high fidelity audio codec,” inProc. Interspeech, 2023

    2023

  18. [18]

    WavTokenizer: An efficient acoustic discrete codec tokenizer for audio language modeling,

    S. Ji, Z. Jiang, W. Wang, Y . Chen, M. Fang, J. Zuo, Q. Yang, X. Cheng, Z. Wang, R. Liet al., “WavTokenizer: An efficient acoustic discrete codec tokenizer for audio language modeling,” inProc. ICLR, 2025

    2025

  19. [19]

    SNAC: Multi- scale neural audio codec,

    H. Siuzdak, F. Gr ¨otschla, and L. A. Lanzend¨orfer, “SNAC: Multi- scale neural audio codec,” inNeurIPS Workshop on AI-Driven Speech, Music, and Sound Generation, 2024

    2024

  20. [20]

    FlexiCodec: A dynamic neural audio codec for low frame rates,

    J. Li, Y . Qian, Y . Hu, L. Zhang, X. Wang, H. Lu, M. Thakker, J. Li, S. Zhao, and Z. Wu, “FlexiCodec: A dynamic neural audio codec for low frame rates,”arXiv preprint arXiv:2510.00981, 2025

    arXiv 2025

  21. [21]

    SpeechTok- enizer: Unified speech tokenizer for speech large language mod- els,

    X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “SpeechTok- enizer: Unified speech tokenizer for speech large language mod- els,”arXiv preprint arXiv:2308.16692, 2023

    Pith/arXiv arXiv 2023

  22. [22]

    FlowDec: A flow-based full-band general audio codec with high perceptual quality,

    S. Welker, M. Le, R. T. Chen, W.-N. Hsu, T. Gerkmann, A. Richard, and Y .-C. Wu, “FlowDec: A flow-based full-band general audio codec with high perceptual quality,” inProc. ICLR, 2025

    2025

  23. [23]

    Scaling transformers for low-bitrate high-quality speech coding,

    J. D. Parker, A. Smirnov, J. Pons, C. Carr, Z. Zukowski, Z. Evans, and X. Liu, “Scaling transformers for low-bitrate high-quality speech coding,” inProc. ICLR, 2025

    2025

  24. [24]

    Neural dis- crete representation learning,

    A. van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural dis- crete representation learning,” inProc. NeurIPS, vol. 30, 2017

    2017

  25. [25]

    Product quantization for nearest neighbor search,

    H. J ´egou, M. Douze, and C. Schmid, “Product quantization for nearest neighbor search,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 1, pp. 117–128, 2010

    2010

  26. [26]

    SRCodec: Split-residual vector quantization for neural speech codec,

    Y . Zheng, W. Tu, L. Xiao, and X. Xu, “SRCodec: Split-residual vector quantization for neural speech codec,” inProc. ICASSP, 2024, pp. 451–455

    2024

  27. [27]

    Finite scalar quantization: VQ-V AE made simple,

    F. Mentzer, D. Minnen, E. Agustsson, and M. Tschannen, “Finite scalar quantization: VQ-V AE made simple,” inProc. ICLR, 2024

    2024

  28. [28]

    Image and video to- kenization with binary spherical quantization,

    Y . Zhao, Y . Xiong, and P. Kraehenbuehl, “Image and video to- kenization with binary spherical quantization,” inProc. ICLR, 2025

    2025

  29. [29]

    Addressing representa- tion collapse in vector quantized models with one linear layer,

    Y . Zhu, B. Li, Y . Xin, Z. Xia, and L. Xu, “Addressing representa- tion collapse in vector quantized models with one linear layer,” in Proc. ICCV, 2025, pp. 22 968–22 977

    2025

  30. [30]

    SPCODEC: Split and prediction for neural speech codec,

    L. Wen, L. Wang, Y . Zheng, W. Shi, and K. P. Choi, “SPCODEC: Split and prediction for neural speech codec,” inProc. Inter- speech, 2025

    2025

  31. [31]

    Latent-domain predictive neural speech coding,

    X. Jiang, X. Peng, H. Xue, Y . Zhang, and Y . Lu, “Latent-domain predictive neural speech coding,”IEEE/ACM Transactions on Au- dio, Speech, and Language Processing, vol. 31, pp. 2111–2123, 2023

    2023

  32. [32]

    APCodec: A neural audio codec with parallel amplitude and phase spec- trum encoding and decoding,

    Y . Ai, X.-H. Jiang, Y .-X. Lu, H.-P. Du, and Z.-H. Ling, “APCodec: A neural audio codec with parallel amplitude and phase spec- trum encoding and decoding,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 3256–3269, 2024

    2024

  33. [33]

    Soft disentangle- ment in frequency bands for neural audio codecs,

    B. Gini `es, X. Bie, O. Fercoq, and G. Richard, “Soft disentangle- ment in frequency bands for neural audio codecs,” inProc. EU- SIPCO, 2025

    2025

  34. [34]

    Perceptual coding of digital audio,

    T. Painter and A. Spanias, “Perceptual coding of digital audio,” Proceedings of the IEEE, vol. 88, no. 4, pp. 451–515, 2000

    2000

  35. [35]

    Definition of the Opus audio codec,

    J.-M. Valin, K. V os, and T. Terriberry, “Definition of the Opus audio codec,”RFC 6716, 2012

    2012

  36. [36]

    Multi-band frequency reconstruction for neural psychoacoustic coding,

    D. Ng, K. Zhou, Y .-W. Chao, Z. Xiong, B. Ma, and E. Chng, “Multi-band frequency reconstruction for neural psychoacoustic coding,” inProc. ICML, 2025

    2025

  37. [37]

    Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,

    R. Yamamoto, E. Song, and J.-M. Kim, “Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,” inProc. ICASSP, 2020, pp. 6199–6203

    2020

  38. [38]

    HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,” inProc. NeurIPS, 2020

    2020

  39. [39]

    Estimating or propa- gating gradients through stochastic neurons for conditional com- putation,

    Y . Bengio, N. L ´eonard, and A. Courville, “Estimating or propa- gating gradients through stochastic neurons for conditional com- putation,” 2013, arXiv preprint arXiv:1308.3432

    Pith/arXiv arXiv 2013

  40. [40]

    MUSDB18-HQ: An uncompressed version of MUSDB18,

    Z. Rafii, A. Liutkus, F.-R. St ¨oter, S. I. Mimilakis, and R. Bit- tner, “MUSDB18-HQ: An uncompressed version of MUSDB18,” 2019

    2019

  41. [41]

    The MTG-Jamendo dataset for automatic music tagging,

    D. Bogdanov, M. Won, P. Tovstogan, A. Porter, and X. Serra, “The MTG-Jamendo dataset for automatic music tagging,” in Proc. ICML Workshop on Machine Learning for Music Discov- ery, 2019

    2019

  42. [42]

    AudioSet: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “AudioSet: An ontology and human-labeled dataset for audio events,” inProc. ICASSP, 2017, pp. 776–780

    2017

  43. [43]

    LibriTTS: A corpus derived from LibriSpeech for text-to-speech,

    H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “LibriTTS: A corpus derived from LibriSpeech for text-to-speech,” inProc. Interspeech, 2019, pp. 1526–1530

    2019

  44. [44]

    FSD50K: An open dataset of human-labeled sound events,

    E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “FSD50K: An open dataset of human-labeled sound events,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 829–852, 2022

    2022

  45. [45]

    KAD: No more FAD! an effective and efficient evaluation metric for au- dio generation,

    Y . Chung, P. Eu, J. Lee, K. Choi, J. Nam, and B. S. Chon, “KAD: No more FAD! an effective and efficient evaluation metric for au- dio generation,” 2025, arXiv preprint arXiv:2502.15602

    Pith/arXiv arXiv 2025

  46. [46]

    Per- ceptual evaluation of speech quality (PESQ)—a new method for speech quality assessment of telephone networks and codecs,

    A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Per- ceptual evaluation of speech quality (PESQ)—a new method for speech quality assessment of telephone networks and codecs,” in Proc. ICASSP, 2001, pp. 749–752

    2001

  47. [47]

    A short- time objective intelligibility measure for time-frequency weighted noisy speech,

    C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “A short- time objective intelligibility measure for time-frequency weighted noisy speech,” inProc. ICASSP, 2010, pp. 4214–4217

    2010

  48. [48]

    Method for the subjective assessment of intermedi- ate quality level of audio systems (MUSHRA),

    ITU-R, “Method for the subjective assessment of intermedi- ate quality level of audio systems (MUSHRA),” International Telecommunication Union, Recommendation BS.1534-3, 2015

    2015