Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

CoDiCodec proposes a single audio autoencoder that yields both compressed continuous embeddings (~11 Hz) and discrete tokens (2.38 kbps) from one end-to-end consistency-trained model, and reports better reconstruction quality than existing

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

CoDiCodec unifies continuous and discrete audio compression in one consistency-trained autoencoder, using FSQ-dropout to serve both continuous ~11 Hz embeddings and 2.38 kbps discrete tokens.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Credible and useful unified codec, but the headline 'reconstruction quality' claim is built on a distribution-level metric that the paper's own per-sample numbers contradict. the 3 major comments →

arxiv 2509.09836 v1 pith:HOXUIMAY submitted 2025-09-11 cs.SD cs.AIcs.LGeess.AS

CoDiCodec: Unifying Continuous and Discrete Compressed Representations of Audio

Marco Pasini , Stefan Lattner , George Fazekas This is my paper
classification cs.SD cs.AIcs.LGeess.AS
keywords audio compressionconsistency modelsfinite scalar quantizationdiscrete tokenscontinuous latent representationssummary embeddingsparallel decodingautoregressive decoding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether one compressed audio representation can serve both continuous generative models, like diffusion, and discrete ones, like autoregressive transformers, without sacrificing quality. It introduces CoDiCodec, an autoencoder trained end-to-end with a single consistency loss, which emits summary embeddings that can be read either as ~11 Hz continuous vectors or, after finite scalar quantization, as 2.38 kbps discrete tokens. The same decoder accepts both forms, enabled by FSQ-dropout, which randomly bypasses quantization during training. On reconstruction-quality metrics, CoDiCodec reports lower FAD and FAD_clap than existing continuous and discrete codecs at comparable bitrates, with a parallel decoding strategy that also speeds up inference. A sympathetic reader would care because it removes the need to choose a latent type before building a generative model.

Core claim

The central discovery is that continuous and discrete compressed audio can come from one model rather than two. CoDiCodec encodes a spectrogram chunk into 128 four-dimensional summary embeddings; the same latents can be decoded as continuous values or as FSQ tokens with an implicit codebook of 14,641 entries, at 2.38 kbps for stereo 44.1 kHz audio. The paper shows that during training, randomly bypassing the FSQ rounding step with probability p=0.75 keeps the continuous distribution spread across [-1,1] instead of collapsing near quantization levels, while preserving discrete-token quality. With this single model, autoregressive decoding and a newly proposed parallel decoding with shifted ch

What carries the argument

The load-bearing object is the consistency-trained decoder conditioned on summary embeddings, plus FSQ-dropout. Summary embeddings are learned tokens that pool global context from a chunk, replacing a temporal sequence so that 128 latent vectors of dimension 4 can be reshaped freely across time and channels. Finite Scalar Quantization (FSQ) rounds each dimension to one of 11 levels, creating a discrete codebook without auxiliary losses. FSQ-dropout randomly feeds the unrounded tanh(z) to the upsampler during training, so one decoder learns to accept both continuous and discrete latents. The consistency loss trains the decoder to map noisy spectrograms of adjacent chunk pairs back to clean on

Load-bearing premise

The headline comparison to other codecs assumes that FAD and FAD_clap scores on the same test set are a fair common yardstick for reconstruction quality across codecs with different architectures, bitrates, and mono/stereo outputs, although the paper cites evidence that FAD's correlation with human judgment depends on the embedding backbone and reports no listening test or confidence intervals.

What would settle it

Run a blind listening test on held-out music where human listeners rate reconstruction quality of CoDiCodec against the top-scoring baseline; if CoDiCodec's FAD and FAD_clap advantage is not reflected in listener preference, the central outperformance claim is not established.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A single codec can feed both diffusion-style and autoregressive generative models from the same latent space, removing the need to retrain a separate autoencoder for each paradigm.
  • The same model supports two decoding modes: sequential autoregressive decoding for low latency, and parallel decoding with shifted pairs that the paper reports as higher quality and faster for long sequences.
  • FSQ-dropout improves continuous reconstruction without hurting discrete-token quality, implying the continuous path is not an afterthought of the discrete codebook.
  • Because summary embeddings are not temporally ordered, the same total latent dimensionality can be re-shaped across time and channels, letting downstream models choose their preferred sequence length.
  • The paper reports lower FAD and FAD_clap than prior discrete codecs at higher bitrates while using only 2.38 kbps, suggesting the compact 4-dimensional latents carry the needed information.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the FAD ranking holds up under listening tests, the uniformity induced by FSQ-dropout may be useful beyond codecs: any quantized latent that later feeds a continuous generative model could benefit from dropout-style regularization instead of a separate dequantization stage.
  • The parallel decoding with shifted pairs is a general recipe for consistency models on long sequences; it could transfer to other consistency-based autoencoders or to image and video generation where boundary artifacts appear when decoding in blocks.
  • A testable extension: train the same architecture with p closer to 1 or with a curriculum that anneals p, and measure whether the continuous-discrete quality trade-off shifts; the paper fixes p=0.75, so the optimum across bitrates is unexplored.
  • Because the codebook is implicit and small (14,641 tokens), the discrete side may be easy to scale by increasing N or dlat while keeping FSQ's straight-through training; whether 2.38 kbps is a sweet spot or a floor is not tested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CoDiCodec, a single audio autoencoder that produces both compressed continuous embeddings (~11 Hz) and discrete FSQ tokens (2.38 kbps) from the same model. The method builds on Music2Latent/Music2Latent2 and uses summary embeddings, a consistency-training loss, and a newly introduced FSQ-dropout technique that bypasses quantization with probability p during training so that one decoder can operate on both discrete and continuous latents. The architecture also supports autoregressive and a novel parallel decoding strategy. The authors report reconstruction/audio-quality metrics (FAD, FAD_clap, SI-SDR, ViSQOL) on MusicCaps, claiming that CoDiCodec outperforms existing continuous and discrete autoencoders at similar bitrates in reconstruction audio quality, and they provide ablation studies and inference speed comparisons.

Significance. If the claims hold, the contribution is practically valuable: a single codec feeding both continuous diffusion-style generators and discrete autoregressive generators would remove a major architectural choice in audio generation. The FSQ-dropout idea is simple and plausible, and the paper includes a thorough ablation, a code/weights release link, and comparisons to several existing systems. The manuscript also makes a credible methodological step by showing that a consistency model can be trained end-to-end with one loss while supporting both latent types. However, the headline comparative claim depends on distribution-level metrics (FAD/FAD_clap) for what is presented as reconstruction quality, and the baseline selection omits standard codecs such as EnCodec and SoundStream; the evidence as presented is therefore not yet fully convincing even though the core model is defensible.

major comments (3)
  1. [§5.3 / Table 2] The central claim that CoDiCodec 'outperforms existing continuous and discrete autoencoders ... in terms of reconstruction audio quality' rests on FAD and FAD_clap, which are distribution-level metrics computed on feature sets without per-sample correspondence. Low FAD can be achieved by a decoder that produces plausible music statistically similar to the evaluation set without reconstructing the specific input. This is not hypothetical: Table 2 shows CoDiCodec has SI-SDR of -0.01 to -0.95 dB and ViSQOL of 3.88–3.95, while DAC 8k achieves +9.48 dB and 4.21 on the same per-sample metrics. The paper itself concedes in §5.3 that 'general audio quality is thus prioritised over reconstruction of the exact same signal.' The headline claim should either be rephrased to 'distribution-level audio quality' or be supported by per-sample reconstruction metrics at matched bitrates (or a subjective li
  2. [§5.3 / Table 2] The baseline set omits two standard discrete neural codecs discussed in the introduction: SoundStream and EnCodec. Since the paper claims to outperform 'existing discrete autoencoders at similar bitrates', these are the most direct competitors. Including EnCodec (e.g., 1.5/3 kbps) and SoundStream (e.g., 3 kbps) would make the comparison credible. The current table compares only DAC (2.67/8 kbps) among RVQ codecs, and even DAC 8k is not clearly beaten on SI-SDR/ViSQOL.
  3. [§5.1 / Table 1] Ablation results are reported without confidence intervals, multiple seeds, or statistical significance tests. Several differences between configurations are very small—e.g., continuous FAD_clap for '+128 lat.' is 0.0154 vs. 0.0161 for 'd.o. p=0.75', and discrete FAD_clap differs by 0.0005–0.0009 across FSQ-dropout probabilities. Without variance estimates it is impossible to know whether the design choices (particularly FSQ-dropout p and the 128-latent reshaping) are actually responsible for the reported improvements. Since these ablations motivate the final architecture, this is a load-bearing issue for the internal validity of the paper.
minor comments (4)
  1. [Abstract / Title] Typographical errors: 'viasummary' and 'representa tions' in the abstract; 'baslines' in §5.3. These should be corrected.
  2. [§4.1 / Eq. (4)] The abbreviation 'd.o.' in Table 1 is not defined; expand to 'FSQ-dropout' there or define it near Eq. (4).
  3. [§5.3 / Table 2] The stereo/mono column is not fully explained. Some baselines are marked mono and CoDiCodec is stereo; it is not clear whether FAD/FAD_clap are computed on downmixed mono for all systems and whether stereo input affects the comparison. This should be stated in the metric description.
  4. [§4.1] The 'Random Mixing' augmentation adds two random training samples. It would be helpful to clarify whether this is applied to the raw waveform or STFT and whether the summary embeddings/decoder targets are also mixed consistently; otherwise the augmentation may be ill-posed.

Circularity Check

0 steps flagged

No circular derivation: CoDiCodec's continuous/discrete outputs and bitrate are defined from architecture/FSQ choices, not from the evaluation targets; self-citations to Music2Latent/Music2Latent2 are incremental and the headline comparison is anchored by external baselines.

full rationale

CoDiCodec's central technical claim—one autoencoder trained with a single consistency loss yields both ~11 Hz continuous embeddings and 2.38 kbps discrete tokens—is a construction, not a derivation from the evaluation metrics. The bitrate follows from the chosen K=128, dlat=4, N=5 (Sec. 4.3) and is not fitted to any target. FSQ-dropout is a defined operation (Eq. 4); its benefit is measured on MusicCaps, but the p=0.75 choice is a hyperparameter selection, not a fitted input renamed as a prediction. The evaluation claims rest on external baselines (DAC, Moûsai, Musika, Stable Audio) and on defined metrics; the authors' own Music2Latent/Music2Latent2 are cited as the architectural starting point (Sec. 2.1, 4, 5.1) and the consistency-training framework is reused from [17]. These self-citations are legitimate incremental-extension references, not load-bearing circularity: they do not forbid alternatives and are not used to define the new model's bitrate or reconstruction. The paper cites an external similar technique [36] for FSQ+dithering. One in-scope concern is not circular but a measurement-validity issue: Table 2's headline uses FAD/FAD_clap, which are distribution-level, while Sec. 5.3 concedes 'general audio quality is thus prioritised over reconstruction of the exact same signal' and CoDiCodec's SI-SDR/ViSQOL are below DAC 8k. This undermines the phrase 'reconstruction audio quality' in the abstract, but it is a mismatch between metric and claim, not a reduction of the result to its inputs. No step in the paper's derivation chain is definitionally equal to its conclusion.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

No new physical entities are introduced; FSQ-dropout and parallel decoding are training/decoding techniques, not postulated entities. The free parameters are design choices and hyperparameters that set the bitrate and quality; several are inherited from prior work by the same authors.

free parameters (6)
  • FSQ quantization levels N = N=5 (11 levels per dimension)
    Chosen by hand to set the 2.38 kbps bitrate (11^4=14641 codes for 128 tokens per chunk); not ablated.
  • FSQ-dropout probability p = 0.75
    Selected via ablation over p in {0.25, 0.5, 0.75} (Table 1) as the best trade-off between continuous and discrete FAD.
  • Number of summary embeddings K and latent dim dlat = K=128, dlat=4
    Chosen via ablation: 128x4 outperforms 8x64 at the same total dimensionality (Table 1).
  • Random mixing probability = 0.5
    Chosen by hand; ablation shows it improves FAD.
  • Consistency training schedule = Delta t0=0.1, final exponent eK=2
    Adopted from prior work [17] and [35]; not tuned here.
  • Training data mixture weights = [4,1.5,4,1] for MTG-Jamendo, DNS speech, DNS general, M4singer
    Chosen by hand to balance domains; affects robustness but not the core unification claim.
axioms (5)
  • domain assumption Consistency training converges to the true consistency function for the data distribution, enabling single-step decoding.
    Invoked in Section 3.2 and Section 4.1 to justify the training loss; convergence guarantees hold only for idealized settings and are used empirically here.
  • domain assumption The upsampler cross-connections provide sufficient conditioning for the consistency decoder to reconstruct the original chunk in one denoising step.
    Introduced in Section 4.1 forward pass; this design is inherited from Music2Latent [17] and is not proven.
  • standard math FSQ straight-through estimation gives usable gradients for the rounding operation.
    Background Section 3.3, standard technique from [28,29].
  • domain assumption FAD and FAD_clap on MusicCaps are valid cross-model measures of reconstruction quality.
    Underpins the headline comparison in Section 5.3; the paper itself cites [52] showing the FAD-human correlation is embedding-dependent.
  • domain assumption Chunked causal masking (right attends to left) yields consistent behavior for autoregressive decoding.
    Architecture description in Section 4.1; correctness is assumed from prior autoregressive latent decoding work.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of CoDiCodec: Unifying Continuous and Discrete Compressed Representations of Audio." pith.science (2026). https://pith.science/paper/HOXUIMAY

@misc{pith2026250909836,
  author       = {Pith},
  title        = {Pith review of: CoDiCodec: Unifying Continuous and Discrete Compressed Representations of Audio},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HOXUIMAY}},
  note         = {Machine review of arXiv:2509.09836}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Efficiently representing audio signals in a compressed latent space is critical for latent generative modelling. However, existing autoencoders often force a choice between continuous embeddings and discrete tokens. Furthermore, achieving high compression ratios while maintaining audio fidelity remains a challenge. We introduce CoDiCodec, a novel audio autoencoder that overcomes these limitations by both efficiently encoding global features via summary embeddings, and by producing both compressed continuous embeddings at ~ 11 Hz and discrete tokens at a rate of 2.38 kbps from the same trained model, offering unprecedented flexibility for different downstream generative tasks. This is achieved through Finite Scalar Quantization (FSQ) and a novel FSQ-dropout technique, and does not require additional loss terms beyond the single consistency loss used for end-to-end training. CoDiCodec supports both autoregressive decoding and a novel parallel decoding strategy, with the latter achieving superior audio quality and faster decoding. CoDiCodec outperforms existing continuous and discrete autoencoders at similar bitrates in terms of reconstruction audio quality. Our work enables a unified approach to audio compression, bridging the gap between continuous and discrete generative modelling paradigms.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. On the Geometry of Music Bandwidth Extension in Latent Spaces of Audio Codecs

    cs.SD 2026-08 conditional novelty 6.0

    A single mean shift in the latent space of several neural codecs achieves competitive music bandwidth extension on some metrics, implying a largely linear structure.

  2. Unsupervised Evaluation of Deep Audio Embeddings for Music Structure Analysis

    cs.SD 2026-03 conditional novelty 5.0

    Off-the-shelf deep audio embeddings generally improve unsupervised music boundary detection over spectrogram features, with CBM the best segmenter, but not all models help and standard scores are inflated by edge boundaries.

Reference graph

Works this paper leans on

61 extracted references · 17 linked inside Pith · cited by 2 Pith papers

  1. [1]

    CoDiCodec: Unifying Continuous and Discrete Compressed Representations of Audio

    INTRODUCTION Efficient, compact audio representations are crucial for ap- plications in Music Information Retrieval (MIR), generative modelling, and compression. While recent advances in deep learning have demonstrated impressive results in the learn- ing of compressed representations, several key challenges remain. These include balancing high compressio...

    Pith/arXiv arXiv 2025

  2. [2]

    These can be broadly divided into those producing discrete and con- tinuous compressed latent representations

    RELA TED WORK 2.1 Audio Autoencoders Audio autoencoders aim to learn compressed latent rep- resentations of audio signals, typically for dimensionality reduction, generative modeling, or MIR tasks. These can be broadly divided into those producing discrete and con- tinuous compressed latent representations. Discrete Latent Representations:Vector Quantizat...

  3. [3]

    BACKGROUND 3.1 Consistency Models Consistency models [19,20] are a class of generative models that learn to map any point on a diffusion process trajectory back to the origin of that trajectory. They are based on the probability flow (PF) ordinary differential equation (ODE) [27], which describes the evolution of a data sample x perturbed by Gaussian nois...

  4. [4]

    de-compress

    CODICODEC Following previous work [17, 18, 30, 31], CoDiCodec op- erates on complex Short-Time Fourier Transform (STFT) spectrograms. To address the skewed distribution of dif- ferent frequency bins, we apply an amplitude transforma- tion [32]: ˜c=β|c| αei∠(c), where c and ˜care the original and transformed STFT coefficients, α∈(0,1] is a compres- sion ex...

    2048

  5. [5]

    + 128 lat

    EXPERIMENTS AND RESULTS Data: We train CoDiCodec on a combination of three datasets: MTG-Jamendo [43] for music ( 3k hours), the speech ( 800 hours) and general audio ( 200 hours) sam- ples from DNS Challenge 4 [44], and M4singer [45] for singing voice ( 30 hours). We sample the training datasets with weights [4,1.5,4,1] , respectively, during training. W...

  6. [6]

    CONCLUSION This paper introduced a novel audio autoencoder producing both continuous embeddings and discrete tokens from a single model, trained end-to-end with a single consistency loss. This is achieved via finite scalar quantization and our proposed FSQ-dropout technique, which allows for expres- sive continuous latents that perform well for downstream...

  7. [7]

    ACKNOWLEDGEMENTS This work is supported by the EPSRC UKRI Centre for Doctoral Training in Artificial Intelligence and Music (EP/S022694/1) and Sony Computer Science Laborato- ries Paris

  8. [8]

    SoundStream: An End- to-End Neural Audio Codec,

    N. Zeghidour, A. Luebset al., “SoundStream: An End- to-End Neural Audio Codec,”IEEE ACM Trans. Audio Speech Lang. Process., vol. 30, 2022

    2022

  9. [9]

    High fidelity neural audio compression,

    A. Défossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,”Transactions on Ma- chine Learning Research, 2023

    2023

  10. [10]

    High-fidelity audio compression with im- proved RVQGAN,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with im- proved RVQGAN,” inThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  11. [11]

    Simple and control- lable music generation,

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Syn- naeve, Y . Adi, and A. Défossez, “Simple and control- lable music generation,” inThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  12. [12]

    Jukebox: A generative model for music,

    P. Dhariwal, H. Junet al., “Jukebox: A generative model for music,”arXiv preprint arXiv:2005.00341, 2020

    Pith/arXiv arXiv 2005

  13. [13]

    MusicLM: Generating Music From Text,

    A. Agostinelli, T. I. Denket al., “MusicLM: Generating Music From Text,” Jan. 2023, arXiv:2301.11325 [cs, eess]

    Pith/arXiv arXiv 2023

  14. [14]

    Generative adversarial nets,

    I. J. Goodfellow, J. Pouget-Abadieet al., “Generative adversarial nets,” inAdvances in Neural Information Processing Systems 27, Dec. 2014

    2014

  15. [15]

    Deep unsuper- vised learning using nonequilibrium thermodynamics,

    J. Sohl-Dickstein, E. A. Weisset al., “Deep unsuper- vised learning using nonequilibrium thermodynamics,” inProceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, ser. JMLR Workshop and Conference Proceed- ings, vol. 37, 2015

    2015

  16. [16]

    Score-based generative mod- eling through stochastic differential equations,

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative mod- eling through stochastic differential equations,” inIn- ternational Conference on Learning Representations, 2021

    2021

  17. [17]

    Denoising Diffusion Probabilistic Models,

    J. Ho, A. Jainet al., “Denoising Diffusion Probabilistic Models,” inAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020

    2020

  18. [18]

    Mo\^usai: Text-to-Music Generation with Long-Context Latent Diffusion,

    F. Schneider, Z. Jinet al., “Mo\^usai: Text-to-Music Generation with Long-Context Latent Diffusion,” Jan. 2023, arXiv:2301.11757 [cs, eess]

    Pith/arXiv arXiv 2023

  19. [19]

    Musika! Fast Infinite Wave- form Music Generation,

    M. Pasini and J. Schlüter, “Musika! Fast Infinite Wave- form Music Generation,” inProceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Bengaluru, India, December 4-8, 2022, 2022

    2022

  20. [20]

    Bass accompaniment generation via latent diffusion,

    M. Pasini, M. Grachtenet al., “Bass accompaniment generation via latent diffusion,” inICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024

    2024

  21. [21]

    Fast timing-conditioned latent audio diffusion,

    Z. Evans, C. Carr, J. Taylor, S. H. Hawley, and J. Pons, “Fast timing-conditioned latent audio diffusion,” in Forty-first International Conference on Machine Learn- ing, 2024

    2024

  22. [22]

    Long-form music generation with latent diffusion,

    Z. Evans, J. D. Parker, C. Carr, Z. Zukowski, J. Taylor, and J. Pons, “Long-form music generation with latent diffusion,” inProceedings of the 25th International So- ciety for Music Information Retrieval Conference, IS- MIR 2024, San Francisco, California, USA and Online, November 10-14, 2024, 2024

    2024

  23. [23]

    Stable audio open,

    ——, “Stable audio open,” inICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025

    2025

  24. [24]

    Music2latent: Consistency autoencoders for latent audio compression,

    M. Pasini, S. Lattner, and G. Fazekas, “Music2latent: Consistency autoencoders for latent audio compression,” inProceedings of the 25th International Society for Music Information Retrieval Conference, ISMIR 2024, San Francisco, California, USA and Online, November 10-14, 2024, 2024

    2024

  25. [25]

    Music2latent2: Audio compression with sum- mary embeddings and autoregressive decoding,

    ——, “Music2latent2: Audio compression with sum- mary embeddings and autoregressive decoding,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  26. [26]

    Consistency Models,

    Y . Song, P. Dhariwalet al., “Consistency Models,” May 2023, arXiv:2303.01469 [cs, stat]

    Pith/arXiv arXiv 2023

  27. [27]

    Improved techniques for training consistency models,

    Y . Song and P. Dhariwal, “Improved techniques for training consistency models,”arXiv preprint arXiv:2310.14189, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Fréchet audio distance: A reference-free metric for evaluating music enhance- ment algorithms,

    K. Kilgour, M. Zuluagaet al., “Fréchet audio distance: A reference-free metric for evaluating music enhance- ment algorithms,” in20th Annual Conference of the International Speech Communication Association (IN- TERSPEECH), Sep. 2019

    2019

  29. [29]

    Neural discrete representation learning,

    A. van den Oord, O. Vinyalset al., “Neural discrete representation learning,” inAdvances in Neural Infor- mation Processing Systems 30, Dec. 2017

    2017

  30. [30]

    Generating diverse high-fidelity images with VQ-V AE-2,

    A. Razavi, A. van den Oordet al., “Generating diverse high-fidelity images with VQ-V AE-2,” inAdvances in Neural Information Processing Systems 32, Dec. 2019

    2019

  31. [31]

    An image is worth 32 tokens for reconstruc- tion and generation,

    Q. Yu, M. Weber, X. Deng, X. Shen, D. Cremers, and L.- C. Chen, “An image is worth 32 tokens for reconstruc- tion and generation,”arXiv preprint arXiv:2406.07550, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    La- tent consistency models: Synthesizing high-resolution images with few-step inference,

    S. Luo, Y . Tan, L. Huang, J. Li, and H. Zhao, “La- tent consistency models: Synthesizing high-resolution images with few-step inference,”arXiv preprint arXiv:2310.04378, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    Comospeech: One-step speech and singing voice synthesis via consistency model,

    Z. Ye, W. Xueet al., “Comospeech: One-step speech and singing voice synthesis via consistency model,” in Proceedings of the 31st ACM International Conference on Multimedia, MM 2023, Ottawa, ON, Canada, 29 October 2023- 3 November 2023, 2023

    2023

  34. [34]

    Denoising Diffusion Implicit Models,

    J. Song, C. Menget al., “Denoising Diffusion Implicit Models,” in9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021

    2021

  35. [35]

    Finite scalar quantization: VQ-V AE made simple,

    F. Mentzer, D. Minnen, E. Agustsson, and M. Tschan- nen, “Finite scalar quantization: VQ-V AE made simple,” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024

    2024

  36. [36]

    Estimat- ing or propagating gradients through stochastic neu- rons for conditional computation,

    Y . Bengio, N. Léonard, and A. Courville, “Estimat- ing or propagating gradients through stochastic neu- rons for conditional computation,”arXiv preprint arXiv:1308.3432, 2013

    Pith/arXiv arXiv 2013

  37. [37]

    DRUMGAN: synthesis of drum sounds with timbral feature conditioning using generative adversarial networks,

    J. Nistal, S. Lattneret al., “DRUMGAN: synthesis of drum sounds with timbral feature conditioning using generative adversarial networks,” inProceedings of the 21th International Society for Music Information Re- trieval Conference (ISMIR), Oct. 2020

    2020

  38. [38]

    Comparing rep- resentations for audio synthesis using generative adver- sarial networks,

    J. Nistal, S. Lattner, and G. Richard, “Comparing rep- resentations for audio synthesis using generative adver- sarial networks,” in28th European Signal Processing Conference (EUSIPCO), Jan. 2020

    2020

  39. [39]

    Speech enhancement and dereverberation with diffusion-based generative mod- els,

    J. Richter, S. Welkeret al., “Speech enhancement and dereverberation with diffusion-based generative mod- els,”IEEE ACM Trans. Audio Speech Lang. Process., vol. 31, 2023

    2023

  40. [40]

    Attention is all you need,

    A. Vaswani, N. Shazeeret al., “Attention is all you need,” inAdvances in Neural Information Processing Systems 30, Dec. 2017

    2017

  41. [41]

    Elucidating the Design Space of Diffusion-Based Generative Models,

    T. Karras, M. Aittalaet al., “Elucidating the Design Space of Diffusion-Based Generative Models,” Oct. 2022, arXiv:2206.00364 [cs, stat]

    Pith/arXiv arXiv 2022

  42. [42]

    Consistency models made easy,

    Z. Geng, A. Pokle, W. Luo, J. Lin, and J. Z. Kolter, “Consistency models made easy,” inThe Thirteenth In- ternational Conference on Learning Representations, 2025

    2025

  43. [43]

    Scaling transformers for low- bitrate high-quality speech coding,

    J. D. Parker, A. Smirnov, J. Pons, C. Carr, Z. Zukowski, Z. Evans, and X. Liu, “Scaling transformers for low- bitrate high-quality speech coding,”arXiv preprint arXiv:2411.19842, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  45. [45]

    Training compute- optimal large language models,

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hen- dricks, J. Welbl, A. Clarket al., “Training compute- optimal large language models,”arXiv preprint arXiv:2203.15556, 2022

    Pith/arXiv arXiv 2022

  46. [46]

    On the variance of the adaptive learning rate and beyond,

    L. Liu, H. Jianget al., “On the variance of the adaptive learning rate and beyond,” in8th International Confer- ence on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020

    2020

  47. [47]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  48. [48]

    Llama 2: Open foundation and fine- tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Alma- hairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine- tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  49. [49]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Ka- dian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    The mtg-jamendo dataset for automatic music tagging,

    D. Bogdanov, M. Wonet al., “The mtg-jamendo dataset for automatic music tagging,” inMachine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML 2019), Long Beach, CA, United States, 2019

    2019

  51. [51]

    Icassp 2022 deep noise sup- pression challenge,

    H. Dubey, V . Gopalet al., “Icassp 2022 deep noise sup- pression challenge,” inIEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022, 2022

    2022

  52. [52]

    M4singer: A multi-style, multi-singer and musical score provided mandarin singing corpus,

    L. Zhang, R. Li, S. Wang, L. Deng, J. Liu, Y . Ren, J. He, R. Huang, J. Zhu, X. Chen, and Z. Zhao, “M4singer: A multi-style, multi-singer and musical score provided mandarin singing corpus,” inAdvances in Neural In- formation Processing Systems, vol. 35, 2022, pp. 6914– 6926

    2022

  53. [53]

    SDR - half-baked or well done?

    J. L. Roux, S. Wisdomet al., “SDR - half-baked or well done?” inIEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019, 2019

    2019

  54. [54]

    Visqol: an objective speech quality model,

    A. Hines, J. Skoglundet al., “Visqol: an objective speech quality model,”EURASIP J. Audio Speech Mu- sic. Process., vol. 2015, 2015

    2015

  55. [55]

    Objective assessment of per- ceptual audio quality using visqolaudio,

    C. Sloan, N. Harteet al., “Objective assessment of per- ceptual audio quality using visqolaudio,”IEEE Trans. Broadcast., vol. 63, no. 4, 2017

    2017

  56. [56]

    Visqol v3: An open source production ready objective speech and audio met- ric,

    M. Chinen, F. S. C. Limet al., “Visqol v3: An open source production ready objective speech and audio met- ric,” inTwelfth International Conference on Quality of Multimedia Experience, QoMEX 2020, Athlone, Ireland, May 26-28, 2020, 2020

    2020

  57. [57]

    CNN architectures for large-scale audio clas- sification,

    S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gem- meke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. W. Wilson, “CNN architectures for large-scale audio clas- sification,” in2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-...

    2017

  58. [58]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chenet al., “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” inIEEE Interna- tional Conference on Acoustics, Speech and Signal Pro- cessing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023, 2023

    2023

  59. [59]

    Correlation of fr \’echet au- dio distance with human perception of environmen- tal audio is embedding dependant,

    M. Tailleur, J. Leeet al., “Correlation of fr \’echet au- dio distance with human perception of environmen- tal audio is embedding dependant,”arXiv preprint arXiv:2403.17508, 2024

    Pith/arXiv arXiv 2024

  60. [60]

    Flow straight and fast: Learning to generate and transfer data with rectified flow,

    X. Liu, C. Gong, and Q. Liu, “Flow straight and fast: Learning to generate and transfer data with rectified flow,” inThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023

    2023

  61. [61]

    Scalable diffusion models with transformers,

    W. Peebles and S. Xie, “Scalable diffusion models with transformers,” inIEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, Octo- ber 1-6, 2023, 2023

    2023

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.