Pith. sign in

REVIEW 3 major objections 6 minor 105 references

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Causal audio language models with continuous tokens and masked next-token prediction match or beat diffusion-based text-to-audio systems on AudioCaps.

desk verdict Solid empirical paper: continuous-token causal LMs with MNTP genuinely close the gap to diffusion SOTA on AudioCaps, but the headline cross-model ranking rests on a mixed evaluation pipeline that the paper itself shows to be protocol-sensitive. read the letter →

arxiv 2507.09834 v1 pith:TYL3QKMK submitted 2025-07-14 eess.AS cs.CVcs.SD

Shu-wen Yang , Byeonggeun Kim , Kuan-Po Huang , Qingming Tang , Huy Phan , Bo-Ru Lu , Harsha Sundar , Shalini Ghosh
show 3 more authors
Hung-yi Lee Chieh-Chi Kao Chao Wang
This is my paper
classification eess.AScs.CVcs.SD
keywords text-to-audiogenerationaudiolanguagemodelingcontinuous-valuedtokensmaskednext-tokenpredictiontoken-wisediffusionlosscausalTransformerdecoderstreamableCapsbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to prove that a causal language model can generate high-fidelity audio without quantizing sound into discrete codes, and that it can do so well enough to match systems built on bidirectional diffusion modeling. The authors treat the low-dimensional latent vectors produced by a fixed audio autoencoder as 'continuous-valued tokens,' and they train a standard Transformer decoder to predict the next token with a token-wise diffusion head instead of a cross-entropy classifier. On top of that, they add masked next-token prediction (MNTP): during training, a random fraction of past tokens is dropped and the model learns to predict a random future token, flagged by a target positional embedding; ordinary next-token prediction is the special case with no dropping. Their 462M-parameter AudioMNTP model reports the best FD and FAD among all compared systems on the AudioCaps benchmark, beating much larger discrete-token language models and matching diffusion models like AudioLDM2 and Tango2, while remaining streamable. If the claims hold, LLM-style scaling and serving infrastructure would apply to direct audio generation.

What carries the argument

The argument runs on three coupled pieces. First, continuous-valued tokens: a frozen variational autoencoder converts a 64-band Mel-spectrogram into a 2-D latent map, which is patched and serialized into 256 tokens of dimension 128 per 10-second clip, so the language model sees a short sequence of real vectors instead of a long sequence of discrete codes. Second, the token-wise diffusion head: a small MLP that takes a noised token, the conditioning vector from the Transformer decoder, and a noise level, and predicts the added Gaussian noise; the denoising objective substitutes for the cross-entropy loss, leaving the backbone a standard causal Transformer. Third, masked next-token prediction (MNTP): each training pass samples a masking ratio from a mixture of a normal and a truncated normal distribution over [0,1], drops that fraction of tokens outright, and trains the model to predict a random future token from the surviving past context, using a target positional embedding to disambiguate which future position is being predicted; at inference the target embedding is simply the next position, so decoding is identical to ordinary next-token prediction.

What would settle it

Re-train AudioMNTP and AudioNTP with a different frozen tokenizer (for example, a VAE trained on the same Mel-spectrograms but with different capacity, or a waveform autoencoder), keeping model size, data, and the AudioLDM evaluation pipeline fixed: if the MNTP gains over NTP evaporate or the continuous-token advantage over a discrete-token baseline flips, the central claims are tied to the specific token space rather than to the learning task. A cheaper check is to recompute FAD from scratch with an independent implementation for AudioMNTP Large, AudioLDM2, and Tango2 on identical generated audio files; if the ranking or the reported 1.22 versus 2.12 gap does not survive, the headline comparisons are toolkit-dependent.

Watch

Extended reading notes

Core claim

The paper's central claim is that the two commitments that made causal audio language models underperform—discrete token dictionaries and cross-entropy next-token loss—are optional, not essential. Reinterpreting the continuous latents used by latent diffusion models as tokens, the paper trains a causal Transformer decoder to model the next-token density with a small MLP diffusion head that denoises a Gaussian-corrupted token conditioned on the decoder's context vector. This alone, AudioNTP, beats the discrete-token AudioGen at a third of its size. The paper then argues that masked prediction, not bidirectionality, is what gives masked generative models their edge, and proposes masked next-token prediction as a causal analogue: random tokens are dropped before each forward pass and the model predicts a future token from the remaining sparse context, with a learned target positional embedding telling it which future position to predict. AudioMNTP Large, at 462M parameters and trained on roughly 1,000 hours of audio, attains FD 14.30 and FAD 1.22 on AudioCaps—the best of either metric among every system the paper lists, including 712M–937M bidirectional diffusion models—and the authors note their model is uni-directional and much smaller than those baselines, which they take as evidence that further scaling would push performance higher.

Load-bearing premise

The approach assumes that the frozen audio encoder and vocoder borrowed from the AudioLDM pipeline preserve essentially all information a listener cares about and introduce no systematic bias, because the language model never sees the waveform directly and the headline metrics are computed with that same pipeline's evaluation toolkit; if those components are lossy or biased, they cap generation quality and the reported FAD and FD gains could be partly artifacts of the evaluation setup.

Editorial extensions

If this is right

  • A causal, streamable Transformer decoder can reach objective audio quality on AudioCaps comparable to 712M–937M bidirectional latent diffusion models while using only 462M parameters.
  • Continuous-valued VAE latents shrink the per-clip sequence from roughly 5,000 discrete tokens to 256 continuous tokens, cutting training cost and making LM-style scaling of audio more practical.
  • Masked next-token prediction improves the decoding task itself: AudioMNTP Base beats plain next-token prediction by 26% relative FAD at the same size, and the gain carries over when the model is scaled to 462M.
  • With the prediction head as the only swapped component, the framework is positioned to inherit LLM serving infrastructure (KV caching, paged attention) and to be integrated into multi-modal LMs as a drop-in audio output head.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not train its own discrete-token baseline (footnote 11), so the continuous-token advantage rests on published AudioGen and UniAudio numbers rather than a same-data, same-pipeline ablation in this work.
  • Because only the AudioLDM tokenizer is tested, an untested extension is whether the MNTP recipe transfers to other autoencoder token spaces (or to image and speech latent sequences); if the gains come mainly from the learning task, the same recipe should transfer.
  • The target positional embedding makes skip-ahead prediction an explicit capability, which suggests a speculative-decoding scheme the paper mentions only in passing: predict several future tokens in parallel and then verify them left-to-right, trading computation for lower streaming latency.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes generative audio language modeling with continuous-valued tokens and a novel masked next-token prediction (MNTP) task. The authors replace discrete audio tokens with continuous latent vectors obtained from the AudioLDM VAE/Hifi-GAN pipeline, model each next-token distribution with a small diffusion head, and add MNTP, in which randomly dropped tokens are skipped and a target positional embedding tells the model which future position to predict. The method is evaluated on AudioCaps and WavCaps with objective metrics (FAD, FD, KL, IS, CLAP) and human evaluations. The paper claims that AudioNTP outperforms AudioGen, that MNTP further improves next-token prediction, and that AudioMNTP Large reaches the best FAD/FD among the models listed in Table 1 while being smaller and streamable.

Significance. The central technical idea is credible and the internal ablations are thorough. If the cross-model numbers can be established under a common evaluation protocol, the paper would be a significant demonstration that causal language models with continuous tokens can rival diffusion models in text-to-audio generation while offering streaming and efficiency advantages. Strengths include the controlled comparison of NTP versus MNTP in Tables 3 and 6, the diffusion-head-size ablation, the masking-schedule ablations, the human evaluation, and the real-time-factor analysis. The main weakness is that the headline Table 1 ranking rests on protocol-mixed baseline numbers, and the paper's own re-inference of AudioLDM2 shows that the evaluation protocol alone shifts FAD by 14-22%.

major comments (3)
  1. [Table 1 / Section 5.1] The claim that AudioMNTP Large achieves the best FAD and FD across all listed models is not supported by comparable measurements. Table 1 combines published numbers from heterogeneous evaluation pipelines (AudioGen, UniAudio, Tango, Tango2, Make-An-Audio2, MAGNet) with AudioLDM-toolkit numbers for the authors' models, and only AudioLDM2 is re-inferred with the same toolkit. The authors' own re-inference moves AudioLDM2-Full FAD from 1.78 to 2.17 and AudioLDM2-Full-Large from 1.86 to 2.12, a 14-22% shift that is the same order as several headline gaps, such as AudioMNTP Large FAD 1.22 versus AudioGen Large FAD 1.82. Without re-running all baselines through the same evaluation pipeline, the 'best FAD/FD' and 'SOTA-level' conclusions in the abstract and Section 6 do not follow. The internal AudioNTP-versus-AudioMNTP comparisons and ablations are not affected, but the headline claims should be restricted to controlled comparisons or supported by re-inference of the open baselines.
  2. [Tables 1, 3-6, 8 / Section 5] No confidence intervals or significance tests are reported for any objective metric. FAD, FD, KL, IS, and CLAP are computed on a single generated set, and the paper repeatedly uses language such as 'significant improvements' (Abstract and Section 5.1) and interprets small differences such as Table 3 (G) versus (J) and Table 6 (E) versus (F) as meaningful. Given that the re-inference of AudioLDM2 already demonstrates substantial evaluation sensitivity, the lack of uncertainty quantification is a load-bearing gap for the quantitative claims, and at least a seed-based or bootstrap-based variance estimate is needed.
  3. [Section 4 / Appendix F.1] The paper relies entirely on a fixed, pre-trained AudioLDM VAE and Hifi-GAN vocoder as the tokenizer and de-tokenizer, as stated in Section 4 and Appendix F.1. This is an acknowledged dependency, but it interacts with the claim that continuous-valued tokens are superior to discrete tokens: because the discrete baselines (AudioGen, UniAudio, MAGNet) use different tokenizers and reconstruction pipelines, part of the observed FAD/KL difference may reflect the evaluation and reconstruction pipeline rather than the language-modeling formulation. The paper should state this limitation explicitly in Section 5.1 and temper the 'continuous-valued tokens are superior' claim accordingly.
minor comments (6)
  1. [Abstract vs. Section 5.1] The abstract reports '20% and 40% relative gains on AudioCaps in Frechet Audio Distance (FAD) and Kullback-Leibler (KL)' while Section 5.1 says '20%, 40% relative improvements over AudioGen Base on FD and KL scores.' Since AudioGen has no FD value in Table 1 and the FAD improvement is 20% relative, the abstract's wording is consistent with the table but the Section 5.1 wording is inconsistent and should be corrected to FAD.
  2. [Table 1] Several rows in Table 1 (AudioLDM2-AC, AudioLDM2-Full, AudioLDM2-Full-Large) have missing FD, IS, and CLAP values, and the re-inference rows report FD values around 32-33 that are far outside the range of other FD values in the table. The table should clearly annotate which numbers come from published papers, which are re-inferred, and why the re-inferred FD values are so different.
  3. [Appendix B / Section 4] The masking schedule is a mixture of a normal and a truncated normal distribution with equal weights, but no sensitivity analysis for the mixture weights is reported; since the schedule is a free parameter, a brief ablation or a statement that equal weights were chosen without tuning would be helpful.
  4. [Section 5.2 / Appendix G.2] The human evaluation is based on 20 samples with at least 10 raters per sample, but no inter-annotator agreement or significance testing is reported, and the speech/non-speech split is based on a small set of prompt keywords. The claims in Section 5.2 would be strengthened by reporting confidence intervals or a paired comparison.
  5. [Equations (2)-(3)] Equation (2) writes the loss as an expectation over epsilon and t but does not explicitly show the expectation over data (a, w) and the token sequence; adding the data expectation would make the training objective fully precise.
  6. [Figure 1] Figure 1 contains both 'Discrete-valued audio tokens' and 'Continuous-valued audio tokens' labels, and the caption says the tokens are continuous; the figure should be relabeled to avoid confusing the reader about which representation is used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core claims are empirical comparisons supported by controlled internal ablations; the cross-model benchmark-protocol mismatch is a validity concern, not a definitional or fitted-input circularity.

full rationale

The derivation chain is self-contained rather than circular. AudioNTP is defined by a standard causal factorization with a diffusion objective (Eq. 1-2), and MNTP is defined as a randomized drop-and-skip-token training task with target positional embeddings; the claim that MNTP helps NTP decoding is an empirical result verified by ablations in Tables 3, 6, and 8, not a consequence of the definition. No fitted parameter is renamed as a prediction: hyperparameters such as CFG scale and temperature are ablated in Appendix C, and the masking schedule is ablated in Appendix B. There is no load-bearing self-citation chain; the paper cites MAR, AudioLDM, and AudioGen as external prior work, and the authors of those works do not overlap with the present authors. The strongest concern in Table 1 is that most baseline FAD/FD numbers are taken from their original papers while AudioNTP/MNTP and the two AudioLDM2 re-inferences use the AudioLDM evaluation toolkit, so the headline 'best FAD/FD' comparison is not fully protocol-matched. That is a benchmark-validity and reproducibility concern, not a circular reduction: the comparison is not entailed by the model definitions, the loss equations, or any fitted parameter. The paper's own re-inference rows for AudioLDM2 actually disclose the protocol sensitivity rather than hiding it. Under the hard rule that circularity requires a demonstrated reduction of a claimed result to its own inputs, none of the paper's equations or ablation arguments exhibit such a reduction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The MNTP task and target positional embeddings are architectural modifications, not new entities. The main load-bearing assumptions are the fidelity of the fixed tokenizer/vocoder and the comparability of evaluation metrics across different pipelines.

free parameters (3)
  • CFG scale omega_0 = 7
    Chosen by sweeping omega_0 in Figure 6(a). The best performance is around 7, and it is set as default throughout the paper.
  • Sampling temperature tau = 1
    Figure 6(b) shows no single tau wins on all metrics; tau=1 is chosen as the default, affecting the diversity/fidelity trade-off.
  • Masking schedule mixture weights = Normal and truncated normal averaged with equal weights
    The mixture of normal and truncated normal distributions is introduced in Section B.1 and Appendix B.2. The equal weights are not derived, and the schedule is tuned on the same AudioCaps dataset used for evaluation.
assumptions (3)
  • domain assumption The pre-trained AudioLDM VAE and Hifi-GAN vocoder are fixed and lossless enough for the language model to generate high-quality audio.
    Section 4 and Appendix F.1 inherit the AudioLDM tokenization pipeline. The paper provides no analysis of reconstruction error or its effect on the final metrics.
  • domain assumption FAD, FD, KL, IS, and CLAP computed with the AudioLDM evaluation toolkit are reliable and comparable across different tokenizers and vocoders.
    The comparison against AudioGen, Tango2, and AudioLDM2 relies on this assumption. The paper notes the re-inference values of AudioLDM2 differ from the published values, which shows the metrics are sensitive to the exact evaluation pipeline.
  • domain assumption Training on the same dataset used for evaluation (AudioCaps) is a standard and acceptable practice for audio generation research.
    The model is trained on AudioCaps and WavCaps, and evaluated on the AudioCaps evaluation set. The paper does not discuss the train/test overlap beyond the standard split, so the evaluation might be optimistic compared to training on other datasets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction." pith.science (2026). https://pith.science/paper/TYL3QKMK

@misc{pith2026250709834,
  author       = {Pith},
  title        = {Pith review of: Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TYL3QKMK}},
  note         = {Machine review of arXiv:2507.09834}
}
read the original abstract

Autoregressive next-token prediction with the Transformer decoder has become a de facto standard in large language models (LLMs), achieving remarkable success in Natural Language Processing (NLP) at scale. Extending this paradigm to audio poses unique challenges due to its inherently continuous nature. We research audio generation with a causal language model (LM) without discrete tokens. We leverage token-wise diffusion to model the continuous distribution of the next continuous-valued token. Our approach delivers significant improvements over previous discrete solution, AudioGen, achieving 20% and 40% relative gains on AudioCaps in Frechet Audio Distance (FAD) and Kullback-Leibler (KL) divergence, respectively. Additionally, we propose a novel masked next-token prediction task that incorporates masked prediction into the causal LM framework. On AudioCaps, the innovation yields 41% and 33% relative FAD improvements over AudioGen Base (285M) and AudioGen Large (1B) models, respectively, and is on par with the state-of-the-art (SOTA) diffusion models. Furthermore, we achieve these results with significantly fewer parameters -- 193M for our Base and 462M for our Large models.

Figures

Figures reproduced from arXiv: 2507.09834 by the authors.

Figure 1
Figure 1. Causal Language Modeling on Continuous-valued Audio Tokens with Masked Next-Token Prediction. The audio tokens are low-dimensional continuous latent. We use a standard Transformer decoder for the audio language modeling. Our masked next-token prediction learns to predict any future token given any subset of the past tokens, which turns out benefit the standard next￾token prediction and rival the bidirectional diffus… view at source ↗
Figure 2
Figure 2. The framework of the continuous-valued audio token proposal. The waveform is transformed into the Mel-spectrogram, subsequently encoded into continuous-valued tokens. The Trans￾former decoder learns the next token prediction on these tokens via the token-wise diffusion loss with a small MLP diffusion head. This framework is termed AudioNTP. tokens in audio language modeling [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The framework of the masked next-token prediction (MNTP). The continuous-valued tokens are first masked (dropped) to form a shorter sequence. The Transformer decoder then learns the next-token prediction on the dropped sequence with the diffu￾sion loss. This framework is termed AudioMNTP. Training. Before feeding the token sequence x = {x 1 , ..., xn} into Cθ, we first apply masking on it. We denote the mask as v = … view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Comparing the speed and quality tradeoff. The RTF is real-time factor, calculated by processing duration clip duration . Our AudioMNTP Large reach near real-time streamable generation with a single NVIDIA A100 GPU. Both AudioMNTP Base and Large achieve the lowest FAD w…
Figure 5
Figure 5. Figure 5: Visualizing the masking schedules. We first sample a masking ratio from the schedule, a probability distribution over [0, 1], and then sample the masking positions in the token sequence based on the ratio. In (C), we average (A) and (B) with equal weights. B.2. Ablatin…
Figure 6
Figure 6. Figure 6: (b) shows that sampling temperatures have varying impacts on different metrics. As a result, given a specific metric to optimize, it is helpful to tune the temperature. However, no single τ achieves the best performance across all metrics. Therefore, we set τ = 1 as th…
Figure 7
Figure 7. Figure 7: Comparing the differences between MLM and MNTP. D.2. Comparing MNTP to Next-Token Prediction. We can view MNTP as a Generalized form of Causal Language Modeling (GCLM), where the model predicts any future timestamp given any subset of past information. We illustrate th…
Figure 8
Figure 8. Figure 8: Visualizing the generalized causal language model. (a) shows the masking patterns. We enumerate the masking patterns with binary numbers where a masked position is denoted by 1 (white) and a unmasked position is denoted by 0 (gray). (b) shows the corresponding dropped …
Figure 9
Figure 9. Figure 9: Visualizing the differences between the ablation variants. The subfigure IDs match the IDs in [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

105 extracted references · 40 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., and Zettlemoyer, L

    Aghajanyan, A., Huang, P.-Y. B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., and Zettlemoyer, L. Cm3: A causal masked multimodal model of the internet. ArXiv, abs/2201.07520, 2022. URL https://api.semanticscholar.org/CorpusID:246035820

    arXiv 2022

  3. [3]

    I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., et al

    Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023

    arXiv 2023

  4. [4]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 0 12449--12460, 2020

    2020

  5. [5]

    Efficient self-supervised learning with contextualized target representations for vision, speech and language

    Baevski, A., Babu, A., Hsu, W.-N., and Auli, M. Efficient self-supervised learning with contextualized target representations for vision, speech and language. In International Conference on Machine Learning, pp.\ 1416--1429. PMLR, 2023

    2023

  6. [6]

    Efficient training of language models to fill in the middle

    Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., and Chen, M. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255, 2022

    arXiv 2022

  7. [7]

    P., Whitman, B., and Lamere, P

    Bertin-Mahieux, T., Ellis, D. P., Whitman, B., and Lamere, P. The million song dataset. In Proceedings of the 12th International Conference on Music Information Retrieval ( ISMIR 2011) , 2011

    2011

  8. [8]

    Audiolm: a language modeling approach to audio generation

    Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grangier, D., Tagliasacchi, M., et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31: 0 2523--2533, 2023

    2023

Show all 105 references
  1. [9]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  2. [10]

    Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 11315--11325, 2022

    2022

  3. [11]

    Vggsound: A large-scale audio-visual dataset

    Chen, H., Xie, W., Vedaldi, A., and Zisserman, A. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 721--725. IEEE, 2020 a

    2020

  4. [12]

    Generative pretraining from pixels

    Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pixels. In International conference on machine learning, pp.\ 1691--1703. PMLR, 2020 b

    2020

  5. [13]

    W., Sutton, C., Gehrmann, S., et al

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  6. [14]

    W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al

    Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25 0 (70): 0 1--53, 2024

    2024

  7. [15]

    and Glass, J

    Chung, Y.-A. and Glass, J. Improved speech representations with multi-target autoregressive predictive coding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.\ 2353--2358, 2020

    2020

  8. [16]

    Simple and controllable music generation

    Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., and D \'e fossez, A. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  9. [17]

    High fidelity neural audio compression

    D \'e fossez, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression. Transactions on Machine Learning Research, 2023

    2023

  10. [18]

    Audio retrieval with wavtext5k and clap training

    Deshmukh, S., Elizalde, B., and Wang, H. Audio retrieval with wavtext5k and clap training. In INTERSPEECH 2023, pp.\ 2948--2952, 2023. doi:10.21437/Interspeech.2023-1136

    2023 doi

  11. [19]

    and Nichol, A

    Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34: 0 8780--8794, 2021

    2021

  12. [20]

    Clotho: An audio captioning dataset

    Drossos, K., Lipping, S., and Virtanen, T. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 736--740. IEEE, 2020

    2020

  13. [21]

    M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A

    Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pp.\ 5547--5569. PMLR, 2022

    2022

  14. [22]

    Taming transformers for high-resolution image synthesis

    Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 12873--12883, 2021

    2021

  15. [23]

    Freesound datasets: a platform for the creation of open audio datasets

    Fonseca, E., Pons Puig, J., Favory, X., Font Corbera, F., Bogdanov, D., Ferraro, A., Oramas, S., Porter, A., and Serra, X. Freesound datasets: a platform for the creation of open audio datasets. In Hu X, Cunningham SJ, Turnbull D, Duan Z, editors. Proceedings of the 18th ISMIR...

    2017

  16. [24]

    Fsd50k: an open dataset of human-labeled sound events

    Fonseca, E., Favory, X., Pons, J., Font, F., and Serra, X. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 0 829--852, 2021

    2021

  17. [25]

    Incoder: A generative model for code infilling and synthesis

    Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, S., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2023

    2023

  18. [26]

    F., Ellis, D

    Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp.\ 776...

    2017

  19. [27]

    Text-to-audio generation using instruction-tuned LLM and latent diffusion model

    Ghosal, D., Majumder, N., Mehrish, A., and Poria, S. Text-to-audio generation using instruction-tuned LLM and latent diffusion model. CoRR, abs/2304.13731, 2023. doi:10.48550/ARXIV.2304.13731. URL https://doi.org/10.48550/arXiv.2304.13731

    2023 doi arXiv

  20. [28]

    Ast: Audio spectrogram transformer

    Gong, Y., Chung, Y.-A., and Glass, J. Ast: Audio spectrogram transformer. In Interspeech 2021, pp.\ 571--575, 2021. doi:10.21437/Interspeech.2021-698

    2021 doi

  21. [29]

    Prompttts: Controllable text-to-speech with text descriptions

    Guo, Z., Leng, Y., Wu, Y., Zhao, S., and Tan, X. Prompttts: Controllable text-to-speech with text descriptions. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 1--5. IEEE, 2023

    2023

  22. [30]

    Masked autoencoders are scalable vision learners

    He, K., Chen, X., Xie, S., Li, Y., Doll \'a r, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 16000--16009, 2022

    2022

  23. [31]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  24. [32]

    P., Fonseca, E., Jansen, A., Liu, C., Moore, R

    Hershey, S., Ellis, D. P., Fonseca, E., Jansen, A., Liu, C., Moore, R. C., and Plakal, M. The benefit of temporally-strong labels in audio event classification. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 366--370...

    2021

  25. [33]

    Denoising diffusion probabilistic models

    Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020

    2020

  26. [34]

    A., Welbl, J., Clark, A., et al

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processi...

    2022

  27. [35]

    Y., Zhou, T., Wu, Y., Song, X., Song, X., and Zhou, D

    Hou, L., Pang, R. Y., Zhou, T., Wu, Y., Song, X., Song, X., and Zhou, D. Token dropping for efficient bert pretraining. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 3774--3784, 2022

    2022

  28. [36]

    Make-an-audio 2: Temporal-enhanced text-to-audio generation

    Huang, J., Ren, Y., Huang, R., Yang, D., Ye, Z., Zhang, C., Liu, J., Yin, X., Ma, Z., and Zhao, Z. Make-an-audio 2: Temporal-enhanced text-to-audio generation. CoRR, abs/2305.18474, 2023 a . doi:10.48550/ARXIV.2305.18474. URL https://doi.org/10.48550/arXiv.2305.18474

    2023 doi arXiv

  29. [37]

    Masked autoencoders that listen

    Huang, P.-Y., Xu, H., Li, J., Baevski, A., Auli, M., Galuba, W., Metze, F., and Feichtenhofer, C. Masked autoencoders that listen. Advances in Neural Information Processing Systems, 35: 0 28708--28720, 2022

    2022

  30. [38]

    Multi-singer: Fast multi-singer singing voice vocoder with a large-scale corpus

    Huang, R., Chen, F., Ren, Y., Liu, J., Cui, C., and Zhao, Z. Multi-singer: Fast multi-singer singing voice vocoder with a large-scale corpus. In Proceedings of the 29th ACM International Conference on Multimedia, pp.\ 3945--3954, 2021

    2021

  31. [39]

    Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models

    Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., Ye, Z., Liu, J., Yin, X., and Zhao, Z. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pp.\ 13916--13932. PMLR, 2023 b

    2023

  32. [40]

    Audiogpt: Understanding and generating speech, music, sound, and talking head

    Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z., Wu, Y., Hong, Z., Huang, J., Liu, J., et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 23802--23804, 2024

    2024

  33. [41]

    Libri-light: A benchmark for asr with limited or no supervision

    Kahn, J., Riviere, M., Zheng, W., Kharitonov, E., Xu, Q., Mazar \'e , P.-E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech...

    2020

  34. [42]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  35. [43]

    Kenton, J. D. M.-W. C. and Toutanova, L. K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1. Minneapolis, Minnesota, 2019

    2019

  36. [44]

    Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms

    Kilgour, K., Zuluaga, M., Roblek, D., and Sharifi, M. Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms. In Interspeech 2019, pp.\ 2350--2354, 2019. doi:10.21437/Interspeech.2019-2219

    2019 doi

  37. [45]

    D., Kim, B., Lee, H., and Kim, G

    Kim, C. D., Kim, B., Lee, H., and Kim, G. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),...

    2019

  38. [46]

    Kingma, D. P. and Welling, M. Auto-Encoding Variational Bayes . In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014

    2014

  39. [47]

    L., and Khudanpur, S

    Ko, T., Peddinti, V., Povey, D., Seltzer, M. L., and Khudanpur, S. A study on data augmentation of reverberant speech for robust speech recognition. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp.\ 5220--5224. IEEE, 2017

    2017

  40. [48]

    Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis

    Kong, J., Kim, J., and Bae, J. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33: 0 17022--17033, 2020

    2020

  41. [49]

    Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities

    Kong, Z., Goel, A., Badlani, R., Ping, W., Valle, R., and Catanzaro, B. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. arXiv preprint arXiv:2402.01831, 2024 a

    2024 arXiv

  42. [50]

    Improving text-to-audio models with synthetic captions

    Kong, Z., Lee, S.-g., Ghosal, D., Majumder, N., Mehrish, A., Valle, R., Poria, S., and Catanzaro, B. Improving text-to-audio models with synthetic captions. In Proc. SynData4GenAI 2024, pp.\ 1--5, 2024 b

    2024

  43. [51]

    Audiogen: Textually guided audio generation

    Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., D \' e fossez, A., Copet, J., Parikh, D., Taigman, Y., and Adi, Y. Audiogen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . Ope...

    2023

  44. [52]

    H., Gonzalez, J., Zhang, H., and Stoica, I

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp.\ 611--626, 2023

    2023

  45. [53]

    BART : Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

    Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. BART : Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault...

    2020 doi

  46. [54]

    Mage: Masked generative encoder to unify representation learning and image synthesis

    Li, T., Chang, H., Mishra, S., Zhang, H., Katabi, D., and Krishnan, D. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 2142--2152, 2023

    2023

  47. [55]

    Return of unconditional generation: A self-supervised representation generation method

    Li, T., Katabi, D., and He, K. Return of unconditional generation: A self-supervised representation generation method. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 a

    2024

  48. [56]

    Autoregressive image generation without vector quantization

    Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11838, 2024 b

    2024 arXiv

  49. [57]

    T., Ben-Hamu, H., Nickel, M., and Le, M

    Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023

    2023

  50. [58]

    H., Le, M., Vyas, A., Shi, B., Tjandra, A., and Hsu, W.-N

    Liu, A. H., Le, M., Vyas, A., Shi, B., Tjandra, A., and Hsu, W.-N. Generative pre-training for speech with flow matching. In The Twelfth International Conference on Learning Representations, 2024 a

    2024

  51. [59]

    Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D. Audioldm: text-to-audio generation with latent diffusion models. In Proceedings of the 40th International Conference on Machine Learning, pp.\ 21450--21474, 2023 a

    2023

  52. [60]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36, 2024 b

    2024

  53. [61]

    Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q., Tian, Q., Wang, Y., Wang, W., Wang, Y., and Plumbley, M. D. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE ACM Trans. Audio Speech Lang. Process. , 32: 0 2871--2883, 2024 c . doi:10.1109/TASL...

    2024

  54. [62]

    D., et al

    Liu, X., Zhu, Z., Liu, H., Yuan, Y., Cui, M., Huang, Q., Liang, J., Cao, Y., Kong, Q., Plumbley, M. D., et al. Wavjourney: Compositional audio creation with large language models. arXiv preprint arXiv:2307.14335, 2023 b

    2023 arXiv

  55. [63]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017. URL https://api.semanticscholar.org/CorpusID:53592270

    2017

  56. [64]

    Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization

    Majumder, N., Hung, C., Ghosal, D., Hsu, W., Mihalcea, R., and Poria, S. Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization. CoRR, abs/2404.09956, 2024. doi:10.48550/ARXIV.2404.09956. URL https://doi.org/10.48550/arXiv.2404.09956

    2024 doi arXiv

  57. [65]

    and Mesaros, A

    Mart \' n-Morat \'o , I. and Mesaros, A. What is the ground truth? reliability of multi-annotator data for audio tagging. In 2021 29th European Signal Processing Conference (EUSIPCO), pp.\ 76--80. IEEE, 2021

    2021

  58. [66]

    D., Zou, Y., and Wang, W

    Mei, X., Meng, C., Liu, H., Kong, Q., Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., and Wang, W. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  59. [67]

    Autoregressive speech synthesis without vector quantization

    Meng, L., Zhou, L., Liu, S., Chen, S., Han, B., Hu, S., Liu, Y., Li, J., Zhao, S., Wu, X., et al. Autoregressive speech synthesis without vector quantization. arXiv preprint arXiv:2407.08551, 2024

    2024 arXiv

  60. [68]

    Tut database for acoustic scene classification and sound event detection

    Mesaros, A., Heittola, T., and Virtanen, T. Tut database for acoustic scene classification and sound event detection. In 2016 24th European Signal Processing Conference (EUSIPCO), pp.\ 1128--1132. IEEE, 2016

    2016

  61. [69]

    Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In Proceedings of the International Conference on Machine Learning, 2021

    2021

  62. [70]

    Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  63. [71]

    V oice C raft: Zero-shot speech editing and text-to-speech in the wild

    Peng, P., Huang, P.-Y., Li, S.-W., Mohamed, A., and Harwath, D. V oice C raft: Zero-shot speech editing and text-to-speech in the wild. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (...

    2024 doi

  64. [72]

    Piczak, K. J. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pp.\ 1015--1018, 2015

    2015

  65. [73]

    Efficiently scaling transformer inference

    Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5: 0 606--624, 2023

    2023

  66. [74]

    Mls: A large-scale multilingual dataset for speech research

    Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., and Collobert, R. Mls: A large-scale multilingual dataset for speech research. In Interspeech 2020, pp.\ 2757--2761, 2020. doi:10.21437/Interspeech.2020-2826

    2020 doi

  67. [75]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  68. [76]

    High-resolution image synthesis with latent diffusion models

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10684--10695, 2022

    2022

  69. [77]

    Salamon, J., Jacoby, C., and Bello, J. P. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pp.\ 1041--1044, 2014

    2014

  70. [78]

    Edinburgh neural machine translation systems for wmt 16

    Sennrich, R., Haddow, B., and Birch, A. Edinburgh neural machine translation systems for wmt 16. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pp.\ 371--376, 2016

    2016

  71. [79]

    Aishell-3: A multi-speaker mandarin tts corpus

    Shi, Y., Bu, H., Xu, X., Zhang, S., and Li, M. Aishell-3: A multi-speaker mandarin tts corpus. In Interspeech 2021, pp.\ 2756--2760, 2021. doi:10.21437/Interspeech.2021-755

    2021 doi

  72. [80]

    Multimodal latent language modeling with next-token diffusion

    Sun, Y., Bao, H., Wang, W., Peng, Z., Dong, L., Huang, S., Wang, J., and Wei, F. Multimodal latent language modeling with next-token diffusion. arXiv preprint arXiv:2412.08635, 2024

    2024 arXiv

  73. [81]

    M., Hauth, A., Millican, K., et al

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  74. [82]

    Givt: Generative infinite-vocabulary transformers

    Tschannen, M., Eastwood, C., and Mentzer, F. Givt: Generative infinite-vocabulary transformers. In European Conference on Computer Vision, pp.\ 292--309. Springer, 2025

    2025

  75. [83]

    and Cook, P

    Tzanetakis, G. and Cook, P. Musical genre classification of audio signals. IEEE Transactions on speech and audio processing, 10 0 (5): 0 293--302, 2002

    2002

  76. [84]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  77. [85]

    Audiobox: Unified audio generation with natural language prompts

    Vyas, A., Shi, B., Le, M., Tjandra, A., Wu, Y.-C., Guo, B., Zhang, J., Zhang, X., Adkins, R., Ngan, W., et al. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312.15821, 2023

    2023 arXiv

  78. [86]

    GLUE : A multi-task benchmark and analysis platform for natural language understanding

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. GLUE : A multi-task benchmark and analysis platform for natural language understanding. In Linzen, T., Chrupa a, G., and Alishahi, A. (eds.), Proceedings of the 2018 EMNLP Workshop B lackbox NLP : Analyzing a...

    2018 doi

  79. [87]

    Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm

    Wang, X., Li, Y., Fu, C., Xie, L., Li, K., Sun, X., and Ma, L. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm. arXiv preprint arXiv:2411.00774, 2024 a

    2024 arXiv

  80. [88]

    Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition

    Wang, Y., Huang, R., Song, S., Huang, Z., and Huang, G. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. Advances in neural information processing systems, 34: 0 11960--11973, 2021

    2021

  81. [89]

    Opencpop: A high-quality open source chinese popular song corpus for singing voice synthesis

    Wang, Y., Wang, X., Zhu, P., Wu, J., Li, H., Xue, H., Zhang, Y., Xie, L., and Bi, M. Opencpop: A high-quality open source chinese popular song corpus for singing voice synthesis. In Interspeech 2022, pp.\ 4242--4246, 2022. doi:10.21437/Interspeech.2022-48

    2022 doi

  82. [90]

    Audio-agent: Leveraging llms for audio generation, editing and composition

    Wang, Z., Tai, Y.-W., and Tang, C.-K. Audio-agent: Leveraging llms for audio generation, editing and composition. arXiv preprint arXiv:2410.03335, 2024 b

    2024 arXiv

  83. [91]

    Diffusion models as masked autoencoders

    Wei, C., Mangalam, K., Huang, P.-Y., Li, Y., Fan, H., Xu, H., Wang, H., Xie, C., Yuille, A., and Feichtenhofer, C. Diffusion models as masked autoencoders. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 16284--16294, 2023

    2023

  84. [92]

    Next-gpt: Any-to-any multimodal llm

    Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.-S. Next-gpt: Any-to-any multimodal llm. In Forty-first International Conference on Machine Learning, 2024

    2024

  85. [93]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., and Dubnov, S. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (...

    2023

  86. [94]

    Uniaudio: An audio foundation model toward universal audio generation

    Yang, D., Tian, J., Tan, X., Huang, R., Liu, S., Chang, X., Shi, J., Zhao, S., Bian, J., Wu, X., Zhao, Z., Watanabe, S., and Meng, H. Uniaudio: An audio foundation model toward universal audio generation. CoRR, abs/2310.00704, 2023 a . doi:10.48550/ARXIV.2310.00704. URL https:...

    doi arXiv

  87. [95]

    Diffsound: Discrete diffusion model for text-to-sound generation

    Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., and Yu, D. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31: 0 1720--1733, 2023 b

    2023

  88. [96]

    A survey on multimodal large language models

    Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023

    2023 arXiv

  89. [97]

    Megabyte: Predicting million-byte sequences with multiscale transformers

    Yu, L., Simig, D., Flaherty, C., Aghajanyan, A., Zettlemoyer, L., and Lewis, M. Megabyte: Predicting million-byte sequences with multiscale transformers. Advances in Neural Information Processing Systems, 36: 0 78808--78823, 2023

    2023

  90. [98]

    and Robnik- S ikonja, M

    Z agar, A. and Robnik- S ikonja, M. S lovene S uper GLUE benchmark: Translation and evaluation. In Calzolari, N., B \'e chet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Odijk, J., and Piperidis, S. (eds.),...

    2022

  91. [99]

    Recurrent neural network regularization

    Zaremba, W., Sutskever, I., and Vinyals, O. Recurrent neural network regularization. ArXiv, abs/1409.2329, 2014. URL https://api.semanticscholar.org/CorpusID:17719760

    2014 arXiv

  92. [100]

    Soundstream: An end-to-end neural audio codec

    Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 0 495--507, 2021

    2021

  93. [101]

    J., Jia, Y., Chen, Z., and Wu, Y

    Zen, H., Dang, V., Clark, R., Zhang, Y., Weiss, R. J., Jia, Y., Chen, Z., and Wu, Y. Libritts: A corpus derived from librispeech for text-to-speech. In Interspeech 2019, pp.\ 1526--1530, 2019. doi:10.21437/Interspeech.2019-2441

    2019 doi

  94. [102]

    Intrinsicvoice: Empowering llms with intrinsic real-time voice interaction abilities

    Zhang, X., Lyu, X., Du, Z., Chen, Q., Zhang, D., Hu, H., Tan, C., Zhao, T., Wang, Y., Zhang, B., et al. Intrinsicvoice: Empowering llms with intrinsic real-time voice interaction abilities. arXiv preprint arXiv:2410.08035, 2024

    arXiv

  95. [103]

    Transfusion: Predict the next token and diffuse images with one multi-modal model

    Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024

    2024 arXiv

  96. [104]

    L., Remez, T., Kreuk, F., Copet, J., D \' e fossez, A., Synnaeve, G., and Adi, Y

    Ziv, A., Gat, I., Lan, G. L., Remez, T., Kreuk, F., Copet, J., D \' e fossez, A., Synnaeve, G., and Adi, Y. Masked audio generation using a single non-autoregressive transformer. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, M...

    2024

  97. [105]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.