{"id":"ce913420-59f0-4ccb-8e27-8255442e4652","arxiv_id":"2507.09834","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A causal audio language model with continuous-valued tokens and masked next-token prediction matches diffusion-based text-to-audio quality with smaller, streamable models.","lead":"This paper replaces the discrete audio tokens used in audio language models with continuous-valued tokens, and trains a causal Transformer decoder to generate them with a diffusion loss. The result is a text-to-audio model that matches larger diffusion-based systems on AudioCaps while using far fewer parameters and remaining streamable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's cross-model FAD/FD ranking is the load-bearing link, and it mixes numbers from different evaluation pipelines: AudioLDM2's FAD shifts 1.78→2.17 under the same toolkit, yet most baselines are not re-inferred.","rationale":"Central claim: causal LM with continuous tokens and MNTP achieves SOTA-level TTA, with Table 1 as its primary quantitative support. For that claim to hold, the FAD/FD rankings across models must reflect model quality rather than evaluation-protocol differences. This assumption is least secure because the table is a union of numbers from different papers and toolkits. The authors themselves demonstrate protocol sensitivity by re-inferring AudioLDM2 with the AudioLDM toolkit and observing 14–22% FAD shifts; they do not perform the same check for the discrete LMs and diffusion baselines they claim to beat. This is a comparability issue, not an internal inconsistency: their ablations (Table 3), human study (Table 2), and re-inference of AudioLDM2 are evidence in favor of the method. But the strongest claim ('best FAD/FD') is exactly the claim most exposed to this concern. The reader's weakest-assumption field emphasizes the fixed VAE/vocoder tokenizer; I view that as less decisive because the same class of tokenizer is shared with the strongest diffusion baselines, so it does not invalidate the relative comparison. The concrete re-inference test would settle whether the headline ranking survives a controlled protocol.","tokens_in":26117,"tokens_out":11796,"duration_ms":149323,"concrete_test":"Re-inference check: use the public checkpoints of AudioGen Large (if released; otherwise the largest available AudioGen or MAGNet-Large), Tango2, Make-An-Audio2, and MAGNet-Large, generate 10-second audio on the same AudioCaps evaluation captions and random caption seed used for AudioMNTP, and compute FD, FAD, KL, IS, and CLAP with the exact AudioLDM evaluation toolkit used in Table 1. If any re-inferred baseline has FAD ≤ 1.22 or FD ≤ 14.30, the ranking is not robust; if all remain above, the cross-model comparability concern is resolved and the headline claim can be taken at face value.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 is the basis for the reader's strongest claim (AudioMNTP Large has FAD 1.22/FD 14.30, best among listed models). The table mixes published numbers from heterogeneous evaluation setups: AudioGen, UniAudio, Tango, Tango2, Make-An-Audio2, and MAGNet are taken from their original papers, while AudioMNTP is computed with the AudioLDM evaluation toolkit. The paper's own re-inference of AudioLDM2 shows the protocol sensitivity: AudioLDM2-Full FAD moves from 1.78 to 2.17 and AudioLDM2-Full-Large from 1.86 to 2.12, with FD 32.14/33.18 where no FD was originally reported. A 14–22% FAD shift from evaluation protocol alone is the same order as several headline gaps (e.g., AudioMNTP Large 1.22 vs Tango2 2.69, or AudioNTP Base 2.28 vs AudioGen Base 2.84). Unless all baselines are re-run through the same toolkit, the 'best FAD/FD' and 'SOTA-level' claims rest on an untested comparability assumption. The internal AudioNTP/MNTP ablations are not affected by this concern, but the headline comparison is.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes generative audio language modeling with continuous-valued tokens and a novel masked next-token prediction (MNTP) task. The authors replace discrete audio tokens with continuous latent vectors obtained from the AudioLDM VAE/Hifi-GAN pipeline, model each next-token distribution with a small diffusion head, and add MNTP, in which randomly dropped tokens are skipped and a target positional embedding tells the model which future position to predict. The method is evaluated on AudioCaps and WavCaps with objective metrics (FAD, FD, KL, IS, CLAP) and human evaluations. The paper claims that AudioNTP outperforms AudioGen, that MNTP further improves next-token prediction, and that AudioMNTP Large reaches the best FAD/FD among the models listed in Table 1 while being smaller and streamable.","tokens_in":26410,"tokens_out":5140,"duration_ms":57764,"significance":"The central technical idea is credible and the internal ablations are thorough. If the cross-model numbers can be established under a common evaluation protocol, the paper would be a significant demonstration that causal language models with continuous tokens can rival diffusion models in text-to-audio generation while offering streaming and efficiency advantages. Strengths include the controlled comparison of NTP versus MNTP in Tables 3 and 6, the diffusion-head-size ablation, the masking-schedule ablations, the human evaluation, and the real-time-factor analysis. The main weakness is that the headline Table 1 ranking rests on protocol-mixed baseline numbers, and the paper's own re-inference of AudioLDM2 shows that the evaluation protocol alone shifts FAD by 14-22%.","major_comments":[{"comment":"The claim that AudioMNTP Large achieves the best FAD and FD across all listed models is not supported by comparable measurements. Table 1 combines published numbers from heterogeneous evaluation pipelines (AudioGen, UniAudio, Tango, Tango2, Make-An-Audio2, MAGNet) with AudioLDM-toolkit numbers for the authors' models, and only AudioLDM2 is re-inferred with the same toolkit. The authors' own re-inference moves AudioLDM2-Full FAD from 1.78 to 2.17 and AudioLDM2-Full-Large from 1.86 to 2.12, a 14-22% shift that is the same order as several headline gaps, such as AudioMNTP Large FAD 1.22 versus AudioGen Large FAD 1.82. Without re-running all baselines through the same evaluation pipeline, the 'best FAD/FD' and 'SOTA-level' conclusions in the abstract and Section 6 do not follow. The internal AudioNTP-versus-AudioMNTP comparisons and ablations are not affected, but the headline claims should be restricted to controlled comparisons or supported by re-inference of the open baselines.","section":"Table 1 / Section 5.1"},{"comment":"No confidence intervals or significance tests are reported for any objective metric. FAD, FD, KL, IS, and CLAP are computed on a single generated set, and the paper repeatedly uses language such as 'significant improvements' (Abstract and Section 5.1) and interprets small differences such as Table 3 (G) versus (J) and Table 6 (E) versus (F) as meaningful. Given that the re-inference of AudioLDM2 already demonstrates substantial evaluation sensitivity, the lack of uncertainty quantification is a load-bearing gap for the quantitative claims, and at least a seed-based or bootstrap-based variance estimate is needed.","section":"Tables 1, 3-6, 8 / Section 5"},{"comment":"The paper relies entirely on a fixed, pre-trained AudioLDM VAE and Hifi-GAN vocoder as the tokenizer and de-tokenizer, as stated in Section 4 and Appendix F.1. This is an acknowledged dependency, but it interacts with the claim that continuous-valued tokens are superior to discrete tokens: because the discrete baselines (AudioGen, UniAudio, MAGNet) use different tokenizers and reconstruction pipelines, part of the observed FAD/KL difference may reflect the evaluation and reconstruction pipeline rather than the language-modeling formulation. The paper should state this limitation explicitly in Section 5.1 and temper the 'continuous-valued tokens are superior' claim accordingly.","section":"Section 4 / Appendix F.1"}],"minor_comments":[{"comment":"The abstract reports '20% and 40% relative gains on AudioCaps in Frechet Audio Distance (FAD) and Kullback-Leibler (KL)' while Section 5.1 says '20%, 40% relative improvements over AudioGen Base on FD and KL scores.' Since AudioGen has no FD value in Table 1 and the FAD improvement is 20% relative, the abstract's wording is consistent with the table but the Section 5.1 wording is inconsistent and should be corrected to FAD.","section":"Abstract vs. Section 5.1"},{"comment":"Several rows in Table 1 (AudioLDM2-AC, AudioLDM2-Full, AudioLDM2-Full-Large) have missing FD, IS, and CLAP values, and the re-inference rows report FD values around 32-33 that are far outside the range of other FD values in the table. The table should clearly annotate which numbers come from published papers, which are re-inferred, and why the re-inferred FD values are so different.","section":"Table 1"},{"comment":"The masking schedule is a mixture of a normal and a truncated normal distribution with equal weights, but no sensitivity analysis for the mixture weights is reported; since the schedule is a free parameter, a brief ablation or a statement that equal weights were chosen without tuning would be helpful.","section":"Appendix B / Section 4"},{"comment":"The human evaluation is based on 20 samples with at least 10 raters per sample, but no inter-annotator agreement or significance testing is reported, and the speech/non-speech split is based on a small set of prompt keywords. The claims in Section 5.2 would be strengthened by reporting confidence intervals or a paired comparison.","section":"Section 5.2 / Appendix G.2"},{"comment":"Equation (2) writes the loss as an expectation over epsilon and t but does not explicitly show the expectation over data (a, w) and the token sequence; adding the data expectation would make the training objective fully precise.","section":"Equations (2)-(3)"},{"comment":"Figure 1 contains both 'Discrete-valued audio tokens' and 'Continuous-valued audio tokens' labels, and the caption says the tokens are continuous; the figure should be relabeled to avoid confusing the reader about which representation is used.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The core method and internal ablations are strong and, in my view, publishable after the evaluation-comparability issue is addressed. I would ask the authors to either re-run the main open baselines through the AudioLDM toolkit or clearly downgrade the headline 'best FAD/FD' and 'SOTA-level' claims to claims about controlled comparisons. I would also encourage the authors to state, in the final version, whether code and checkpoints will be released, since the evaluation protocol sensitivity makes reproducibility important."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your attention. The paper shows that a causal Transformer LM over continuous VAE latents, trained with token-wise diffusion loss, beats discrete-token AudioGen on AudioCaps, and adding masked next-token prediction (MNTP) brings it to roughly the level of bidirectional diffusion models like AudioLDM2 and Tango2, with far fewer parameters. The core idea is borrowed from MAR, but the application to causal audio LM is new and the MNTP variant with target positional embeddings is a clean, well-motivated modification.\n\nWhat's good: the ablations are unusually thorough. Table 3 isolates the effect of each design choice—zero vs Gaussian vs drop masking, skip-token prediction, target positional embeddings, masking schedule, initialization. The comparison between AudioNTP and AudioMNTP is internally controlled, and the conclusion that MNTP helps causal decoding survives the evaluation caveats. The human evaluation, though small, supports the objective numbers. The paper also honestly reports limitations: it does not train a discrete-token baseline due to compute, and it notes that masking schedules were explored and the best one reported.\n\nThe soft spot is the headline comparison. Table 1 mixes numbers from different evaluation pipelines: AudioGen, Tango, etc. use their own published FAD/KL values, while AudioMNTP uses the AudioLDM toolkit. The paper's own re-inference of AudioLDM2 shows FAD shifting from 1.78 to 2.17 and 1.86 to 2.12 under that toolkit. That is a 14–22% swing, the same order as some of the headline gaps. So the claim of \"best FAD/FD\" across all listed models is not established unless the baselines are re-run with the same tokenizer and evaluation code. The core conclusion—continuous tokens beat discrete, MNTP beats NTP—does not depend on that cross-pipeline ranking, but the SOTA claim does.\n\nThere are also no confidence intervals or significance tests on the FAD/KL numbers, and CFG and temperature were tuned on the same test set (Appendix C). That is minor given the size of the internal improvements, but it means the exact margins should not be taken at face value.\n\nBottom line: this is a serious paper with a real contribution. It deserved peer review, and it got it. For our purposes, cite it for the continuous-token LM direction; bring it to reading group if you want a concrete example of how evaluation pipeline sensitivity can muddy otherwise solid empirical work.","headline":"Solid empirical paper: continuous-token causal LMs with MNTP genuinely close the gap to diffusion SOTA on AudioCaps, but the headline cross-model ranking rests on a mixed evaluation pipeline that the paper itself shows to be protocol-sensitive.","tokens_in":26962,"tokens_out":2810,"would_cite":true,"duration_ms":31050,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Causal audio language models with continuous tokens and masked next-token prediction match or beat diffusion-based text-to-audio systems on AudioCaps.","keywords":["text-to-audio generation","audio language modeling","continuous-valued tokens","masked next-token prediction","token-wise diffusion loss","causal Transformer decoder","streamable audio generation","AudioCaps benchmark"],"falsifier":"Re-train AudioMNTP and AudioNTP with a different frozen tokenizer (for example, a VAE trained on the same Mel-spectrograms but with different capacity, or a waveform autoencoder), keeping model size, data, and the AudioLDM evaluation pipeline fixed: if the MNTP gains over NTP evaporate or the continuous-token advantage over a discrete-token baseline flips, the central claims are tied to the specific token space rather than to the learning task. A cheaper check is to recompute FAD from scratch with an independent implementation for AudioMNTP Large, AudioLDM2, and Tango2 on identical generated audio files; if the ranking or the reported 1.22 versus 2.12 gap does not survive, the headline comparisons are toolkit-dependent.","tokens_in":25939,"feed_emoji":"🔊","tokens_out":12734,"duration_ms":128199,"temperature":0.7,"pith_summary":"This paper tries to prove that a causal language model can generate high-fidelity audio without quantizing sound into discrete codes, and that it can do so well enough to match systems built on bidirectional diffusion modeling. The authors treat the low-dimensional latent vectors produced by a fixed audio autoencoder as 'continuous-valued tokens,' and they train a standard Transformer decoder to predict the next token with a token-wise diffusion head instead of a cross-entropy classifier. On top of that, they add masked next-token prediction (MNTP): during training, a random fraction of past tokens is dropped and the model learns to predict a random future token, flagged by a target positional embedding; ordinary next-token prediction is the special case with no dropping. Their 462M-parameter AudioMNTP model reports the best FD and FAD among all compared systems on the AudioCaps benchmark, beating much larger discrete-token language models and matching diffusion models like AudioLDM2 and Tango2, while remaining streamable. If the claims hold, LLM-style scaling and serving infrastructure would apply to direct audio generation.","feed_headline":"462M-parameter audio model tops sound-quality scores on AudioCaps","feed_subtitle":"It matches diffusion-based systems while staying streamable, using continuous tokens and masked next-token prediction.","key_machinery":"The argument runs on three coupled pieces. First, continuous-valued tokens: a frozen variational autoencoder converts a 64-band Mel-spectrogram into a 2-D latent map, which is patched and serialized into 256 tokens of dimension 128 per 10-second clip, so the language model sees a short sequence of real vectors instead of a long sequence of discrete codes. Second, the token-wise diffusion head: a small MLP that takes a noised token, the conditioning vector from the Transformer decoder, and a noise level, and predicts the added Gaussian noise; the denoising objective substitutes for the cross-entropy loss, leaving the backbone a standard causal Transformer. Third, masked next-token prediction (MNTP): each training pass samples a masking ratio from a mixture of a normal and a truncated normal distribution over [0,1], drops that fraction of tokens outright, and trains the model to predict a random future token from the surviving past context, using a target positional embedding to disambiguate which future position is being predicted; at inference the target embedding is simply the next position, so decoding is identical to ordinary next-token prediction.","core_discovery":"The paper's central claim is that the two commitments that made causal audio language models underperform—discrete token dictionaries and cross-entropy next-token loss—are optional, not essential. Reinterpreting the continuous latents used by latent diffusion models as tokens, the paper trains a causal Transformer decoder to model the next-token density with a small MLP diffusion head that denoises a Gaussian-corrupted token conditioned on the decoder's context vector. This alone, AudioNTP, beats the discrete-token AudioGen at a third of its size. The paper then argues that masked prediction, not bidirectionality, is what gives masked generative models their edge, and proposes masked next-token prediction as a causal analogue: random tokens are dropped before each forward pass and the model predicts a future token from the remaining sparse context, with a learned target positional embedding telling it which future position to predict. AudioMNTP Large, at 462M parameters and trained on roughly 1,000 hours of audio, attains FD 14.30 and FAD 1.22 on AudioCaps—the best of either metric among every system the paper lists, including 712M–937M bidirectional diffusion models—and the authors note their model is uni-directional and much smaller than those baselines, which they take as evidence that further scaling would push performance higher.","pith_inferences":["The paper does not train its own discrete-token baseline (footnote 11), so the continuous-token advantage rests on published AudioGen and UniAudio numbers rather than a same-data, same-pipeline ablation in this work.","Because only the AudioLDM tokenizer is tested, an untested extension is whether the MNTP recipe transfers to other autoencoder token spaces (or to image and speech latent sequences); if the gains come mainly from the learning task, the same recipe should transfer.","The target positional embedding makes skip-ahead prediction an explicit capability, which suggests a speculative-decoding scheme the paper mentions only in passing: predict several future tokens in parallel and then verify them left-to-right, trading computation for lower streaming latency."],"forward_implications":["A causal, streamable Transformer decoder can reach objective audio quality on AudioCaps comparable to 712M–937M bidirectional latent diffusion models while using only 462M parameters.","Continuous-valued VAE latents shrink the per-clip sequence from roughly 5,000 discrete tokens to 256 continuous tokens, cutting training cost and making LM-style scaling of audio more practical.","Masked next-token prediction improves the decoding task itself: AudioMNTP Base beats plain next-token prediction by 26% relative FAD at the same size, and the gain carries over when the model is scaled to 462M.","With the prediction head as the only swapped component, the framework is positioned to inherit LLM serving infrastructure (KV caching, paged attention) and to be integrated into multi-modal LMs as a drop-in audio output head."],"supporting_citations":[{"why":"Supplies the token-wise diffusion loss, the MLP diffusion head, and the continuous-token idea for masked generative modeling that the paper adapts into a causal LM.","marker":"(Li et al., 2024b)"},{"why":"The discrete-token causal LM baseline whose FAD and KL the paper improves by 20% and 40% (Base) and whose FAD AudioMNTP Base improves by 41% in relative terms.","marker":"(Kreuk et al., 2023)"},{"why":"Donates the VAE plus Hifi-GAN tokenization pipeline and the evaluation toolkit/protocol used to compute the AudioCaps metrics.","marker":"(Liu et al., 2023a)"},{"why":"The bidirectional latent diffusion model whose reported re-inference FAD (2.12) the 462M AudioMNTP Large beats (1.22), establishing the SOTA comparison.","marker":"(Liu et al., 2024c)"},{"why":"The bidirectional diffusion model compared on FD/FAD and in the human study; its high CLAP score stems from an extra preference-optimization dataset the paper lacks.","marker":"(Majumder et al., 2024)"},{"why":"Provides the reverse-diffusion update and noise schedule used by the token-wise diffusion head during inference.","marker":"(Ho et al., 2020)"}],"fun_headline_variants":["Continuous tokens and masked prediction give audio LM SOTA FAD","AudioMNTP: 462M model beats diffusion baselines on FAD","Causal audio LM with continuous tokens matches diffusion SOTA","Small audio LM sets AudioCaps record via masked next-token prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that the frozen audio encoder and vocoder borrowed from the AudioLDM pipeline preserve essentially all information a listener cares about and introduce no systematic bias, because the language model never sees the waveform directly and the headline metrics are computed with that same pipeline's evaluation toolkit; if those components are lossy or biased, they cap generation quality and the reported FAD and FD gains could be partly artifacts of the evaluation setup.","fun_headline_variants_meta":{"raw":{"variants":["Continuous tokens and masked prediction give audio LM SOTA FAD","AudioMNTP: 462M model beats diffusion baselines on FAD","Causal audio LM with continuous tokens matches diffusion SOTA","Small audio LM sets AudioCaps record via masked next-token prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000793,"raw_usage":{"total_tokens":3538,"prompt_tokens":1032,"completion_tokens":2506,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":2431}},"tokens_in":648,"tokens_out":2506,"duration_ms":20866,"temperature":1.0,"reasoning_tokens":2431,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:44:49.620388+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-train AudioMNTP and AudioNTP with a different frozen tokenizer (for example, a VAE trained on the same Mel-spectrograms but with different capacity, or a waveform autoencoder), keeping model size, data, and the AudioLDM evaluation pipeline fixed: if the MNTP gains over NTP evaporate or the continuous-token advantage over a discrete-token baseline flips, the central claims are tied to the specific token space rather than to the learning task. A cheaper check is to recompute FAD from scratch with an independent implementation for AudioMNTP Large, AudioLDM2, and Tango2 on identical generated audio files; if the ranking or the reported 1.22 versus 2.12 gap does not survive, the headline comparisons are toolkit-dependent.","supporting_citations":[{"cited_title":"Audiogen: Textually guided audio generation","cited_arxiv_id":null,"evidence_quote":"The discrete-token causal LM baseline whose FAD and KL the paper improves by 20% and 40% (Base) and whose FAD AudioMNTP Base improves by 41% in relative terms."}],"review_version":1}