{"id":"87b324f2-6444-41d8-b860-44d9280646ad","arxiv_id":"2608.11804","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MiDashengLM-Gen uses an LLM backbone with per-token flow matching to generate variable-length multilingual audio scenes with near-TTS speech intelligibility and competitive mixed-scene quality.","lead":"A Xiaomi team combined a large language model with per-token flow matching to generate mixed audio scenes that contain speech, music, and sound effects in one variable-length pass. On an English speech benchmark, their model cut word error rate from 12.15% to 2.79%, approaching dedicated text-to-speech systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DashengTokenizer reconstruction WER is never reported; without it, the headline 2.79% Seed-TTS WER is an uninterpretable lower bound on the codec, not evidence of the LLM/flow-matching design.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing point: all intelligibility numbers pass through the frozen DashengTokenizer decoder, and the paper never reports the tokenizer's own reconstruction WER. Figure 3's topline is only an STFT L2 distance, which is a poor proxy for phoneme-level intelligibility; a 1.30 STFT distance could correspond to a wide range of WERs. Since the model's generated latents are decoded by the same frozen decoder, the generated WER cannot be lower than the oracle reconstruction WER, so the headline comparison to dedicated TTS is only meaningful when the oracle is known. I found no more load-bearing issue: the architecture is described with sufficient detail, the alignment ablation supports the alignment stage, and the raw WER number is externally benchmarked. A secondary inconsistency exists in Table 10—width 1280 with 1280-dim latent is marked convergent despite the text claiming width must 'strictly exceed' the latent dimension—but this affects a design insight, not the central intelligibility claim. The appropriate verdict remains conditional, pending the oracle-WER check; no change to the reader's verdict is needed.","tokens_in":13555,"tokens_out":5051,"duration_ms":51944,"concrete_test":"Run the frozen DashengTokenizer encoder+decoder (same checkpoints used in Section 3) on the Seed-TTS English test set, and transcribe the reconstructed audio with Whisper-large-v3 (the ASR used in Table 4). Compute the reconstruction WER. For completeness, also compute WER after applying the k=5 group-MLP-unproject path, to isolate any additional loss from the 5-Hz projection. Compare the oracle WER with the reported 2.79%. If oracle WER >= 2.79%, the headline 'approaching TTS' claim is dominated by codec loss; if oracle WER < 1%, the generation path is the main bottleneck.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 freezes the DashengTokenizer encoder and groups k=5 25-Hz frames into 200-ms MLP-projected tokens; Section 3.2 decodes generated latents with the same frozen DashengTokenizer decoder. The abstract's central claim (Seed-TTS EN WER 2.79% vs. 1.24% for Qwen3-TTS) is therefore bounded below by the tokenizer's own encode-decode WER. The paper never reports this oracle: Figure 3 gives only STFT L2 reconstruction distance (topline 1.30), and Table 4 gives only generated-audio WER. If the frozen codec's reconstruction WER on Seed-TTS is already ~2.7%, the model is essentially operating at the codec ceiling and the '2.2x gap' to TTS is attributable to the tokenizer, not to the autoregressive flow-matching architecture. If the oracle is much lower (e.g., 0.5%), then a 2.3% generation gap remains. Either way, the load-bearing comparison in the abstract is not interpretable without the oracle. The 200-ms grouping further means each LLM hidden state must carry enough phonemic information to synthesize a 200-ms chunk, so a per-chunk intelligibility ceiling is a real risk that cannot be dismissed by STFT distance alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces MiDashengLM-Gen, a unified audio scene generation model that couples a pretrained Qwen3 LLM with per-token conditional flow matching over continuous latents produced by a frozen DashengTokenizer encoder. It uses structured multi-view captions, an audio-text alignment pre-training stage, and a learned stop head to support variable-length, multilingual generation of mixed scenes containing speech, music, and sound effects. Experiments report improved speech intelligibility on the Seed-TTS benchmark (English WER 2.79% vs. 12.15% for Dasheng AudioGen), competitive multilingual WERs on the MiniMax-Speech test set, emotion expressiveness results on CV3-Eval, and mixed-scene generation results on MECAT. Ablations study the importance of audio-text alignment and the scaling of DiT width relative to audio latent dimensionality.","tokens_in":13832,"tokens_out":7017,"duration_ms":69253,"significance":"If the results hold, this is a meaningful advance for unified audio scene generation: it demonstrates that an LLM combined with per-token flow matching over a continuous semantic-acoustic latent can bring speech intelligibility much closer to dedicated TTS systems than prior unified models, while retaining mixed-scene and multilingual capability. The main WER claims are evaluated on external benchmarks (Seed-TTS, MiniMax-Speech, CV3-Eval) rather than only on self-created data, and the authors release code and checkpoints, which are concrete strengths. The use of a self-created tokenizer and MECAT benchmark introduces a mild self-evaluation risk, and the absence of uncertainty quantification tempers the comparative claims.","major_comments":[{"comment":"The frozen DashengTokenizer encoder and decoder used to produce and reconstruct audio latents define an upper bound on achievable intelligibility, yet the paper never reports the tokenizer's own reconstruction WER on the Seed-TTS benchmark. Figure 3 reports only STFT L2 distance (topline 1.30) and Table 4 reports only generated-audio WER, so the headline comparison of 2.79% to Qwen3-TTS's 1.24% cannot be attributed to the proposed LLM/flow-matching architecture without knowing the codec oracle. If encode-decode through the frozen tokenizer already yields WER near 2.7%, the model is effectively operating at the codec ceiling and the claimed 2.2x gap is dominated by tokenizer loss; if the oracle is much lower, a real generation gap remains. Please report the encode-decode WER/CER of the frozen tokenizer on the same Seed-TTS and multilingual evaluation sets, and discuss how the oracle affects the abstract's claim.","section":"Section 3.1-3.2 and Table 4"},{"comment":"The paper claims that 'the DiT width must strictly exceed the audio latent dimensionality' and states this generalizes across LLM scales and latent dimensions, but Table 10 contradicts this: for 1280-dimensional latents with both Qwen3-0.6B and Qwen3-4B, width equal to 1280 converges successfully. Figure 3 only demonstrates failure at width 768 for 768-dimensional latents. Please revise the claim to a threshold phenomenon, explain why equality fails at 768 but succeeds at 1280, and define the criteria used to mark convergence in Table 10.","section":"Section 4.5, Table 10 and Figure 3"},{"comment":"All reported metrics are point estimates without confidence intervals, error bars, or significance tests, while several comparative conclusions rest on small differences (e.g., Table 7 S00 CLAP 0.35 vs. 0.34; Table 8 S0A FAD 1.54 vs. 1.75). Given the emphasis on 'consistently lower FAD' and 'better' claims in Section 4.4, please provide variance estimates or significance tests, such as bootstrap intervals over evaluation items or multiple sampling runs, and temper claims for differences that are not statistically distinguished.","section":"Section 4.2-4.4, Tables 4-8"}],"minor_comments":[{"comment":"The hyperlinks appear as bare footnote markers (1, 2, 3); please render them as actual URLs in the final version so that the code, checkpoint, and demo links are directly accessible.","section":"Abstract"},{"comment":"The parameter counts and depth configurations for the 'fixed parameter budget' are not reported; please include them so the convergence comparisons across width/depth trade-offs are reproducible.","section":"Section 4.5, Table 10"},{"comment":"A CER value of '>100' is not a valid error rate; please provide the actual values or explain the transcription pathology that produces this result.","section":"Table 4"},{"comment":"The sentence 'Instead of full-sequence flow matching that generates the entire audio latent sequence in one pass we perform' is missing punctuation; additionally, define z_hist explicitly in a displayed equation or in the text adjacent to Eq. (3).","section":"Section 3.2"},{"comment":"The claim that MiDashengLM-Gen 'represents a first approach for general text-to-audio generation with one end-to-end trained model' is difficult to verify against related work cited in Section 2 (UniAudio, AudioX, UniFlow-Audio, Dasheng AudioGen); please provide a precise definition of the claimed first or soften the phrasing.","section":"Section 1 and throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's evaluation infrastructure is partly self-created (DashengTokenizer, MECAT, and the MiDashengLM-0.6B encoder all come from the same group), while the abstract makes strong 'first' and 'approaching TTS' claims. I recommend that the editor require an explicit statement of data and benchmark provenance, and that the oracle-WER analysis be completed before acceptance; the central WER improvement over Dasheng AudioGen is plausible, but the comparison to dedicated TTS is not interpretable without the codec ceiling."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a serious engineering paper with one genuinely new combination — LLM autoregression over continuous DashengTokenizer latents with per-token flow matching, plus a stop head for variable length. The alignment-stage ablation is the cleanest evidence for a real design choice: removing it degrades Seed-TTS EN WER from 2.79% to 12.17%, which is a fourfold jump and not something you'd get from a trivial baseline. That ablation alone justifies a careful read. The multilingual WER table also shows a real capability: 2.42% EN, 3.51% ZH, and mostly single-digit WERs across nine languages, versus triple-digit failures from Dasheng AudioGen. Credit where it's due: the system appears to work, and the public benchmarks used for the main speech claims are external.\n\nNow the soft spots, in proportion. The biggest issue is one the stress-test note nails: the frozen DashengTokenizer's encode-decode reconstruction WER is never reported. Figure 3 gives only STFT L2 (topline 1.30), which does not tell you how much phonetic detail the codec preserves after grouping five 25-Hz frames into 200-ms tokens. Without that oracle, the abstract's headline \"2.79% vs 1.24%\" is an uninterpretable lower bound. If the tokenizer already loses phonemic detail at 2.7%, the model is at ceiling and the 2.2x gap to TTS is a codec property, not a generation property. If the oracle is 0.5%, then there's a real 2.3% generation gap. Either way the paper should report it; this is a fixable omission, not a fatal flaw.\n\nSecond, all tables are point estimates without error bars or significance testing. For a benchmark like Seed-TTS that's a minor but real weakness, typical for the field but worth noting.\n\nThird, the claim that \"DiT width must strictly exceed latent dimensionality\" is contradicted by their own Table 10: width=1280 with 1280-dim latents converges. The text says \"below or equals\" and marks 1280 as a checkmark, so the strict inequality claim is overstated. The pattern is interesting and probably worth a softer claim: width >= latent dim works, width just below fails.\n\nFourth, the \"first end-to-end model\" framing is broader than the evidence; Dasheng AudioGen already did end-to-end scene generation, just with a frozen text encoder. The novelty is the specific LLM+flow-matching combination, which is enough.\n\nMECAT being from the same group is real but mild given that the main intelligibility claims rest on external benchmarks.\n\nWho should read this: people working on unified audio generation, TTS/audio codec interfaces, and anyone thinking about latent-space ceilings in multi-task audio models. It deserves a proper referee — send it out. The code and checkpoints are promised, and the oracle-WER issue can be resolved in revision.\n\nRecommendation: conditional accept with a request for the tokenizer reconstruction WER, error bars on the main tables, and a corrected width-rule statement.\n\nBest.","headline":"A credible end-to-end unified audio-scene model with a strong intelligibility result, but the headline WER needs the tokenizer's own reconstruction WER reported before the 2.79% number is interpretable.","tokens_in":14376,"tokens_out":787,"would_cite":true,"duration_ms":9762,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single end-to-end model can generate mixed audio scenes—speech, music, and sound effects together—with speech intelligibility close to that of dedicated text-to-speech systems, while retaining variable-length and…","keywords":["audio scene generation","text-to-audio","flow matching","large language model","autoregressive generation","speech intelligibility","multilingual speech","multi-view captions"],"falsifier":"Measure the tokenizer's own encode-decode reconstruction WER on the Seed-TTS English test set, encoding speech and decoding with DashengTokenizer plus the vocoder without any LLM generation. If this reconstruction WER is at or above the model's 2.79%, then the LLM contributes none of the claimed intelligibility and the result is bounded by the frozen encoder.","tokens_in":13363,"feed_emoji":"🎧","tokens_out":4333,"duration_ms":37608,"temperature":0.7,"pith_summary":"The paper proposes an end-to-end framework, MiDashengLM-Gen, that couples a pretrained large language model with per-token conditional flow matching to generate coherent audio scenes containing speech, music, and sound effects. It argues that previous unified models fail because a frozen text encoder is decoupled from the audio decoder, and replaces that design with an LLM that jointly encodes text and audio history, feeding each autoregressive step to a flow-matching decoder. On the Seed-TTS English benchmark, the model reduces word error rate from 12.15% to 2.79%, compared with 1.24% for a dedicated TTS system. The paper also reports competitive music and effects quality on MECAT and multilingual speech intelligibility across nine languages. The central claim is that high-dimensional semantic-acoustic latents, an LLM backbone, and per-token flow matching are sufficient for end-to-end unified audio scene generation.","feed_headline":"LLM-driven audio model cuts scene speech errors from 12.15% to 2.79%","feed_subtitle":"Unified speech, music, and effects generation now approaches dedicated TTS intelligibility.","key_machinery":"The load-bearing mechanism is per-token conditional flow matching over DashengTokenizer's high-dimensional semantic-acoustic latents, with the LLM hidden state as the conditioning signal. Structured multi-view captions decompose a scene into six textual views—global description, transcript, speaker, effects, music, and environment—each marked by special tokens, letting the LLM condition different control factors separately. An alignment stage first projects 25 Hz encoder frames, grouped to $k=5$ frames into 5 Hz tokens, into the LLM token space; then the LLM autoregressively predicts each token's flow field, and a DiT decoder with width greater than the latent dimension solves a 10-step Euler ODE, with a stop head determining generation length.","core_discovery":"MiDashengLM-Gen is presented as the first general text-to-audio scene generator trained end to end in a single model: a Qwen3 LLM encodes the structured caption and audio history, and a DiT-based flow-matching decoder produces each 200 ms audio token conditioned on the LLM hidden state. Speech intelligibility measured by WER on Seed-TTS English falls to 2.79%, down from 12.15% for the previous unified model and within a factor of 2.2 of dedicated TTS at 1.24%. The model supports variable-length output through a learned stop head and multilingual generation. The paper identifies two design requirements: an audio-text alignment stage that maps audio latents into the LLM token space before generation training, and a DiT width that strictly exceeds the audio latent dimension for convergence.","pith_inferences":["If the tokenizer's own reconstruction WER is not substantially below 2.79%, part of the claimed intelligibility may be inherited from the frozen encoder rather than from the LLM-driven generative process; the paper reports STFT distance but not this ceiling.","The 5 Hz grouping into 200 ms tokens likely limits temporal precision for fast consonants and could explain residual gaps in low-resource languages; a testable extension is variable-rate grouping or $k=1$ tokens.","The width condition suggests a general principle for any high-dimensional continuous latent: a diffusion decoder's hidden width must exceed latent rank to avoid manifold expansion failure, which may transfer to video and multimodal generation.","Because the paper reports only objective metrics, a subjective listening test on scene coherence would be a natural next check; the demo page hints at this but the paper does not include it."],"forward_implications":["If the claims hold, end-to-end LLM-based generation closes most of the intelligibility gap with dedicated TTS, making scene generators usable for dialogue in games and film preproduction.","Variable-length generation means one model can produce a 1-to-20-second scene without post-hoc stitching or fixed-duration constraints.","Multilingual speech with competitive WERs may reduce the need for separate TTS adapters in mixed-audio pipelines.","The width-must-exceed-latent-dimension rule gives a concrete scaling guideline for future LLM-conditioned audio decoders.","The alignment-stage ablation suggests that cross-modal token mapping must be learned before generation training, not from scratch."],"supporting_citations":[{"why":"Dasheng AudioGen, the unified-scene baseline whose 12.15% Seed-TTS English WER is the number MiDashengLM-Gen must beat; also the source of structured multi-view captions.","marker":"[9]"},{"why":"DashengTokenizer, the frozen semantic-acoustic encoder/decoder that supplies the 25 Hz latent space and defines the generation target.","marker":"[25]"},{"why":"MiDashengLM, whose audio-text alignment procedure maps audio latents into the LLM token space and is ablated as strictly required.","marker":"[27]"},{"why":"Qwen3, the pretrained LLM backbone that jointly encodes text and audio history and replaces the frozen text encoder.","marker":"[26]"},{"why":"Seed-TTS benchmark and its reported WER, the speech intelligibility yardstick against which the 2.79% result is compared.","marker":"[2]"},{"why":"Source of the width-must-exceed-latent-dimension convergence principle that the paper verifies for its DiT decoder.","marker":"[44]"},{"why":"MiniMax-Speech's multilingual test set, used to measure nine-language WER/CER and to compare against that system's baseline.","marker":"[20]"}],"fun_headline_variants":["First end-to-end audio scene model nears TTS speech clarity","LLM flow matching unifies speech, music, SFX in one model","Audio scene WER slashed from 12.15% to 2.79% by LLM","One model, all audio: LLM-driven scene generation","End-to-end audio scenes: LLM + flow matching beats past"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The frozen DashengTokenizer encoder is assumed to preserve enough phonetic and acoustic detail, after grouping five 25 Hz frames into one 200 ms token, that near-TTS intelligibility is possible; if this latent space loses phonemic information, no amount of LLM conditioning can recover it.","fun_headline_variants_meta":{"raw":{"variants":["First end-to-end audio scene model nears TTS speech clarity","LLM flow matching unifies speech, music, SFX in one model","Audio scene WER slashed from 12.15% to 2.79% by LLM","One model, all audio: LLM-driven scene generation","End-to-end audio scenes: LLM + flow matching beats past"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000773,"raw_usage":{"total_tokens":3447,"prompt_tokens":994,"completion_tokens":2453,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":2354}},"tokens_in":610,"tokens_out":2453,"duration_ms":19463,"temperature":1.0,"reasoning_tokens":2354,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:25:57.572435+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the tokenizer's own encode-decode reconstruction WER on the Seed-TTS English test set, encoding speech and decoding with DashengTokenizer plus the vocoder without any LLM generation. If this reconstruction WER is at or above the model's 2.79%, then the LLM contributes none of the claimed intelligibility and the result is bounded by the frozen encoder.","supporting_citations":[],"review_version":1}