{"id":"14b14c84-a0b4-4560-a9b1-a4475c31c47c","arxiv_id":"2501.14680","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Combining global and local text embeddings in a diffusion UNet for music improves text adherence, while mean-pooling T5 local embeddings yields the best audio quality without extra parameters.","lead":"This paper tests how different text encoders condition a diffusion model that turns text prompts into short music clips. It finds that blending a global sentence embedding with per-word local embeddings improves text adherence, and a simple averaging scheme gives similar quality with fewer model parameters.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline CLAP+T5 text-adherence gain is confounded: the T5-only baseline is a different architecture with 124M fewer parameters; a matched-capacity, information-free global conditioning run is needed to attribute KL=1.47 to CLAP semantics.","rationale":"I agree with the reader that the headline quantitative comparisons are not robustly supported, and I narrow the concern to the specific confound: the T5-only baseline is not the proposed architecture with the global FiLM branch removed, and it has 124M fewer parameters. This is more directly testable than the general lack of error bars, though both matter. The mean-pooling result (482M, FAD=1.89, KL=1.51) actually strengthens the paper's parameter-efficiency narrative, because it beats the CLAP+T5 model (606M, FAD=1.94) on FAD with fewer parameters and only a 0.04 KL sacrifice. But the abstract's first claim about CLAP enhancing text adherence is exactly the comparison most confounded by parameters and architecture. The proposed shuffled-CLAP ablation would settle the causal role of the global text embedding. As a secondary observation, the CFG equation in Section II-C is written with the unconditional and conditional terms interchanged relative to standard classifier-free guidance; if the implementation matched the printed formula, the reported KL=1.47 would be hard to explain, so the authors should correct or clarify this point. Overall, the paper's contribution is plausible and the ablation design is mostly reasonable, but the headline claim needs the matched-capacity control. The CONDITIONAL verdict remains appropriate, so I recommend no change.","tokens_in":9574,"tokens_out":10142,"duration_ms":88256,"concrete_test":"Retrain the LDM(ours) CLAP+T5-base configuration with identical hyperparameters, but randomly permute the CLAP global embeddings across training samples (so the FiLM input is distribution-matched yet carries no text-specific information). If FAD/KL remain near the reported 1.94/1.47, the text-adherence gain is explained by added parameters or the conditioning pathway rather than by CLAP semantics; if performance regresses toward the 2.02/1.54 T5-only baseline, the global semantic conditioning is the active ingredient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim that adding CLAP global embeddings to T5 local embeddings enhances text adherence (KL 1.47 vs 1.54) relies on Table II's comparison between 'CLAP T5-base' (606M parameters, FAD=1.94, KL=1.47) and '- T5-base' (482M parameters, FAD=2.02, KL=1.54). However, the '- T5-base' row reproduces the AudioLDM (retrain) numbers from Table I, not a T5-only variant of the proposed LDM architecture shown in Fig. 1. Thus the KL gain may be due to the architectural difference (FiLM global branch, cross-attention placement) or the 124M additional parameters of the CLAP encoder, rather than to the semantic content of the CLAP global text embedding. The mean-pooling row (482M) provides a better controlled anchor: it achieves FAD=1.89, KL=1.51 compared to the same 482M T5-only baseline, supporting a smaller text-adherence gain (0.03) and a FAD gain (0.13) without added parameters. But the specific claim about CLAP's contribution requires a matched-capacity ablation that destroys the text-global association while preserving parameter count. Without that, the quantitative headline is not established; a reviewer cannot tell whether CLAP's global semantics or simply extra capacity/conditioning pathway drives the reported improvement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a diffusion-based text-to-music generation system in which a UNet is conditioned on both local text embeddings from T5 (through cross-attention) and global text embeddings either from CLAP or from pooling the T5 local embeddings (through FiLM). The authors evaluate the proposed architecture against retrained AudioLDM baselines and publicly available systems on MusicCaps, reporting FAD and KL scores. They claim that adding CLAP global embeddings to T5 local embeddings improves text adherence (KL 1.47 vs 1.54) and that a proposed mean-pooling variant is more parameter-efficient while achieving competitive or slightly better FAD. The paper includes ablations over global text conditioners (CLAP, Sentence-T5, SimCSE, mean pooling, self-attention pooling) and over local text encoders (T5-base, FLANT5-large).","tokens_in":9906,"tokens_out":6057,"duration_ms":55039,"significance":"If the central attribution claim is confirmed, the paper offers a useful and practical study: it retrains AudioLDM baselines on the same data, reports parameter counts for each configuration, and includes a focused ablation over global conditioners. The honest reporting of the self-attention pooling failure is a further strength. The proposed mean-pooling variant, if it holds with proper uncertainty quantification, would be a simple parameter-efficient way to obtain global text conditioning. However, the headline claim that CLAP's semantic content is responsible for the text-adherence gain is not yet supported by the experiments, because the comparison baseline differs in both architecture and parameter count. The reported differences are also small and are presented without error bars or repeated runs.","major_comments":[{"comment":"The central claim that adding CLAP global embeddings to T5 local embeddings improves text adherence (KL 1.47 vs 1.54) is not a controlled ablation. The row labelled '- T5-base' in Table II corresponds to 'AudioLDM (retrain) - T5-base' from Table I, which is a different architecture: it lacks the proposed FiLM-based global-conditioning branch and has 482M parameters versus 606M for the CLAP+T5 model. The KL gain could therefore be caused by the additional 124M parameters, the extra conditioning pathway, or the architectural differences between AudioLDM and the proposed LDM, rather than by the semantic content of the CLAP global embedding. The authors should include a matched-capacity T5-only variant of the proposed LDM architecture, ideally with an uninformative global vector of the same dimension injected through the same FiLM path, to attribute the improvement to CLAP semantics.","section":"Section IV, Table II; abstract"},{"comment":"All quantitative claims rest on a single run and a single checkpoint. The checkpoint is selected on the validation set as the one with the best FAD and KL scores, and no confidence intervals, error bars, multiple seeds, or repeated inference draws are reported. The differences interpreted as meaningful are small (e.g., KL 1.47 vs 1.51, FAD 1.89 vs 1.94), and such differences could plausibly arise from run-to-run variance or selection effects. The paper should report at least multiple training runs with standard deviations, or, failing that, bootstrap confidence intervals over evaluation draws, and should disclose whether the same checkpoint is used for all metrics.","section":"Tables I-III and Section III-A"}],"minor_comments":[{"comment":"The classifier-free guidance formula appears to have the conditional and unconditional terms reversed: the equation gives \\hat{v} = \\omega v_\\theta(z_t,t,\\emptyset,\\emptyset) + (1-\\omega) v_\\theta(z_t,t,G_y,F_y), which with \\omega = 9 gives a negative coefficient to the conditional prediction. If this is a typo, the corrected formula should be provided; if implemented as written, it is anti-guidance rather than standard classifier-free guidance.","section":"Section II (Conditioning)"},{"comment":"The pooling equations are underspecified: the summation index i over F_y^{(i)} is not defined in terms of the number of tokens M, and the dimensions of G_y and F_y are not stated. Please clarify the notation for reproducibility.","section":"Section II, Eqs. (2)-(3)"},{"comment":"The v-objective in Eq. (4) uses \\alpha_t and \\sigma_t without defining the noise schedule or their dependence on t. A brief definition or reference to a specific schedule would help.","section":"Section II, Eq. (4)"},{"comment":"The text contains a typo and an incomplete sentence: 'mapping emebeddings to an isotropic distribution' should read 'embeddings', and the sentence beginning 'This may require adding regularisation...' should be completed.","section":"Section IV-A"},{"comment":"The validation procedure on MusicBench is only described as checkpoint selection based on 'best FAD and KL scores'. Please specify the number of generated samples, prompt set, and whether FAD and KL are computed on the same audio set that is later used for testing.","section":"Section III-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a compact empirical study with a clear ablative structure. The main gap is fixable within the scope of a revision: a matched-capacity T5-only version of the proposed LDM and uncertainty estimates would directly address the confounding concern. I would not reject the paper on current evidence, but the headline claim as written is stronger than the data support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a look if you work on text-to-music conditioning. It systematically compares global and local text conditioners in a single AudioLDM-style pipeline, retrains baselines on the same data, and reports parameter counts. The most defensible finding is that mean pooling of T5 gives you a global embedding at zero extra parameters and lands close to CLAP+T5 on FAD/KL. That is a practical recipe many engineers would want.\n\nThe ablations over global conditioners (CLAP, Sentence-T5, SimCSE, mean pooling, self-attention pooling) are the real contribution. The authors honestly report that Sentence-T5 and SimCSE hurt, and that their self-attention pooling collapses to a narrow token set. This is exactly the kind of negative result that saves other people time.\n\nThe soft spot is the headline claim. The stress-test is right: the '- T5-base' row in Table II reproduces the AudioLDM (retrain) numbers from Table I, not a T5-only variant of the proposed LDM architecture. So the 1.54 vs 1.47 KL gain attributed to CLAP conflates the global conditioning pathway with architectural differences and 124M extra parameters. The mean-pooling row is a better anchor, but even there you don't have a no-global variant of the same LDM. Also, there are no error bars or multiple seeds; the reported differences are 0.02–0.05 in KL and 0.08–0.13 in FAD, which may be within run-to-run noise. The checkpoint selection on validation FAD/KL is fine, but it doesn't replace repeated runs. The abstract overreaches with 'enhances text adherence.'\n\nMinor: no code or data is released, and the comparison against public pre-trained models is useful context but not a controlled comparison. These are secondary.\n\nWho this is for: engineers building TTM systems who need a quick answer about whether to add CLAP or just pool T5. It is not a top-conference paper as written, but it is a solid empirical study with honest reporting. I would accept it for peer review and ask for a matched T5-only baseline of the proposed architecture, confidence intervals from multiple seeds, and a softened abstract claim. That revision would make the paper's actual contribution stand rather than the current overreach.","headline":"A useful empirical map of global/local text conditioning for TTM, but the headline CLAP claim is not properly isolated from architecture and parameter count.","tokens_in":10421,"tokens_out":2230,"would_cite":true,"duration_ms":21627,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Global plus local text conditioning improves text adherence in a diffusion text-to-music model, and a mean-pooled T5 alone achieves better audio quality at lower cost.","keywords":["text-to-music generation","latent diffusion","global text embedding","local text embedding","mean pooling","self-attention pooling","CLAP","T5"],"falsifier":"Re-train the T5-local baseline with the same 606M parameter budget as the CLAP+T5 model (for example, using a larger T5) across at least three seeds; if that baseline reaches FAD at or below 1.89 and KL at or below 1.47, the reported advantages of global conditioning and mean pooling would not be established.","tokens_in":9403,"feed_emoji":"🎵","tokens_out":8909,"duration_ms":65890,"temperature":0.7,"pith_summary":"Text-to-music diffusion models rely on text encoders to condition generation, but the type of text embedding matters more than the number of encoders. This paper proposes a latent diffusion UNet conditioned on both a local word-level embedding from T5 (via cross-attention) and a global sentence-level embedding from CLAP (via FiLM), and reports that this pairing improves text adherence over T5 alone: KL=1.47 versus 1.54. The paper then shows that a global embedding extracted from T5's own token embeddings by simple mean pooling yields FAD=1.89 and KL=1.51, better audio quality and nearly the same adherence as the dual-encoder setup (FAD=1.94, KL=1.47), while saving about 124 million parameters. The claim is that fusing global and local text representations helps, and that the global representation can come from the same frozen language model.","feed_headline":"Mean-pooled T5 lifts fidelity; dual text conditions lift adherence","feed_subtitle":"FAD 1.89 for pooled T5 vs 1.94 for CLAP+T5; KL 1.51 vs 1.47, with 124M fewer parameters","key_machinery":"The load-bearing mechanism is the placement of the two conditioning signals. The global text embedding $G_y$ is injected through Feature-wise Linear Modulation (FiLM), which applies a learned scale and shift to the UNet's intermediate features, alongside the time embedding; the local text embedding $F_y$ enters through standard cross-attention. The paper's proposed mean pooling, $G_{\\text{mean}} = \\frac{1}{M}\\sum_{i=1}^{M} F^{(i)}_y$, derives the global embedding from the same T5 token embeddings, and self-attention pooling $G_{\\text{SAP}} = \\mathrm{Softmax}(W_{\\text{SAP}} F_y^T) F_y$ is a trainable alternative. The diffusion model is trained with the v-objective and classifier-free guidance with 10 percent unconditional dropout, which lets a single model produce both conditional and unconditional outputs.","core_discovery":"The central claim is that conditioning a latent diffusion UNet on both local and global text representations improves text-to-music generation, and that the global representation can be obtained without a separate cross-modal encoder. The paper reports that CLAP global plus T5 local conditioning achieves KL=1.47 versus KL=1.54 with T5 local embeddings alone, showing better text adherence. It also reports that mean pooling of the T5 token embeddings, which adds no parameters, achieves FAD=1.89 and KL=1.51, whereas the dual-encoder model achieves FAD=1.94 and KL=1.47. In the paper's telling, the discovery is that a parameter-free pooling operation on a frozen T5 encoder produces a global text embedding that matches or exceeds the quality of a dedicated CLAP encoder.","pith_inferences":["The reported FAD and KL gaps come from single runs without error bars, and compared configurations differ in parameter count; a matched-parameter, multi-seed re-run would be needed to confirm that the improvements arise from the conditioning mechanism rather than capacity or variance.","A natural testable extension is to fine-tune the text encoder on music-related instructions and pair it with mean pooling; since FLANT5-large already outperforms T5-base under mean pooling, such tuning may close the remaining KL gap entirely.","The self-attention pooling collapse the authors observed (attention concentrating on a few frequent tokens) could be addressed by isotropic regularization or principal-component whitening; a controlled comparison would show whether SAP becomes competitive once that collapse is prevented.","The evaluation is on instrument-only MusicCaps audio; the conclusions may shift for vocal music or other domains because FAD and KL are computed on a specific feature distribution."],"forward_implications":["If the results generalize, text-to-music systems can drop the CLAP encoder and use a single frozen T5 with mean pooling, cutting about 124 million parameters without losing audio quality.","The FiLM-for-global and cross-attention-for-local split is a reusable architecture pattern for conditioning diffusion models in other modalities.","The failure of Sentence-T5 and SimCSE as global conditioners suggests the global embedding must be aligned with the generation task, not just semantically strong.","Scaling the language model (FLANT5-large) further improves the pooled approach, pointing to language-model capacity as the main lever for better music generation."],"supporting_citations":[{"why":"Supplies the latent diffusion backbone, autoencoder, vocoder, and the evaluation toolkit that the experiments build on.","marker":"[4]"},{"why":"Introduces the prior fusion of global and local text via a GPT-2 bridge, which is the comparison baseline for generation quality.","marker":"[5]"},{"why":"T5 is the uni-modal language model that provides the local text embeddings used across all conditioning configurations.","marker":"[9]"},{"why":"CLAP is the cross-modal audio-language model that supplies the global text embedding in the dual-encoder system.","marker":"[11]"},{"why":"FiLM is the conditioning layer through which the global text embedding is injected into the UNet.","marker":"[26]"},{"why":"Prior work that concatenates global and local text embeddings, motivating the proposed fusion design.","marker":"[18]"},{"why":"Self-attention pooling is the trainable pooling method evaluated as an alternative to mean pooling.","marker":"[27]"},{"why":"MusicCaps is the benchmark dataset on which FAD and KL scores are computed for comparison.","marker":"[13]"}],"fun_headline_variants":["T5 mean pooling: better fidelity, fewer parameters","No CLAP, no problem: T5 pooling ups fidelity, slight KL cost","Pool T5 for global text: higher fidelity, similar adherence","One encoder, two text views: T5 mean pool beats dual on fidelity","124M fewer params with mean-pooled T5 global text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the single-run differences in FAD and KL arise from the conditioning method, even though the compared systems differ in parameter count and show no error bars, so the gains could instead come from extra capacity or training luck.","fun_headline_variants_meta":{"raw":{"variants":["T5 mean pooling: better fidelity, fewer parameters","No CLAP, no problem: T5 pooling ups fidelity, slight KL cost","Pool T5 for global text: higher fidelity, similar adherence","One encoder, two text views: T5 mean pool beats dual on fidelity","124M fewer params with mean-pooled T5 global text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001109,"raw_usage":{"total_tokens":4665,"prompt_tokens":1033,"completion_tokens":3632,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":3540}},"tokens_in":649,"tokens_out":3632,"duration_ms":21135,"temperature":1.0,"reasoning_tokens":3540,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:54:33.204684+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-train the T5-local baseline with the same 606M parameter budget as the CLAP+T5 model (for example, using a larger T5) across at least three seeds; if that baseline reaches FAD at or below 1.89 and KL at or below 1.47, the reported advantages of global conditioning and mean pooling would not be established.","supporting_citations":[{"cited_title":"AudioLDM: Text-to-audio generation with latent diffusion models,","cited_arxiv_id":null,"evidence_quote":"Supplies the latent diffusion backbone, autoencoder, vocoder, and the evaluation toolkit that the experiments build on."},{"cited_title":"AudioLDM 2: Learning holistic audio generation with self-supervised pretraining,","cited_arxiv_id":null,"evidence_quote":"Introduces the prior fusion of global and local text via a GPT-2 bridge, which is the comparison baseline for generation quality."},{"cited_title":"Exploring the limits of transfer learning with a unified text-to-text transformer,","cited_arxiv_id":null,"evidence_quote":"T5 is the uni-modal language model that provides the local text embeddings used across all conditioning configurations."},{"cited_title":"Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,","cited_arxiv_id":null,"evidence_quote":"CLAP is the cross-modal audio-language model that supplies the global text embedding in the dual-encoder system."},{"cited_title":"FiLM: Visual reasoning with a general conditioning layer,","cited_arxiv_id":null,"evidence_quote":"FiLM is the conditioning layer through which the global text embedding is injected into the UNet."},{"cited_title":"Self-attention encoding and pooling for speaker recognition,","cited_arxiv_id":null,"evidence_quote":"Self-attention pooling is the trainable pooling method evaluated as an alternative to mean pooling."}],"review_version":1}