Pith. sign in

REVIEW 5 major objections 6 minor 70 references

GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Models

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Generating ghost audio queries from text improves in-domain text-music retrieval and adds inference-time controls.

desk verdict A clever diffusion-based retrieval mechanism with genuine controllability affordances, but the headline retrieval gains rest on a confounded comparison and need a fairer baseline before they hold up. read the letter →

arxiv 2506.17886 v2 pith:YL6HVVUW submitted 2025-06-22 cs.SD eess.AS

Julien Guinot , Elio Quinton , György Fazekas This is my paper · ORCID
classification cs.SDeess.AS
keywords text-musicretrievalgenerativelatentdiffusionghostqueriesnegativepromptingDDIMinversioncontrastivelearningmusicinformation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Text-music retrieval systems usually encode a text query and candidate audio clips into a shared embedding space and compare the resulting vectors. GD-Retriever instead trains a diffusion model to turn a text caption into a "ghost" audio embedding in the frozen audio encoder's own space, then retrieves by nearest-neighbor search with that generated query. The paper argues this generative route outperforms the contrastive teacher models it builds on for in-domain retrieval, works even when the text and audio encoders were never jointly trained, and brings generative controllability, negative prompting and DDIM inversion, to retrieval. A sympathetic reader would care because the proposal points from rigid joint-embedding search toward interactive, steerable music retrieval.

What carries the argument

The load-bearing mechanism is ghost-query generation in a retrieval-optimized latent space: a conditional latent diffusion model $G$ trained by minimizing $\mathcal{L}_G = \mathbb{E}_{\tau, Z_a, z_T^q}[\lVert z_a - G(z_a, \tau, z_T^q)\rVert_2^2]$ learns to produce audio-embedding sequences from text-embedding sequences, and the generated sequences are averaged over time and samples to form the retrieval query $\tilde{Z}_A^q$. Unlike prior generative retrieval, the model emits full embedding sequences rather than pooled embeddings, so text conditions interact with audio latents through cross-attention at a finer grain. Two standard diffusion controls then carry the interactivity: classifier-free guidance with a negative prompt, whose update replaces the unconditional prediction with a negatively conditioned prediction (Eq. 2), and DDIM inversion, which re-noises a generated latent partway and re-denoises it under a modified prompt.

What would settle it

Train GDR-CLAP on PrivateCaps with the 5,500 evaluation tracks and their captions strictly excluded, then measure Recall@1 on those held-out tracks; if the improvement over the CLAP teacher disappears or reverses, the in-domain gain is memorization rather than generative generalization. A sharper check is to compare each ghost query's distance to training-track latents versus test-track latents: if ghost queries sit closer to training tracks, the retriever is effectively returning memorized neighbours.

Watch

Extended reading notes

Core claim

GD-Retriever's central claim is that a text-conditioned latent diffusion model can serve as the query encoder of a retrieval system. The diffusion model is trained to reconstruct audio-embedding sequences extracted from a frozen audio encoder (CLAP, MusCALL, or MULE), conditioned on text-embedding sequences coming from any text encoder, including Flan-T5, which was never trained jointly with the audio encoder. At inference, the model generates $n_q=5$ ghost queries for a caption, averages them over the sequence dimension and over the samples, and uses the averaged latent $\tilde{Z}_A^q$ for nearest-neighbor retrieval in the audio space. On the in-domain PrivateCaps benchmark, GDR-CLAP raises Recall@1, the fraction of queries whose top retrieved track is the correct one, from 2.2 to 6.9 against the CLAP teacher, and GDR-MULE with a Flan-T5 text encoder outperforms the CLAP teacher on PrivateCaps despite MULE's space being audio-only. The paper further shows negative prompting shifts generated queries away from undesired attributes while keeping them realistic (low FAD), and DDIM inversion allows re-noising a query and re-guiding it toward a modified prompt while preserving similarity to the original. The authors report that out-of-domain MusicCaps performance degrades, and they attribute the drop to distribution shift inherited from the teacher's embedding space, offering a post-hoc mean-and-covariance alignment as a partial fix.

Load-bearing premise

The load-bearing premise is that the 5,500-track PrivateCaps test subset is a fair out-of-sample test of a model trained on the 251k-track PrivateCaps corpus, so the reported gains over contrastive teachers reflect genuine generative retrieval rather than memorization of the training distribution.

Editorial extensions

If this is right

  • Text-music retrieval becomes possible on top of any frozen audio encoder: a separately trained text encoder such as Flan-T5 can drive retrieval in MULE's audio-only space, removing the requirement for joint multimodal pretraining.
  • Inference-time negative prompting lets a user remove an undesired attribute (for example "a rock song") from a query, and the modified query stays in-distribution rather than drifting into unrealistic audio.
  • DDIM inversion supports a refine-and-search loop: a user takes a partly satisfactory retrieval result, re-noises it, re-guides it with a modified caption, and keeps the result close to the original while aligning with the new description.
  • GDR inherits the teacher's domain shift: the largest gains appear where the teacher's embedding distribution matches the training data, and a post-hoc mean/covariance shift toward the evaluation set reduces Fréchet audio distance and improves R@5.
  • On generation-quality metrics, the diffusion UNet beats regression UNet and MLP baselines on retrieval and fidelity, while an MLP diffusion baseline produces more diverse queries, a fidelity-diversity trade-off in generative retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the in-domain gains survive strict exclusion of the evaluation tracks from training, the same ghost-query recipe should transfer to other modalities with pre-trained frozen encoders and no joint text encoder, such as images, sound effects, or speech.
  • Editorial inference: because the text encoder is fully decoupled, practitioners could swap in a larger instruction-tuned language model at inference time to improve caption interpretation without any retraining of the audio side.
  • Editorial inference: the controllability demonstrations suggest an interactive retrieval loop, retrieve, invert, edit, re-retrieve, whose end-to-end value would need user studies or retrieval metrics on systematically perturbed captions rather than the paper's 50-prompt evaluations alone.
  • Editorial inference: a sharper test of the mechanism would compare ghost-query retrieval against the teacher's own text encoder combined with test-time distribution normalization; if normalization closes most of the gap, the generative model's benefit may be mainly adaptation to the training distribution.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces GD-Retriever, a framework that trains a conditional latent diffusion model to generate 'ghost' audio queries in the latent space of a frozen audio encoder, conditioned on text embeddings. At inference, several generated queries are averaged over time and over the number of generated samples, then used for nearest-neighbor retrieval in the audio space. The authors evaluate retrieval on PrivateCaps, SongDescriber, and MusicCaps against CLAP and MusCALL teachers, study domain shift and a post-hoc alignment mitigation, demonstrate retrieval with non-jointly-trained text encoders such as Flan-T5 and audio encoders such as MULE, and present qualitative and quantitative controllability experiments using negative prompting and DDIM inversion. The central claims are that GD-Retriever improves retrieval over contrastive teachers in in-domain settings, enables audio-only latent-space retrieval, and adds inference-time controllability that joint-embedding models do not natively provide.

Significance. If the claims hold, the paper makes a useful contribution: it connects generative diffusion models to controllable music retrieval in a way that is not a trivial re-implementation of existing generative retrieval work, because it operates on latent sequences rather than pooled embeddings and demonstrates controllability mechanisms (negative prompting, DDIM inversion) that are native to diffusion but absent from standard contrastive retrieval. The release of code and the inclusion of multiple encoder-pair ablations (Tables 2-4) are strengths. The main retrieval claim, however, is not yet supported as stated: the paper's own results show out-of-domain degradation, and the largest in-domain gains are confounded with training on the evaluation domain. The controllability experiments are suggestive but rely on a metric computed in the same embedding family used to train the model. These issues are addressable within the scope of the manuscript, so the work warrants a major revision rather than rejection.

major comments (5)
  1. [Abstract and Table 2] The unqualified abstract claim that 'GDR improves retrieval performance over contrastive teacher models' is contradicted by the paper's own results. GDR-CLAP is worse than CLAP on MusicCaps (R@5 7.6 vs. 12.9) and GDR-MusCALL is worse than MusCALL on PrivateCaps (R@5 25.1 vs. 26.2). Section 4.1 itself states that 'GDR-CLAP underperforms on MC relative to the CLAP teacher,' so the improvement claim should either be made explicitly conditional (e.g., limited to in-domain or specific teacher-dataset combinations) or be supported by additional evidence isolating when and why generative query formation helps.
  2. [Sections 3.3 and 4.1, Table 2] The main retrieval comparison is confounded by training distribution. GDR is trained on PrivateCaps, and its largest improvements over the CLAP teacher appear on PrivateCaps, where CLAP was not trained. The paper never includes a control in which the contrastive teacher is fine-tuned on the same PrivateCaps training data. Without this control, the PrivateCaps gains can be explained by GDR fitting the target audio/text distribution rather than by the generative retrieval mechanism itself. Notably, GDR-MusCALL, whose teacher is described as in-domain on PC in Section 4.1.1, does not improve over MusCALL on PC (R@5 25.1 vs. 26.2), which is consistent with the confound.
  3. [Section 4.1.1, Table 3] The alignment experiment fits a mean and covariance shift to the evaluation set and then reports retrieval R@5 on that same evaluation set. This is in-sample evaluation and inflates the reported improvements. The alignment parameters should be fitted on a held-out split (or via cross-validation) before evaluating retrieval, or the numbers should be presented only as a diagnostic of distribution shift rather than as evidence of an effective mitigation.
  4. [Tables 2-6] No error bars or significance tests are reported anywhere, which is especially important because GDR uses stochastic generation with nq = 5 and several reported differences are small (e.g., GDR-MusCALL on SD R@5 16.9 vs. 13.6; GDR-MusCALL on PC R@1 10.8 vs. 10.1). Without repeated-generation statistics or confidence intervals, these improvements cannot be reliably distinguished from sampling variability. At minimum, report means and standard deviations over multiple generations.
  5. [Section 4.3.1, Table 6] The negative-prompting evaluation uses CLAP score as the semantic ground truth, but GDR-CLAP is trained in CLAP's latent space and conditioned on CLAP's text encoder. If the CLAP model used for scoring is the same one used for training, the controllability metric may be partially circular. Please state explicitly which CLAP model is used for scoring and discuss this dependence; adding a human evaluation or a third-model semantic similarity metric would strengthen the controllability claim.
minor comments (6)
  1. [Equation (1)] The generator call 'G(za, tau, tau, zqT)' appears to include a duplicated time argument; please correct the notation.
  2. [Section 4.2, Table 5] The table header says 'Retrieval (SD)' while the text says the results are on PrivateCaps; please clarify which evaluation dataset is used and make the table and text consistent.
  3. [Tables 2 and 4] Table 2 uses 'MedR (%)' while Table 4 uses 'MR'; define the metric and use a single notation throughout.
  4. [Section 4.3.1] The 'CLAP score' is used as a central evaluation metric but is never defined or referenced; please add a definition and a citation.
  5. [Section 3.3] The paper states that classifier-free guidance is applied on text conditioning with a 10% masking probability, but does not report the guidance weight w used at inference; please specify it.
  6. [Section 4.1] The retrieval protocol does not state whether nearest-neighbor search uses cosine similarity, L2 distance, or another metric in the audio latent space; please specify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GDR's derivation is a trained conditional diffusion model with independently evaluated retrieval, not a reduction of predictions to fitted inputs.

full rationale

The central mechanism is a conditional latent diffusion model trained with Eq. (1), where the model learns to reconstruct frozen audio latents from text embeddings. Retrieval is then standard nearest-neighbor search in that frozen audio space. The claimed improvements over contrastive teachers are empirical results from Tables 2-4, not quantities forced by construction. The PrivateCaps training/evaluation overlap is a potential confound for interpreting the improvement, but it is a dataset-design issue, not a circular derivation: the teacher was not trained on PrivateCaps, and GDR's output is not a re-statement of its training labels. The only self-citation is MULE reimplemented following [47]; MULE is used as one of several encoders, and the GDR-MULE results are compared directly against an external CLAP teacher, so the central generative-retrieval claim does not reduce to that citation. The controllability experiments use CLAP score from the same embedding family used for GDR-CLAP training, which is a mild evaluation-metric overlap, but the paper reports measured similarity behavior rather than deriving the conclusion from the metric definition; no fitted parameter is renamed as a prediction. Overall, the derivation chain is self-contained and no load-bearing step reduces to its own input.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method relies on standard diffusion and contrastive learning assumptions. The main hand-chosen quantities are the number of generated queries nq=5, the latent sequence length T=64, and the unreported CFG weight w. It also assumes the private PrivateCaps corpus is representative and leakage-free, and that averaging over time and samples retains retrieval-relevant semantics. No new physical entities are introduced; the 'ghost query' is an inference-time artifact.

free parameters (3)
  • classifier-free guidance weight w = not reported
    Controls conditioning strength in generation and negative prompting (Eq. 2); no value is given for standard retrieval or controllability experiments, so results may depend on unreported tuning.
  • number of generated queries nq = 5
    Chosen without ablation; retrieval results average over 5 stochastic queries, so sensitivity is unknown.
  • latent sequence length T = 64 (1 minute of audio)
    Training uses 1-minute windows; evaluation datasets have different durations (10s, 2min), and the paper does not explain how variable-length inputs are handled at inference.
assumptions (4)
  • domain assumption The pretrained audio encoder's latent space is a suitable target for text-conditioned diffusion generation, i.e., generated points in this space correspond to plausible audio tracks.
    Section 3: GDR is trained to reconstruct frozen audio latents; if the space is not semantically smooth, generated ghost queries may be off-manifold.
  • domain assumption Averaging generated query latents over time steps and nq stochastic samples preserves the semantic content needed for retrieval.
    Section 3: 'We aggregate ~zq_A into ~Zq_A through averaging over the sequence timesteps and use ~Zq_A as a query'; averaging discards structure that could matter for retrieval.
  • domain assumption The PrivateCaps train/test split is leakage-free and representative of text-music retrieval queries.
    Section 3.2: PrivateCaps is a private dataset; the paper does not describe how the 5500-sample test subset is disjoint from the 251k training tracks.
  • domain assumption CLAP score (cosine similarity between CLAP embeddings) is a valid measure of semantic alignment and controllability quality.
    Section 4.2 and 4.3: controllability is evaluated with CLAP score, using the same embedding family that GDR is trained on for GDR-CLAP.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Models." pith.science (2026). https://pith.science/paper/YL6HVVUW

@misc{pith2026250617886,
  author       = {Pith},
  title        = {Pith review of: GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YL6HVVUW}},
  note         = {Machine review of arXiv:2506.17886}
}
read the original abstract

Multimodal contrastive models have achieved strong performance in text-audio retrieval and zero-shot settings, but improving joint embedding spaces remains an active research area. Less attention has been given to making these systems controllable and interactive for users. In text-music retrieval, the ambiguity of freeform language creates a many-to-many mapping, often resulting in inflexible or unsatisfying results. We introduce Generative Diffusion Retriever (GDR), a novel framework that leverages diffusion models to generate queries in a retrieval-optimized latent space. This enables controllability through generative tools such as negative prompting and denoising diffusion implicit models (DDIM) inversion, opening a new direction in retrieval control. GDR improves retrieval performance over contrastive teacher models and supports retrieval in audio-only latent spaces using non-jointly trained encoders. Finally, we demonstrate that GDR enables effective post-hoc manipulation of retrieval behavior, enhancing interactive control for text-music retrieval tasks.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 58 canonical work pages

  1. [1]

    I would like this retrieval result to be punchier

    INTRODUCTION Multimodal text-music joint embedding models have largely facilitated text-queried music retrieval applications [1–5]. Multimodal contrastive learning of text and music joint embedding spaces specifically have shown high per- formance on frozen probing tasks and promise for zero- shot classification approaches, with strong representation lear...

    arXiv 2025

  2. [2]

    BACKGROUND 2.1 Text-music contrastive learning and retrieval Multimodal contrastive learning has shown strong results in computer vision [6, 21–23], and has been successfully extended to audio and music domains [1,2,5]. These mod- els encode paired text and audio inputs using encodersET andEA, project them into a shared latent space, and apply a contrasti...

    work page

  3. [3]

    Using a pretrained latent space op- timized for audio-audio retrieval, we train a generative dif- fusion model conditioned on text to generate audio latent embeddings in this space

    GENERA TIVE DIFFUSION RETRIEV AL We propose an intuitive generative approach to retrieval using diffusion models, which we name Generative Diffusion Retriever. Using a pretrained latent space op- timized for audio-audio retrieval, we train a generative dif- fusion model conditioned on text to generate audio latent embeddings in this space. At inference ti...

    work page

  4. [4]

    a rock song

    EXPERIMENTS 4.1 Retrieval T→A Eval dataset Model Metric PC SD MC CLAP R@1↑ 2.2 3.1 3.8 R@5↑ 7.2 13.7 12.9 R@10↑ 12.3 23.2 19.5 MedR (%)↓ 3.7 4.0 1.4 GDR-CLAP R@1↑ 6.9 4.7 2.7 R@5↑ 17.1 15.3 7.6 R@10↑ 22.9 24.7 11.5 MedR (%)↓ 1.6 3.8 2.9 MusCALL R@1↑ 10.1 3.6 1.0 R@5↑ 26.2 13.6 3.9 R@10↑ 35.1 22.0 7.0 MedR (%)↓ 0.4 4.2 5.1 GDR-MusCALL R@1↑ 10.8 5.1 1.8 R@5...

    work page

  5. [5]

    CONCLUSION AND FUTURE WORK We present GD-Retriever, a generative framework for text-to-music retrieval that uses diffusion models to pro- duce latent queries in retrieval-relevant spaces. GD- Retriever outperforms contrastive teacher models on in- domain data and enables retrieval in unimodal audio spaces by leveraging independently pretrained text and au...

    work page

  6. [6]

    does not mean

    Fine-grained category experiments are shown Table 3. key Zq+ A Modified query˜ZmodA NP ∆T ˜∆A PNP CLAP↑ ZA 0.69 0.66 0.38 0.39 0.51˜ZA 1 0.85 0.65 0.62 0.49 ZT 0.42 0.39 0.28 0.33 0.21 CLAP↓ ˜ZnA 0.41 0.21 -0.02 -0.23 0.46 ZnT 0.17 0.08 -0.51 -0.04 0.21 Fidelity FAD 0.11 0.12 3.12 0.60 0.12 Table 6: Negative prompting experiments - CLAP score of modified ...

    work page

  7. [7]

    ACKNOWLEDGEMENT This work is supported by the EPSRC UKRI Centre for Doctoral Training in Artificial Intelligence and Music (EP/S022694/1) and Universal Music Group

  8. [8]

    Contrastive audio-language learning for music,

    I. Manco, E. Benetos, E. Quinton et al., “Contrastive audio-language learning for music,” in Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Bengaluru, India, December 4-8, 2022, 2022, pp. 640–649

    work page 2022

Show all 70 references
  1. [9]

    Mulan: A joint em- bedding of music audio and natural language,

    Q. Huang, A. Jansen, J. Lee et al., “Mulan: A joint em- bedding of music audio and natural language,” in Pro- ceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Ben- galuru, India, December 4-8, 2022 , 2022, pp. 559– 566

    2022

  2. [10]

    Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang et al. , “Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in ICASSP 2023-2023 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  3. [11]

    Collap: Contrastive long-form language-audio pretraining with musical temporal structure augmentation,

    J. Wu, W. Li, Z. Novack et al., “Collap: Contrastive long-form language-audio pretraining with musical temporal structure augmentation,” in ICASSP 2025- 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  4. [12]

    Clap learning audio concepts from natural language supervi- sion,

    B. Elizalde, S. Deshmukh, M. Al Ismail et al., “Clap learning audio concepts from natural language supervi- sion,” in ICASSP 2023-2023 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  5. [13]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020

    2020

  6. [14]

    AudioLDM: Text- to-audio generation with latent diffusion models,

    H. Liu, Z. Chen, Y . Yuan et al. , “AudioLDM: Text- to-audio generation with latent diffusion models,”Pro- ceedings of the International Conference on Machine Learning, 2023

    2023

  7. [15]

    Audioldm 2: Learning holistic audio generation with self-supervised pretrain- ing,

    H. Liu, Y . Yuan, X. Liu et al., “Audioldm 2: Learning holistic audio generation with self-supervised pretrain- ing,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  8. [16]

    Musicldm: En- hancing novelty in text-to-music generation using beat-synchronous mixup strategies,

    K. Chen, Y . Wu, H. Liu et al. , “Musicldm: En- hancing novelty in text-to-music generation using beat-synchronous mixup strategies,” arXiv preprint arXiv:2308.01546, 2023

    2023 arXiv

  9. [17]

    Music con- trolnet: Multiple time-varying controls for music gen- eration,

    S.-L. Wu, C. Donahue, S. Watanabe et al., “Music con- trolnet: Multiple time-varying controls for music gen- eration,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  10. [18]

    Text-to- audio generation using instruction guided latent diffu- sion model,

    D. Ghosal, N. Majumder, A. Mehrish et al., “Text-to- audio generation using instruction guided latent diffu- sion model,” in Proceedings of the 31st ACM Interna- tional Conference on Multimedia, 2023, p. 3590–3598

    2023

  11. [19]

    Diff-a-riff: Musical accompaniment co-creation via latent diffu- sion models,

    J. Nistal, M. Pasini, C. Aouameur et al., “Diff-a-riff: Musical accompaniment co-creation via latent diffu- sion models,” in ISMIR, 2024, 2024

    2024

  12. [20]

    Ditto: diffusion inference-time t-optimization for mu- sic generation,

    Z. Novack, J. McAuley, T. Berg-Kirkpatrick et al. , “Ditto: diffusion inference-time t-optimization for mu- sic generation,” in Proceedings of the 41st Interna- tional Conference on Machine Learning , 2024, pp. 38 426–38 447

    2024

  13. [21]

    Long-form mu- sic generation with latent diffusion,

    Z. Evans, J. D. Parker, C. Carr et al., “Long-form mu- sic generation with latent diffusion,” in Proceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR), 2024

    2024

  14. [22]

    Fast timing- conditioned latent audio diffusion,

    Z. Evans, C. Carr, J. Taylor et al. , “Fast timing- conditioned latent audio diffusion,” in Proceedings of the 41st International Conference on Machine Learn- ing, 2024, pp. 12 652–12 665

    2024

  15. [23]

    An im- age is worth one word: Personalizing text-to-image generation using textual inversion,

    R. Gal, Y . Alaluf, Y . Atzmon et al. , “An im- age is worth one word: Personalizing text-to-image generation using textual inversion,” arXiv preprint arXiv:2208.01618, 2022

    2022 arXiv

  16. [24]

    Null-text inversion for editing real images using guided diffusion models,

    R. Mokady, A. Hertz, K. Aberman et al. , “Null-text inversion for editing real images using guided diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 6038–6047

    2023

  17. [25]

    Dynamic prompt learning: Addressing cross-attention leakage for text- based image editing,

    F. Yang, S. Yang, M. A. Butt et al., “Dynamic prompt learning: Addressing cross-attention leakage for text- based image editing,” Advances in Neural Information Processing Systems, vol. 36, pp. 26 291–26 303, 2023

    2023

  18. [26]

    Con- tinuous, subject-specific attribute control in t2i mod- els by identifying semantic directions,

    S. A. Baumann, F. Krause, M. Neumayr et al., “Con- tinuous, subject-specific attribute control in t2i mod- els by identifying semantic directions,” arXiv preprint arXiv:2403.17064, 2024

    2024 arXiv

  19. [27]

    Compositional in- version for stable diffusion models,

    X. Zhang, X.-Y . Wei, J. Wu et al., “Compositional in- version for stable diffusion models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 7, 2024, pp. 7350–7358

    2024

  20. [28]

    Learn- ing transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy et al. , “Learn- ing transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763

    2021

  21. [29]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov et al., “Sigmoid loss for language image pre-training,” in Proceedings of the IEEE/CVF International Conference on Com- puter Vision, 2023, pp. 11 975–11 986

    2023

  22. [30]

    Improving fine- grained understanding in image-text pre-training,

    I. Bica, A. Ili ´c, M. Bauer et al. , “Improving fine- grained understanding in image-text pre-training,” in Proceedings of the 41st International Conference on Machine Learning, 2024, pp. 3974–3995

    2024

  23. [31]

    A simple framework for contrastive learning of visual represen- tations,

    T. Chen, S. Kornblith, M. Norouzi et al. , “A simple framework for contrastive learning of visual represen- tations,” inInternational conference on machine learn- ing. PMLR, 2020, pp. 1597–1607

    2020

  24. [32]

    T-clap: Temporal- enhanced contrastive language-audio pretraining,

    Y . Yuan, Z. Chen, X. Liu et al. , “T-clap: Temporal- enhanced contrastive language-audio pretraining,” in 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2024, pp. 1–6

    2024

  25. [33]

    Augment, drop & swap: Improving diversity in llm captions for efficient music-text representation learning,

    I. Manco, J. Salamon, and O. Nieto, “Augment, drop & swap: Improving diversity in llm captions for efficient music-text representation learning,” in Proceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR), 2024

    2024

  26. [34]

    Cacophony: An improved contrastive audio-text model,

    G. Zhu, J. Darefsky, and Z. Duan, “Cacophony: An improved contrastive audio-text model,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, 2024

    2024

  27. [35]

    Spec- MaskGIT: Masked Generative Modeling of Audio Spectrograms for Efficient Audio Synthesis and Be- yond,

    M. Comunità, Z. Zhong, A. Takahashi et al., “Spec- MaskGIT: Masked Generative Modeling of Audio Spectrograms for Efficient Audio Synthesis and Be- yond,” inProceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR) , 2024

    2024

  28. [36]

    Drcap: Decoding clap la- tents with retrieval-augmented generation for zero-shot audio captioning,

    X. Li, W. Chen, Z. Maet al., “Drcap: Decoding clap la- tents with retrieval-augmented generation for zero-shot audio captioning,” in ICASSP 2025-2025 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  29. [37]

    Recap: Retrieval-augmented audio captioning,

    S. Ghosh, S. Kumar, C. K. R. Evuru et al. , “Recap: Retrieval-augmented audio captioning,” in ICASSP 2024-2024 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 1161–1165

    2024

  30. [38]

    Taming trans- formers for high-resolution image synthesis,

    P. Esser, R. Rombach, and B. Ommer, “Taming trans- formers for high-resolution image synthesis,” in Pro- ceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, 2021, pp. 12 873–12 883

    2021

  31. [39]

    Scaling autoregres- sive models for content-rich text-to-image generation,

    J. Yu, Y . Xu, J. Y . Koh et al. , “Scaling autoregres- sive models for content-rich text-to-image generation,” Transactions on Machine Learning Research, 2022

    2022

  32. [40]

    Hierarchi- cal text-conditional image generation with clip latents,

    A. Ramesh, P. Dhariwal, A. Nichol et al., “Hierarchi- cal text-conditional image generation with clip latents,” arXiv preprint arXiv:2204.06125 , vol. 1, no. 2, p. 3, 2022

    2022 arXiv

  33. [41]

    Diffgap: A lightweight diffusion module in contrastive space for bridging cross-model gap,

    S. Mo, Z. Chen, F. Bao et al., “Diffgap: A lightweight diffusion module in contrastive space for bridging cross-model gap,” in ICASSP 2025-2025 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  34. [42]

    Classifier-free diffusion guid- ance,

    J. Ho and T. Salimans, “Classifier-free diffusion guid- ance,” in NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2022

    2021

  35. [43]

    Scalable diffusion models with transformers,

    W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision , 2023, pp. 4195–4205

    2023

  36. [44]

    High- resolution image synthesis with latent diffusion mod- els,

    R. Rombach, A. Blattmann, D. Lorenz et al., “High- resolution image synthesis with latent diffusion mod- els,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 10 684–10 695

    2022

  37. [45]

    Moûsai: Text-to-music generation with long-context latent dif- fusion,

    F. Schneider, Z. Jin, and B. Schölkopf, “Moûsai: Text-to-music generation with long-context latent dif- fusion,” arXiv e-prints, pp. arXiv–2301, 2023

    2023

  38. [46]

    Prompt tuning inver- sion for text-driven image editing using diffusion mod- els,

    W. Dong, S. Xue, X. Duan et al., “Prompt tuning inver- sion for text-driven image editing using diffusion mod- els,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 7430–7440

    2023

  39. [47]

    Prompt- to-prompt image editing with cross-attention control,

    A. Hertz, R. Mokady, J. Tenenbaum et al., “Prompt- to-prompt image editing with cross-attention control,” in The Eleventh International Conference on Learning Representations, 2022

    2022

  40. [48]

    Prompt sliders for fine-grained control, editing and erasing of concepts in diffusion models,

    D. Sridhar and N. Vasconcelos, “Prompt sliders for fine-grained control, editing and erasing of concepts in diffusion models,” arXiv preprint arXiv:2409.16535 , 2024

    2024 arXiv

  41. [49]

    Re- paint: Inpainting using denoising diffusion probabilis- tic models,

    A. Lugmayr, M. Danelljan, A. Romero et al. , “Re- paint: Inpainting using denoising diffusion probabilis- tic models,” in Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, 2022, pp. 11 461–11 471

    2022

  42. [50]

    Negative- prompt inversion: Fast image inversion for editing with text-guided diffusion models,

    D. Miyake, A. Iohara, Y . Saito et al. , “Negative- prompt inversion: Fast image inversion for editing with text-guided diffusion models,” arXiv preprint arXiv:2305.16807, 2023

    2023 arXiv

  43. [51]

    Musicmagus: zero-shot text-to-music editing via diffusion models,

    Y . Zhang, Y . Ikemiya, G. Xia et al. , “Musicmagus: zero-shot text-to-music editing via diffusion models,” in Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, 2024, pp. 7805– 7813

    2024

  44. [52]

    Arrange, inpaint, and refine: steerable long-term music audio generation and editing via content-based controls,

    L. Lin, G. Xia, Y . Zhang et al., “Arrange, inpaint, and refine: steerable long-term music audio generation and editing via content-based controls,” in Proceedings of the Thirty-Third International Joint Conference on Ar- tificial Intelligence, 2024, pp. 7690–7698

    2024

  45. [53]

    Disentangled multidimensional metric learning for music similarity,

    J. Lee, N. J. Bryan, J. Salamon et al., “Disentangled multidimensional metric learning for music similarity,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6–10

    2020

  46. [54]

    Leave-one- equivariant: Alleviating invariance-related information loss in contrastive music representations,

    J. Guinot, E. Quinton, and G. Fazekas, “Leave-one- equivariant: Alleviating invariance-related information loss in contrastive music representations,” in ICASSP 2025-2025 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  47. [55]

    Similar but faster: manipulation of tempo in music audio em- beddings for tempo prediction and search,

    M. C. McCallum, F. Henkel, J. Kim et al. , “Similar but faster: manipulation of tempo in music audio em- beddings for tempo prediction and search,” in ICASSP 2024-2024 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 686–690

    2024

  48. [56]

    Diff4steer: Steer- able diffusion prior for generative music retrieval with semantic guidance,

    X. Bao, J. Y . Li, Z. Y . Wan et al., “Diff4steer: Steer- able diffusion prior for generative music retrieval with semantic guidance,” inICASSP 2025-2025 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  49. [57]

    Supervised and unsupervised learning of audio rep- resentations for music understanding,

    M. C. McCallum, F. Korzeniowski, S. Oramas et al., “Supervised and unsupervised learning of audio rep- resentations for music understanding,” in Ismir 2022 Hybrid Conference, 2022

    2022

  50. [58]

    Hts-at: A hierarchical token-semantic audio transformer for sound classifica- tion and detection,

    K. Chen, X. Du, B. Zhu et al., “Hts-at: A hierarchical token-semantic audio transformer for sound classifica- tion and detection,” in ICASSP 2022-2022 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 646–650

    2022

  51. [59]

    Scal- ing instruction-finetuned language models,

    H. W. Chung, L. Hou, S. Longpre et al. , “Scal- ing instruction-finetuned language models,” Journal of Machine Learning Research, vol. 25, no. 70, pp. 1–53, 2024

    2024

  52. [60]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal et al., “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  53. [61]

    Mustango: Toward controllable text-to-music generation,

    J. Melechovsky, Z. Guo, D. Ghosal et al., “Mustango: Toward controllable text-to-music generation,” in Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies (Volume 1: Long Papers), 20...

    2024

  54. [62]

    Simple and control- lable music generation,

    J. Copet, F. Kreuk, I. Gat et al., “Simple and control- lable music generation,” Advances in Neural Informa- tion Processing Systems , vol. 36, pp. 47 704–47 720, 2023

    2023

  55. [63]

    The song describer dataset: a corpus of audio captions for music-and- language evaluation,

    I. Manco, B. Weck, S. Doh et al., “The song describer dataset: a corpus of audio captions for music-and- language evaluation,” NeurIPS Machine Learning for Audio Workshop, 2023

    2023

  56. [64]

    Mu- sicLM: Generating music from text,

    A. Agostinelli, T. I. Denk, Z. Borsos et al. , “Mu- sicLM: Generating music from text,” arXiv preprint arXiv:2301.11325, 2023

    2023 arXiv

  57. [65]

    Coco-dr: Combating distribution shifts in zero-shot dense retrieval with con- trastive and distributionally robust learning,

    Y . Yu, C. Xiong, S. Sun et al., “Coco-dr: Combating distribution shifts in zero-shot dense retrieval with con- trastive and distributionally robust learning,” in Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, 2022, pp. 1462– 1479

    2022

  58. [66]

    Selecting which dense retriever to use for zero-shot search,

    E. Khramtsova, S. Zhuang, M. Baktashmotlagh et al., “Selecting which dense retriever to use for zero-shot search,” in Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region , 2023, pp. 223–233

    2023

  59. [67]

    Test-time distribution nor- malization for contrastively learned visual-language models,

    Y . Zhou, J. Ren, F. Liet al., “Test-time distribution nor- malization for contrastively learned visual-language models,” Advances in Neural Information Processing Systems, vol. 36, pp. 47 105–47 123, 2023

    2023

  60. [68]

    Correlation alignment for unsupervised domain adaptation,

    B. Sun, J. Feng, and K. Saenko, “Correlation alignment for unsupervised domain adaptation,” Domain adap- tation in computer vision applications , pp. 153–171, 2017

    2017

  61. [69]

    The vendi score: A di- versity evaluation metric for machine learning,

    D. Friedman and A. B. Dieng, “The vendi score: A di- versity evaluation metric for machine learning,” arXiv preprint arXiv:2210.02410, 2022

    2022 arXiv

  62. [70]

    The mtg- jamendo dataset for automatic music tagging,

    D. Bogdanov, M. Won, P. Tovstogan et al., “The mtg- jamendo dataset for automatic music tagging,” in Ma- chine Learning for Music Discovery Workshop, In- ternational Conference on Machine Learning (ICML 2019), Long Beach, CA, United States, 2019

    2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.