Pith. sign in

REVIEW 2 major objections 3 minor 13 references

Generative AI for Music and Audio

T0 review · 2 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Text-queried sound separation and text-to-audio synthesis can be learned from unlabeled video alone, with CLIP's image-text embedding space serving as the bridge from text to audio.

desk verdict This is a thesis-by-publications, not a new research paper: no new technical claims, but the constituent papers are solid, and the only real overstatement is the word 'universal' in the CLIPSep chapter, which the author's own ESC-50 results undercut. read the letter →

arxiv 2411.14627 v1 pith:XQK3XJKJ submitted 2024-11-21 cs.SD cs.AIcs.LGcs.MMeess.AS

Hao-Wen Dong This is my paper · ORCID
classification cs.SDcs.AIcs.LGcs.MMeess.AS
keywords generativeAIformusictext-queriedsoundseparationtext-to-audiosynthesisCLIPunlabeledvideolearningnoiseinvarianttrainingmultitrackgenerationsymbolic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This dissertation collects six research lines in generative AI for music and audio, from symbolic multitrack music generation tools to score-to-audio performance synthesis. Its central technical claim, carried by the CLIPSep and CLIPSonic chapters, is that text-audio alignment for sound separation and synthesis can be learned without any paired text-audio data: a model trained only on noisy, unlabeled videos can be queried with text at test time, because the pretrained CLIP image-text embedding space is used as a bridge between the visual content of a video frame and the words that name it. The CLIPSep system is presented as the first text-queried universal sound separation model that can be trained without labeled text-audio pairs, and it reports competitive performance against a supervised model in some settings. CLIPSonic extends the same video-bridged, label-free idea to text-to-audio synthesis. If the claim is right, the bottleneck for building controllable audio tools shifts from collecting expensive text-audio datasets to exploiting the enormous amount of unlabeled video already available.

What carries the argument

The load-bearing mechanism is the frozen CLIP embedding space, a pretrained joint representation in which matching images and texts are close together, used as a fixed bridge between text, image, and, through video, audio. During training a video frame is encoded by CLIP's image encoder and projected into a query vector that mixes an over-determined set of masks produced by an audio U-Net; at test time a text prompt is encoded by CLIP's text encoder and fed down the same path. The second mechanism, noise invariant training, adds permutation-invariant noise heads whose outputs are interchangeable in the loss so that off-screen and background sounds are absorbed during training and discarded at inference.

What would settle it

Evaluate a CLIPSep-NIT model trained on unlabeled VGGSound video on mixtures whose target sounds are textually clear but visually unobservable, such as a dog barking off-screen or an obscured alarm clock: if text-queried SDR is no better than the input mixture while a supervised text-queried model succeeds, the claimed zero-shot visual bridge is refuted.

Watch

Extended reading notes

Core claim

The central discovery is that the visual modality can supply the semantic supervision that text-audio pairs normally provide. CLIPSep trains an audio separation model on image-audio pairs cropped from video: the CLIP image encoder turns a video frame into a query vector, that query vector weights a bank of intermediate masks from an audio U-Net, and the model is trained to separate the audio that corresponds to the image. At test time the same pathway is fed a text embedding from CLIP, and because CLIP was trained to put matching images and text close together, the text query separates the target sound without any text-audio training data. Because real videos contain off-screen sounds and background noise, CLIPSep-NIT adds permutation-invariant noise heads that absorb query-irrelevant audio during training and are discarded at inference. The reported numbers show text-queried separation learned from unlabeled video reaching competitive performance against a supervised model in some settings, and CLIPSonic carries the same bridge over to text-to-audio synthesis.

Load-bearing premise

The load-bearing premise is that CLIP's shared text-image embedding space lines up with audio content well enough that a text prompt can replace the video-frame query the model was trained on.

Editorial extensions

If this is right

  • Text-queried universal sound separation can be trained without labeled text-audio data, using only the naturally occurring audio-visual correspondence in unlabeled video.
  • On the MUSIC+ evaluation the label-free CLIPSep-NIT model lands within 0.46 dB mean SDR of a fully supervised text-queried model and edges ahead on median SDR.
  • The same bridge is reusable for text-to-audio synthesis, so a single idea covers both separation and generation with no text-audio pairs.
  • Because noise heads absorb irrelevant sound during training, the approach tolerates the off-screen audio and background noise that dominate internet video.
  • Text, image, and hybrid queries share one query pathway, and multiple queries can be combined to extract several target sounds from one mixture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the bridge holds, the practical ceiling for such systems is set by CLIP's text-image alignment rather than by available audio labels, so improving or specializing the joint embedding space should directly improve separation quality.
  • The reported modality-transfer gaps of 0.88 to 1.79 dB SDR measure exactly the cost of replacing an image query with a text query; a testable prediction is that fine-tuning the CLIP encoders on target-domain audio-visual data would shrink those gaps.
  • The video-bridge strategy should falter for sounds whose visual and acoustic identities diverge, such as off-screen or invisible sources, so the approach is best understood as a method for visually grounded sound classes.
  • The same noise-invariant conditioning principle could be carried into adjacent generation tasks, such as text-conditioned music generation or video-to-audio synthesis, where unlabeled video is abundant and paired supervision is scarce.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. This dissertation compiles six previously published papers by the author into three research directions: multitrack music generation (MusPy and MMT), assistive music creation tools (automatic instrumentation and Deep Performer), and multimodal learning for audio and music (CLIPSep and CLIPSonic). The central technical contribution, carried by Chapter 6, is CLIPSep, a text-queried sound separation model trained without labeled audio-text pairs by using CLIP image-text embeddings together with noisy unlabeled videos, and its extension CLIPSep-NIT with noise-invariant training. Chapter 7 applies the same video-bridged idea to CLIPSonic for text-to-audio synthesis. Each chapter reports experiments, ablations, subjective tests, and, for several components, releases code repositories.

Significance. If the central claim holds, the dissertation demonstrates a scalable route to text-audio alignment for separation and synthesis without paired text-audio supervision, using only unlabeled video and a frozen CLIP model; this could substantially reduce data collection cost for controllable audio tools. The work has notable strengths: the MusPy, MMT, and CLIPSep repositories are linked, hyperparameters are reported in detail, systematic ablations are provided (e.g., feature and augmentation ablations in Chapter 4, positional-encoding ablations in Chapter 5, noise-regularization sweeps in Chapter 6), and the self-attention analysis in Chapter 3 is a useful contribution. However, the 'universal' label attached to the Chapter 6 results is broader than the evidence supports, because the zero-shot modality transfer is demonstrated mainly on visually salient classes and fails on ESC-50 without finetuning.

major comments (2)
  1. [Chapter 6 abstract; Table 6.6; Section 6.5] The abstract's claim that CLIPSep successfully learns 'text-queried universal sound separation' is not supported for non-visually-salient or ambient sound classes. In the zero-shot transfer experiment of Section 6.H/Table 6.6, the CLIPSep model trained on VGGSound achieves a mean SDR of 1.07 dB on ESC-50 against a 0.00 dB mixture baseline, which is effectively no separation, whereas finetuning on ESC-50 raises the mean SDR to 6.73 dB. The paper does not report an ESC-50 result for CLIPSep-NIT, so the best unlabeled variant's universality remains untested. Since Section 6.5 itself concedes that the approach 'requires a sufficient audio-visual correspondence,' the qualifier 'universal' should be replaced or explicitly scoped to visually salient classes, or supported by evidence on diverse non-visual and ambient sound categories.
  2. [Section 5.3 and Section 5.4] The Deep Performer results are not fully reproducible because the inversion model is trained on an internal 156-hour commercial recording dataset that is not released. The final audio quality reported in the subjective listening test (Table 5.2) comes from the complete three-stage pipeline, so the unreleased inversion training data is a load-bearing component of the Chapter 5 comparison. I ask the authors to release the dataset or a suitable substitute, provide a released pretrained inversion checkpoint with its provenance, or clearly state this dependency as a reproducibility limitation.
minor comments (3)
  1. [Section 3.4.2, Table 3.2] The subjective listening test in Chapter 3 uses only 9 participants recruited from the authors' social networks, and the overall MOS confidence intervals overlap substantially (MMT 3.33±0.47, REMI+ 3.77±0.41); the 'comparable performance' wording should be tempered or supplemented with a significance test.
  2. [Sections 1.3, 6.1, and 7.1] Several passages assert 'the first' model for a given task without a systematic prior-art search; these priority claims should be softened to 'to our knowledge' or supported by a dedicated related-work survey.
  3. [Sections 3.3.1, 4.3, 6.1, and 6.4.3] There are minor typos and grammatical errors, including 'propsoed' (Section 3.3.1), 'assisstive' (Section 4.3), 'an our demo website' (Section 6.1), and 'we mix the an audio sample' (Section 6.4.3); these should be corrected before publication.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dissertation's core claims are empirically evaluated against external baselines and its acknowledged limitations are not circular reductions.

full rationale

No circularity found. The load-bearing technical claim in Chapter 6 is that CLIPSep learns text-queried separation from unlabeled videos by using the pretrained CLIP image-text embedding as a bridge between audio and text. The defining equations (6.1) and (6.2)-(6.6) train image-conditioned mask predictors with self-supervised mixture losses; the text query at test time is encoded by the frozen external CLIP model, whose stated assumptions do not include the separation result, so no prediction reduces to the paper's own fitted parameters by construction. Text-query SDRs in Tables 6.2 and 6.3 are measured on held-out mixtures and compared against supervised baselines such as LabelSep, BERTSep, and CLIPSep-Text. The paper also explicitly acknowledges the modality-transfer gap and its cause in Section 6.5 ('it still requires a sufficient audio-visual correspondence for training video'), and Appendix 6.H reports a negative zero-shot ESC-50 result (mean SDR 1.07 dB vs 0.00 dB for the mixture), which is an honest limitation statement rather than a circular step. Chapters 2-5 are reprints of the author's prior publications with external comparisons (e.g., MMM, REMI+, Hifi-GAN); the self-citations scattered through the introduction are provenance statements, not load-bearing mathematical premises. There is no self-authored uniqueness theorem invoked to force a choice, and no fitted parameter is renamed as a prediction. The absence of a CLIPSep-NIT ESC-50 result is a missing-evidence concern for the 'universal' claim, but it does not make any derivation equivalent to its inputs.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The dissertation as submitted makes no new derivation, so it contributes no free parameters of its own. The listed items are the hand-tuned settings and domain assumptions that the constituent papers' empirical claims depend on most. No new physical entities, particles, or mediators are introduced; model components such as CLIPSep's noise heads are architectural inventions, not independent entities with external falsifiable handles.

free parameters (3)
  • Noise regularization level gamma (CLIPSep-NIT) = 0.25
    Scanned over five values in Chapter 6.4.3 and chosen by validation SDR on MUSIC+ and VGGSound-Clean+; directly controls the balance between query heads and noise heads.
  • Noise regularization weight lambda (CLIPSep-NIT) = 0.1
    Set by hand for all NIT experiments in Chapter 6.3.2; no sensitivity analysis is reported for this weight.
  • Top-k sampling fraction per field (MMT) = 10% of possible outcomes per field
    Chosen by the authors for decoding in Chapter 3.3.2; affects generated sample quality and length.
assumptions (3)
  • domain assumption The pretrained CLIP text-image embedding space transfers to audio at test time (zero-shot modality transfer).
    CLIPSep and CLIPSonic train with image queries and evaluate with text queries; the transfer gap is measured but not removed (Chapters 6.4.1 and 6.4.2).
  • domain assumption DTW alignments between synthesized scores and recordings provide sufficient supervision for the alignment model.
    Deep Performer uses estimated alignments rather than manual annotations; errors in DTW would propagate into the synthesis stage (Chapter 5.3).
  • domain assumption Small listener panels from the authors' social networks represent target users.
    Subjective results in Chapters 3.4.2 and 5.4.3 use 9 and 15 participants respectively, limiting the generalization of perceptual claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generative AI for Music and Audio." pith.science (2026). https://pith.science/paper/XQK3XJKJ

@misc{pith2026241114627,
  author       = {Pith},
  title        = {Pith review of: Generative AI for Music and Audio},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XQK3XJKJ}},
  note         = {Machine review of arXiv:2411.14627}
}
read the original abstract

Generative AI has been transforming the way we interact with technology and consume content. In the next decade, AI technology will reshape how we create audio content in various media, including music, theater, films, games, podcasts, and short videos. In this dissertation, I introduce the three main directions of my research centered around generative AI for music and audio: 1) multitrack music generation, 2) assistive music creation tools, and 3) multimodal learning for audio and music. Through my research, I aim to answer the following two fundamental questions: 1) How can AI help professionals or amateurs create music and audio content? 2) Can AI learn to create music in a way similar to how humans learn music? My long-term goal is to lower the barrier of entry for music composition and democratize audio content creation

Figures

Figures reproduced from arXiv: 2411.14627 by the authors.

Figure 1.1
Figure 1.1. Looking back to the past, how music and technology interacts has always been a two-way process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 [PITH_FULL_IMAGE:figures/full_fig_p009_1_1.png] view at source ↗
Figure 1.1
Figure 1.1. Looking back to the past, how music and technology interacts has always been a two-way process. (Left) the violin-making industry grows with the classical music, and together create the golden age of classical music. (Right) the invention and development of synthesizers and drum machines helped popularize electronic music. Image sources (from left to right): 1) Mozart83, Public domain, via Wikimedia Commons, 2) Hild… view at source ↗
Figure 1.2
Figure 1.2. An overview of the three main directions of my research. music generation—advancing deep generative models for multitrack music, 2) assistive music creation tools—developing AI tools that can help musicians and amateurs create music, and 3) multimodal learning for audio and music—learning sound separation and synthesis from noisy videos. 1.1 Multitrack Music Generation Researchers have been working on automatic musi… view at source ↗
Figures from the paper (52 more)
Figure 2.1
Figure 2.1. Figure 2.1: An example of a learning-based music generation system. MusPy provides basic routines specific to music as well as interfaces to machine learning frameworks. that implements standard versions of such routines could save a great deal of time and effort and might lead …
Figure 2.2
Figure 2.2. Figure 2.2: System diagram of MusPy. The MusPy Music object at the center is the core element of MusPy. generation. In addition, MusPy provides several objective metrics for evaluating music generation systems. Related cross-dataset generalizability experiments (Donahue et al., …
Figure 2.3
Figure 2.3. Figure 2.3: Examples of (a) training data preparation and (b) result writing pipelines using MusPy [PITH_FULL_IMAGE:figures/full_fig_p029_2_3.png]
Figure 2.4
Figure 2.4. Figure 2.4: Two internal processing modes for iterating over a MusPy Dataset object [PITH_FULL_IMAGE:figures/full_fig_p031_2_4.png]
Figure 2.5
Figure 2.5. Figure 2.5: Length distributions for different datasets. 0 50 100 120 150 200 250 Tempo (qpm) wikifonia lmd hymnal hymnal-tune music21 music21jsb Dataset [PITH_FULL_IMAGE:figures/full_fig_p033_2_5.png]
Figure 2.6
Figure 2.6. Figure 2.6: Initial-tempo distributions for different datasets (those without tempo information are not presented). 15 [PITH_FULL_IMAGE:figures/full_fig_p033_2_6.png]
Figure 2.7
Figure 2.7. Figure 2.7: Key distributions for different datasets. The keys are sorted w.r.t. their frequencies in Lakh MIDI Dataset. of three key elements of a song—length, tempo and key using MusPy, with an eye to unveiling statistical differences among these datasets. First, [PITH_FULL_I…
Figure 2.8
Figure 2.8. Figure 2.8: Log-perplexities for different models on different datasets, sorted by the values for the LSTM model. 10 1 10 2 10 3 10 4 Size (hour) 6 8 10 12 14 16 18 Log-perplexity jsb music21jsb music21 hymnal nes maestro lmd nmd hymnal-tune essen wikifonia unified multipitch st…
Figure 2.9
Figure 2.9. Figure 2.9: Log-perplexities for the LSTM model versus dataset size in hours. Each point corresponds to a dataset. 18 [PITH_FULL_IMAGE:figures/full_fig_p036_2_9.png]
Figure 2
Figure 2. Figure 2: ) [PITH_FULL_IMAGE:figures/full_fig_p037_2.png]
Figure 2.10
Figure 2.10. Figure 2.10: Cross-dataset generalizability results. The values and colors represent the log-perplexities of a LSTM model trained on a specific dataset (row) and tested on another dataset (column). The datasets are sorted by the diagonal values, i.e., trained and tested on the s…
Figure 2
Figure 2. Figure 2: that combining datasets from different sources improves the generalizability of the model. [PITH_FULL_IMAGE:figures/full_fig_p039_2.png]
Figure 3.1
Figure 3.1. Figure 3.1: An example of the proposed representation—(a) an example of the first eight beats of a song in the orchestra dataset, shown as a multitrack piano roll, (b) the same song encoded by our proposed representation, where the grayed out zeros denote undefined values and (c…
Figure 3.2
Figure 3.2. Figure 3.2: Illustration of the proposed MMT model. inference speed and smaller memory footprint thanks to the reduced size of the self-attention matrix. However, since MMT predicts the six output fields nonautoregressively (i.e., independently), it cannot model the interdepende…
Figure 3.3
Figure 3.3. Figure 3.3: Mean relative attention gains (a) 𝛾˜ beat 𝑘 , (b) 𝛾˜ position 𝑘 and (c) 𝛾˜ pitch 𝑘 (see Section 3.4.4 for definitions) of a trained MMT model. Red and blue colors indicate positive and negative values, respectively. 32 [PITH_FULL_IMAGE:figures/full_fig_p050_3_3.png]
Figure 4.1
Figure 4.1. Figure 4.1: Proposed pipeline. By downmixing a symbolic multitrack into a single-track mixture, we acquire paired data of solo music and its instrumentation. We then use these paired data to train a part separation model that aims to infer the part label (e.g., one out of the fi…
Figure 4.2
Figure 4.2. Figure 4.2: Example of the Bach chorales dataset—Wer nur den lieben Gott läßt walten, BWV 434, measures 1–5. The LSTM model makes two errors for the bass, as indicated by the arrow. The BiLSTM model gives a perfect prediction. 4.7 Experiments 4.7.1 Implementation details We use …
Figure 4.3
Figure 4.3. Figure 4.3: Hard excerpt in the string quartets dataset—Beethoven’s String Quartet No. 11 in F minor, Op. 95, movement 1, measures 72–83. The tremolos of the first violin (measures 1–3 and 6–10), the double stops for the second violin, viola and cello (measures 2–3) and the over…
Figure 4.4
Figure 4.4. Figure 4.4: Hard excerpt in the game music dataset—Theme of Universe from Miracle Ropit’s Adventure in 2100. Both models perform poorly when there is a sequence of short notes crossing a single long note. 43 [PITH_FULL_IMAGE:figures/full_fig_p061_4_4.png]
Figure 4.5
Figure 4.5. Figure 4.5: Hard excerpt in the pop music dataset—Blame It On the Boogie by The Jacksons. The BiLSTM model correctly identify and separate the overlapping guitar melody and piano chords, while the LSTM model fails in this case. 4.7.3 Quantitative results We conduct an extensive …
Figure 4.6
Figure 4.6. Figure 4.6: Quando Quando Quando by Tony Renis—(a) original instrumentation and the versions produced by (b) the online LSTM model without entry hints and (b) the offline BiLSTM model with entry hints. The LSTM model assigns the chords to the guitar, the most common instrument i…
Figure 5.1
Figure 5.1. Figure 5.1: An overview of the proposed three-stage pipeline for score-to-audio music performance synthesis. which a musical instrument sounds. Third, it requires an understanding of different playing techniques and styles for a musical instrument. While most existing systems on…
Figure 5.2
Figure 5.2. Figure 5.2: An illustration of the proposed synthesis model. 53 [PITH_FULL_IMAGE:figures/full_fig_p071_5_2.png]
Figure 5.3
Figure 5.3. Figure 5.3: An example of the constant-Q spectrogram of the first 20 seconds of a violin recording and the estimated onsets (white dots) and durations (green lines). (a) (b) (c) [PITH_FULL_IMAGE:figures/full_fig_p073_5_3.png]
Figure 5.4
Figure 5.4. Figure 5.4: Examples of the alignments predicted by (a) the constant-tempo baseline model and (b) Deep Performer, our proposed model. (c) shows the input score. piano dataset as it is only available in this dataset. Since performer information is unavailable for the piano datase…
Figure 5.5
Figure 5.5. Figure 5.5: Examples of the mel spectrograms, in log scale, synthesized by our proposed model for (a) violin and (c) piano. (b) and (d) show the input scores for (a) and (c), respectively. 5.4.2 Qualitative and quantitative results We show in [PITH_FULL_IMAGE:figures/full_fig_p…
Figure 5.6
Figure 5.6. Figure 5.6: Examples of the mel spectrograms, in log scale, synthesized by (a) the baseline model, (b) our proposed synthesis model, and (d) our proposed synthesis model without the note-wise positional encoding. (c) and (e) show the waveforms for (b) and (d), respectively. (f) …
Figure 5.7
Figure 5.7. Figure 5.7: An illustration of the proposed alignment model [PITH_FULL_IMAGE:figures/full_fig_p079_5_7.png]
Figure 6
Figure 6. Figure 6: , during training, the image feature extracted from a video frame by the CLIP-image encoder [PITH_FULL_IMAGE:figures/full_fig_p083_6.png]
Figure 6.1
Figure 6.1. Figure 6.1: An illustration of modality transfer. corresponds to the image query in a self-supervised setting. Thanks to the properties of the CLIP model, which projects corresponding text and images to close embeddings, at test time we instead use the text feature obtained by t…
Figure 6.2
Figure 6.2. Figure 6.2: An illustration of the proposed CLIPSep model for 𝑛 = 2. During training, we mix audio from two videos and train the model to separate each audio source given the corresponding video frame as the query. At test time, we instead use a text query in the form of “a phot…
Figure 6.3
Figure 6.3. Figure 6.3: An illustration of the proposed CLIPSep-NIT model for 𝑛 = 2. Similar to CLIPSep, we train the model to separate each audio source given the corresponding query image during training and switch to using a text query at test time. The two predicted noise masks are inte…
Figure 6.4
Figure 6.4. Figure 6.4: Mean SDR and standard errors of the models trained and tested on different modalities. In addition, we also include the SOP model (Zhao et al., 2018) to investigate the quality of the query vectors as the CLIPSep and SOP models share the same network architecture exc…
Figure 6.5
Figure 6.5. Figure 6.5: Example results of the proposed CLIPSep-NIT model with 𝛾 = 0.25 on the MUSIC+ dataset. We mix the an audio sample (“violin” in this example) in the MUSIC dataset with an interference audio sample (“people sobbing” in this example) in the VGGSound dataset to create an…
Figure 6.6
Figure 6.6. Figure 6.6: Effects of the noise regularization level 𝛾 for the proposed CLIPSep-NIT model—mean SDR for the (a) MUSIC+ and (b) VGGSound-Clean+ evaluations, and (c) the total mean noise head activation, Í𝑛 𝑖=1 mean(𝑀ˆ 𝑁 𝑖 ), on the validation set. The shaded areas show standard e…
Figure 6
Figure 6. Figure 6: illustrates the inference pipeline for the proposed CLIPSep and CLIPSep-NIT models. [PITH_FULL_IMAGE:figures/full_fig_p099_6.png]
Figure 6.7
Figure 6.7. Figure 6.7: Inference pipeline of the proposed CLIPSep and CLIPSep-NIT models. Note that the mixture spectrogram and the masks are shown in log frequency scale, while the masked spectrogram is shown in linear frequency scale [PITH_FULL_IMAGE:figures/full_fig_p100_6_7.png]
Figure 6.8
Figure 6.8. Figure 6.8: An illustration of the PIT model for 𝑛 = 2. The two predicted masks are interchangeable during the loss computation. Since the two predicted masks are interchangeable, the PIT model requires an additional post-selection step to obtain the target sound. 6.D Permutatio…
Figure 6
Figure 6. Figure 6: illustrates the permuatation invariant training (PIT) model (Yu et al., 2017). The [PITH_FULL_IMAGE:figures/full_fig_p101_6.png]
Figure 6.9
Figure 6.9. Figure 6.9: Training and validation losses along the training progress on the VGGSound dataset. We also include the losses computed using text queries instead of image queries. The y-axes are intentionally set to the same range for easy comparison. Note that we do not use the va…
Figure 6.10
Figure 6.10. Figure 6.10: Test SDR along the training progress on the VGGSound-Clean dataset. The y-axes are intentionally set to the same range for easy comparison. 86 [PITH_FULL_IMAGE:figures/full_fig_p104_6_10.png]
Figure 6.11
Figure 6.11. Figure 6.11: Total mean noise head activation, Í𝑛 𝑖=1 mean(𝑀ˆ 𝑁 𝑖 ), on the validation set for the CLIPSep-NIT models along the training progress. 87 [PITH_FULL_IMAGE:figures/full_fig_p105_6_11.png]
Figure 6.12
Figure 6.12. Figure 6.12: Example results on the MUSIC+ dataset. Target source—“violin”; interference—“people sobbing”; query—“violin”. The spectrograms and masks are shown in the log and linear frequency scales, respectively. 88 [PITH_FULL_IMAGE:figures/full_fig_p106_6_12.png]
Figure 6.13
Figure 6.13. Figure 6.13: Example results on the MUSIC+ dataset. Target source—“acoustic guitar”; interference—“cheetah chirrup”, query—“acoustic guitar”. The spectrograms and masks are shown in the log and linear frequency scales, respectively. 89 [PITH_FULL_IMAGE:figures/full_fig_p107_6_13.png]
Figure 6.14
Figure 6.14. Figure 6.14: Example results on the VGGSound-Clean+ dataset. Target source—“cat growling”; interference— “railroad car train wagon”; query—“cat growling”. The spectrograms and masks are shown in the log and linear frequency scales, respectively. 90 [PITH_FULL_IMAGE:figures/full…
Figure 6.15
Figure 6.15. Figure 6.15: Example results on the VGGSound-Clean+ dataset. Target source—“electric grinder grinding”; interference—“vehicle horn car horn honking”; query—“electric grinder grinding”. The spectrograms and masks are shown in the log and linear frequency scales, respectively. Not…
Figure 6.16
Figure 6.16. Figure 6.16: Query robustness experiment on the MUSIC+ dataset. Target source—“acoustic guitar”; interference—“cheetah chirrup”. The spectrograms are shown in the log frequency scale. 92 [PITH_FULL_IMAGE:figures/full_fig_p110_6_16.png]
Figure 7.1
Figure 7.1. Figure 7.1: We learn the text-audio correspondence by leveraging the audio-visual correspondences in videos and the multimodal representation learned by pretrained language-vision models. et al., 2023b; Huang et al., 2023a; Agostinelli et al., 2023), whereas CLIPSonic learns tex…
Figure 7.2
Figure 7.2. Figure 7.2: Proposed CLIPSonic model. During training, CLIPSonic learns to synthesize the audio track of a video given the image in a video frame. At inference time, we feed a text query in the form of “a photo of [label]” to approach text-to-audio synthesis or use a pretrained …
Figure 7.3
Figure 7.3. Figure 7.3: Objective evaluation results on VGGSound and MUSIC. experiments study the quality of the generated audio (without evaluating its semantic grounding) while relevance experiments study the semantic correspondence with respect to the prompt (without evaluating its audio…
Figure 7
Figure 7. Figure 7: shows the results of the studied models as a function of the classifier-free [PITH_FULL_IMAGE:figures/full_fig_p118_7.png]
Figure 8.1
Figure 8.1. Figure 8.1: An overview of my future research directions. professional creative workflows in music and audio production software. I will seek collaborations with other faculty members in human-computer interaction to explore new creative interfaces for music and audio production…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 5 canonical work pages

  1. [1]

    TensorFlow: A system for large- scale machine learning

    Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng (2016). “TensorFlow: A system for large- scale machine learning.” USENIX Symposium on Operating Systems Design and Implementation (OSDI) (cited on pages 9, 12, 42). Yuzo Abe, Yuki Murakami, and Masanobu Miura (2012). “Automatic arrangement for the bass guitar in popular music using principle component ana...

    work page 2016

  2. [2]

    MusicLM: Generating Music From Text

    Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour , and Chris- tian Frank (2023). “MusicLM: Generating Music From Text.” arXiv preprint arXiv:2302.03917 (cited on pages 94, 95). Relja Arandjelović and Andrew Zisserman (2017a). “Look, listen and learn.” International Conference on Computer Vision (ICCV) (cited on page 79). ...

    arXiv 2023

  3. [3]

    Language Models are Few-Shot Learners

    Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei (2020). “Language Models are Few-Shot Learners.” Advances in Neural Information Processing Systems (NeurIPS) (cited on pages 27, 75). Emilios Cambouropoulos (2006). “‘Voice’ separation: theoretical, perceptual and computational per- spectives.” Internatio...

    arXiv 2020

  4. [4]

    CLIPSynth: Learning Text-to-audio Synthesis from Videos using CLIP and Diffusion Models

    Chen, Arpit Gupta, Tagyoung Chung, Jing Huang, Nanyun Peng, and Wenbo Zhao (2023c). “CLIPSynth: Learning Text-to-audio Synthesis from Videos using CLIP and Diffusion Models.” CVPR Workshop on Sight and Sound (WSS) (cited on pages 2, 5, 95). 115 Hao-Wen Dong, Naoya Takahashi, Yuki Mitsufuji, Julian McAuley, and Taylor Berg-Kirkpatrick (2023d). “CLIPSep: Le...

    arXiv 2023

  5. [5]

    AudioGen: Textually Guided Audio Generation

    Parikh, Yaniv Taigman, and Yossi Adi (2023). “AudioGen: Textually Guided Audio Generation.” International Conference on Learning Representations (ICLR) (cited on page 94). Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brebisson, Yoshua Bengio, and Aaron Courville (2019). “MelGAN: Generative Adve...

    work page 2023

  6. [6]

    WavJourney: Compositional Audio Creation with Large Language Models

    Kong, Mark D. Plumbley, and Wenwu Wang (2023b). “WavJourney: Compositional Audio Creation with Large Language Models.” arXiv preprint arXiv:2307.14335 (cited on page 109). Magenta (n.d.). url: https://magenta.tensorflow.org/ (cited on page 9). Rachel Manzelli, Vijay Thakkar, Ali Siahkamari, and Brian Kulis (2018). “Conditioning Deep Generative Raw Audio M...

    arXiv 2023

  7. [7]

    PyTorch: An Imperative Style, High-Performance Deep Learning Library

    Fang, Junjie Bai, and Soumith Chintala (2019). “PyTorch: An Imperative Style, High-Performance Deep Learning Library.” Advances in Neural Information Processing Systems (NeurIPS) (cited on pages 12, 82). Christine Payne (2019).MuseNet. OpenAI Blog. url: https://openai.com/blog/musenet/ (cited on pages 23, 24, 37). Karol J. Piczak (2015). “ESC: Dataset for...

    work page 2019

  8. [8]

    Learning Transferable Visual Models From Natural Language Supervision

    Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever (2021). “Learning Transferable Visual Models From Natural Language Supervision.” International Conference on Machine Learning (ICML) (cited on pages 65, 68, 69, 81, 94). Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever (2019). “Language ...

    arXiv 2021

Show all 13 references
  1. [9]

    DeepSinger: Singing Voice Synthesis with Data Mined From the Web

    Fast, Robust and Controllable Text to Speech.” Advances in Neural Information Processing Systems (NeurIPS) (cited on pages 52, 55, 62). Yi Ren, Xu Tan, Tao Qin, Jian Luan, Zhou Zhao, and Tie-Yan Liu (2020). “DeepSinger: Singing Voice Synthesis with Data Mined From the Web.”Con...

    2020

  2. [10]

    AVLnet: Learning Audio-Visual Language Representations from Instructional Videos

    Picheny, Antonio Torralba, and James Glass (2021). “AVLnet: Learning Audio-Visual Language Representations from Instructional Videos.” Annual Conference of the International Speech Communication Association (INTERSPEECH) (cited on page 103). Andrew Rouditchenko, Hang Zhao, Chu...

    2021 arXiv

  3. [11]

    LAION-5B: An open large-scale dataset for training next generation image-text models

    Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev (2022). “LAION-5B: An open large-scale dataset for training next generation image-text models...

    2022

  4. [12]

    Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions

    Chen, Yu Zhang, Yuxuan Wang, RJ Skerry-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, and Yonghui Wu (2018). “Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions.” IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) (cite...

    2018 arXiv

  5. [13]

    Tacotron: Towards End-to-End Speech Synthesis

    Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous (2017). “Tacotron: Towards End-to-End Speech Synthesis.” Annual Conference of the International Speech Communication Association (INTERSPEECH) (cited on page 51). Ziyu W...

    2017 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.