Pith. sign in

REVIEW 5 major objections 5 minor 42 references

TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrument

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TokenSynth claims one decoder-only transformer can clone an instrument from a short audio clip, follow a text description, or blend both, all while playing polyphonic MIDI, without fine-tuning.

desk verdict Solid engineering contribution, but the CLAP-based evaluation needs an independent timbre metric and baselines before the zero-shot claims can be taken at face value. read the letter →

arxiv 2502.08939 v1 pith:C5U7Q2NX submitted 2025-02-13 cs.SD cs.AI

Kyungsu Kim , Junghyun Koo , Sungho Lee , Haesun Joung , Kyogu Lee This is my paper · ORCID
classification cs.SDcs.AI
keywords neuralsynthesizerinstrumentcloningtext-to-instrumentaudiocodecdecoder-onlytransformerCLAPembeddingMIDItokenizationzero-shottimbrecontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TokenSynth claims that a decoder-only transformer, trained once on pairs of MIDI scores, timbre embeddings, and audio tokens, can act as a polyphonic neural synthesizer that clones an unseen instrument's timbre from a short audio reference, follows a text description instead, or blends the two, all without fine-tuning. If true, one model replaces the usual pipeline of separate cloning and text-to-instrument systems and gives composers direct timbre control from language. The paper reports objective gains in timbral similarity and MIDI-following accuracy on synthetic instrument audio, and it releases code, weights, and audio demos.

What carries the argument

The load-bearing object is a decoder-only transformer that autoregressively produces audio tokens at every timestep for all codebook depths, using the delay-pattern arrangement from MusicGen to handle the multiple DAC codebooks. Inputs are MIDI tokens (onset, offset, pitch, velocity in MT3 style) and a projected CLAP embedding; the CLAP encoder is frozen and provides the shared audio-text timbre space. At inference, the model uses top-p sampling, optionally with first-note guidance, which applies classifier-free guidance at the onset time of the first note. A separate unconditional model and a transcription model, trained independently, support guidance and MIDI-following evaluation.

What would settle it

Take a held-out set of real instrument recordings (not the synthetic NSynth renderings), feed each with a MIDI score, and measure CLAP similarity to the reference and transcription F-score: if TokenSynth's scores drop close to chance or far below the synthetic-test numbers, the zero-shot generalization claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that an audio codec language model can be repurposed as a polyphonic synthesizer: the transformer predicts Descript Audio Codec tokens for the target audio, conditioned on a CLAP embedding that carries timbre and on MIDI tokens that carry the score. Because CLAP maps audio and text into one embedding space, the same conditioning channel accepts reference audio, a text prompt, or an interpolated embedding, so instrument cloning, text-to-instrument synthesis, and text-guided timbre manipulation are all the same mechanism. The paper introduces first-note guidance, applying classifier-free guidance only at the onset of the first note, and reports that this stabilizes synthesis by avoiding the noise that comes from extrapolating logits over silent frames. Evaluated on 10,000 synthetic per-instrument renderings, the model tracks MIDI with transcription F-scores up to 0.84 for cloning and 0.81 for text-to-instrument in the augmented version, and reaches CLAP scores near or above ground truth on cloning.

Load-bearing premise

The load-bearing premise is that a model trained and evaluated only on synthetic polyphonic renderings of isolated NSynth notes will also clone real, unprocessed instrument recordings in zero-shot use, which the paper never tests.

Editorial extensions

If this is right

  • Zero-shot cloning, text-to-instrument, and text-guided timbre interpolation are all achieved by one model without fine-tuning, because the conditioning is a CLAP embedding rather than a task-specific code.
  • Because the CLAP embedding is continuous, users can interpolate between audio and text embeddings, or between multiple references, to sculpt timbre gradually, including extrapolation beyond the given references.
  • First-note guidance implies that timbre can be locked in at the first onset; later frames inherit it, so guidance cost is concentrated at a single timestep.
  • Training on effect-augmented audio (EQ, distortion, reverb) improves MIDI-following accuracy in the reported metrics, suggesting that dataset augmentation is a direct lever for synthesis precision.
  • The 175M-parameter model with frozen codec and CLAP components keeps the approach computationally accessible, and the released code and weights make the system reproducible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same behavior transfers to real instrument recordings, the model could drop into a DAW as a MIDI-driven instrument whose timbre is swapped by dragging in any audio file or typing a description, without retraining.
  • The paper's own observation that CLAP embeddings underrepresent audio effects suggests a natural extension: add a separate effect-conditioning token or codec-derived effect embedding, which might close the wet-audio gap the authors observed.
  • One testable extension would probe interpolation with unseen text prompts (e.g., 'distorted electric guitar' or 'soft music box') and compare CLAP scores or listener ratings against single-modality conditioning, to see whether the shared embedding space stays semantically smooth.
  • Because the unconditional and transcription models are trained independently rather than shared, the design leaves room for scaling the main transformer alone, which could be tested by increasing model size while holding data fixed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. TokenSynth is a decoder-only transformer that generates audio tokens from MIDI tokens and a CLAP timbre embedding, using a pretrained DAC neural audio codec. The model is trained on a synthetic dataset of 9.53M clips rendered from NSynth instruments and Lakh MIDI data, with further augmentation. The paper claims zero-shot instrument cloning, text-to-instrument synthesis, and text-guided timbre manipulation without fine-tuning, and evaluates these claims with multi-scale spectral loss, CLAP score as a timbral similarity metric, and F-score for MIDI adherence. Code, model weights, and audio demos are made publicly available.

Significance. If the central claims hold, TokenSynth would be the first audio-codec language model used as a polyphonic neural synthesizer, offering zero-shot timbre control from audio or text in a single model. This is a plausible step beyond existing monophonic instrument cloning and sample-based text-to-instrument approaches. The paper also contributes a large synthetic training dataset, a 'First-Note Guidance' technique, and a public release of code and weights, all of which are positive. However, the evaluation evidence as presented is not yet sufficient to substantiate the claims: the main timbral similarity metric is partially circular, no comparison to existing baselines is provided, and all experiments are performed on synthetic renderings from the same pipeline used for training.

major comments (5)
  1. [Section IV-C and Section III-A.1] The CLAP score, used as the primary timbral similarity metric, is computed with the same pretrained CLAP encoder that supplies the conditioning embedding. Since the model is trained to produce audio whose CLAP embedding matches the conditioning embedding, high CLAP scores may reflect conditioning-matching rather than perceptual timbre fidelity. This concern is directly supported by Table I, where TokenSynth achieves a CLAP score of 0.860 in the 'True Dry' condition, exceeding the Ground Truth score of 0.798; a well-calibrated similarity metric should not be exceeded by a generated sample relative to the reference-target oracle. The authors should validate timbre fidelity with an independent metric (for example, a different audio embedding model, instrument classification accuracy on synthesized audio, or a formal listening test).
  2. [Section IV-A and Section V-A] The evaluation of zero-shot instrument cloning is entirely confined to the same synthetic NSynth/Lakh rendering pipeline used for training and testing. The claim of generalizing to real instrument recordings is therefore untested; a model trained and tested on synthetic audio may have learned artifacts of the rendering process rather than a general timbre representation. The authors should evaluate on real instrument recordings, either with existing MIDI annotations or via transcription, to support the zero-shot cloning claim.
  3. [Section V-A and Section V-B] The paper provides no quantitative comparison against existing baselines, despite citing InstrumentGen [25], [26] and DDSP-based cloning methods [16], [23] in the related work. For a claim of being the first polyphonic codec-language-model synthesizer, it is essential to compare against the closest available methods under matched conditions (for example, InstrumentGen for text-to-instrument and a DDSP variant for instrument cloning). Without such baselines, the absolute values of MSS loss, CLAP score, and F-score cannot establish the system's relative effectiveness.
  4. [Section V-B] The text-to-instrument results are weak and insufficiently analyzed. The CLAP scores (0.179 and 0.159) are far below those of the instrument-cloning task, and the paper attributes this to a modality gap, but provides no calibration or baseline to show that these values are above chance or comparable to existing text-to-instrument systems. Moreover, the F-score for TokenSynth (0.339) versus TokenSynth-Aug (0.8149) differs dramatically, and this discrepancy is not explained. Without a chance-level CLAP similarity estimate, a text-conditional baseline, or listening tests, the claim that text controls timbre is not substantiated.
  5. [Section III-C.1] First-Note Guidance, a novel contribution, is introduced with the assertion that applying classifier-free guidance only at the first noun onset is sufficient to control timbre throughout the synthesized audio. No ablation is provided comparing (i) no guidance, (ii) full guidance at every timestep, and (iii) the proposed first-note-only guidance, nor is there any quantitative analysis of why the technique works. Since this is presented as a key contribution, it should be empirically justified.
minor comments (5)
  1. [Section III-B] The paragraph describing the model architecture and training loss is duplicated verbatim; one copy should be removed.
  2. [Table II] The F-score for TokenSynth-Aug is reported as 0.8149, while other values in the table use three decimal places; this should be normalized to 0.814 for consistency.
  3. [Section V-A] The text states that 'Listening tests revealed' that TokenSynth-Aug had poor accuracy in applying audio effects, but no listening test methodology, number of listeners, or statistical results are reported. Either a formal listening test should be described or the claim should be removed.
  4. [Section IV-C and Section V] No error bars, confidence intervals, or significance tests are provided for any of the reported objective metrics; given the stochastic nature of top-p sampling, the authors should report variance over multiple samples or seeds.
  5. [Section V-A] The text refers to 'Table 1' in one place and 'Table I' elsewhere; the references should be unified.

Circularity Check

2 steps flagged · score 6.0 of 10

The central timbre-similarity metric (CLAP score) is computed with the same CLAP encoder that provides the conditioning embedding, making the cloning and text-to-instrument scores partially self-referential; independent MSS and F-score metrics prevent a higher score.

  1. self definitional [Section IV-C (Evaluation Metrics) with Section III-A.1 (Timbre Embedding) and Table I]
    "CLAP score measures timbral similarity between synthesized and target audio. For instrument cloning, it is computed as the cosine similarity between CLAP embeddings of the target and synthesized audio. | We utilize a pretrained CLAP model [29], a cross-modal representation learning model inspired by CLIP [35], to obtain timbre embeddings."

    The conditioning input e is the CLAP embedding of the reference audio, and the evaluation metric is the cosine similarity between CLAP embeddings of the synthesized and target audio. In the Ref=Tgt rows of Table I the reference audio is the target, so the conditioning vector equals CLAP(target); the model is trained to generate audio whose re-encoded CLAP embedding approaches e. The CLAP score therefore partly measures how well the model inverts its own conditioning vector, not an independent perceptual timbre match. The reported TokenSynth score (0.860) exceeding the ground-truth self-similarity (0.798) is consistent with this inflation. MSS loss and F-score are independent, so the circularity is partial.

  2. self definitional [Section V-B (Text-to-Instrument) with Section III-A.1 and Table II]
    "The CLAP score represents the cosine similarity between the CLAP embeddings of the synthesized audio and the reference text. | CLAP can generate embeddings from both audio and text inputs."

    For text-to-instrument, the text prompt is encoded with CLAP to form the conditioning embedding e_t, and the reported CLAP score is the cosine similarity between CLAP(synthesized audio) and the very same e_t. A model that learns to produce audio whose CLAP embedding matches the prompt will score well regardless of whether the timbre is perceptually faithful. No baseline, modality-gap calibration, or listening test is provided, so this metric cannot independently substantiate the text-control claim.

full rationale

The paper's central quantitative evidence for timbre cloning is the CLAP score, which shares its encoder with the conditioning signal; this makes the score partially self-referential. However, the model genuinely outputs DAC audio tokens, and the MSS loss and F-score provide independent (if synthetic-pipeline-only) grounding, so the circularity is partial rather than total. The self-citations for the dataset rendering pipeline (Kim et al. 2023) and effects augmentation (Koo et al. 2023) are normal method citations and are not load-bearing circularity. The paper's own limitation section does not disclose the encoder-sharing issue, and all cloning tests use the same synthetic NSynth/Lakh rendering pipeline as training, so real-recording generalization remains untested; that is a generality limitation rather than a circularity defect. Overall, one or more reported 'predictions' (timbral similarity scores) partially reduce to the conditioning representation by construction, giving a score of 6.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on several domain assumptions: the synthetic training distribution stands in for real instrument recordings, CLAP embeddings capture sufficient timbre, the MIDI tokenization is lossless enough for synthesis accuracy, and the independently trained unconditional model supports classifier-free guidance. These are plausible but not independently verified outside the paper's own setup.

free parameters (3)
  • First-note guidance scale gamma = 1.6
    Used for text-to-instrument experiments to strengthen the text condition; chosen empirically, no sweep reported.
  • Top-p sampling thresholds = 0.95 for cloning, 0.6 for text-to-instrument
    Sampling hyperparameters chosen by hand and applied differently per task.
  • Data augmentation probability and effect ranges = 0.5 probability; EQ, distortion, reverb with predefined parameter ranges
    Random effects applied to double the dataset; affects TokenSynth-Aug and its reported scores.
assumptions (5)
  • domain assumption MIDI tokenization with four token types (onset, offset, pitch, velocity) provides a sufficient representation for controlled audio synthesis.
    Adopted from MT3; the model relies on this representation capturing all necessary performance information (Section III-A2).
  • domain assumption CLAP embeddings contain sufficient timbre information for instrument cloning across unseen instruments.
    Supported by a 90.4% instrument classification accuracy, but the transfer to synthesis is assumed (Section III-A1).
  • domain assumption The delay pattern from MusicGen can be directly applied to DAC codebooks for multi-codebook prediction.
    Stated in Section III-A3 without ablations.
  • domain assumption The synthetic rendering pipeline (NSynth + Lakh MIDI) produces audio-MIDI pairs representative of real instrument performances.
    The dataset is generated synthetically; generalization to real recordings is assumed (Section IV-A).
  • ad hoc to paper Applying classifier-free guidance only at the first note onset is sufficient to control timbre throughout the audio.
    First-Note Guidance is introduced to stabilize synthesis; no ablation quantifies its contribution (Section III-C1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrument." pith.science (2026). https://pith.science/paper/C5U7Q2NX

@misc{pith2026250208939,
  author       = {Pith},
  title        = {Pith review of: TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrument},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C5U7Q2NX}},
  note         = {Machine review of arXiv:2502.08939}
}
read the original abstract

Recent advancements in neural audio codecs have enabled the use of tokenized audio representations in various audio generation tasks, such as text-to-speech, text-to-audio, and text-to-music generation. Leveraging this approach, we propose TokenSynth, a novel neural synthesizer that utilizes a decoder-only transformer to generate desired audio tokens from MIDI tokens and CLAP (Contrastive Language-Audio Pretraining) embedding, which has timbre-related information. Our model is capable of performing instrument cloning, text-to-instrument synthesis, and text-guided timbre manipulation without any fine-tuning. This flexibility enables diverse sound design and intuitive timbre control. We evaluated the quality of the synthesized audio, the timbral similarity between synthesized and target audio/text, and synthesis accuracy (i.e., how accurately it follows the input MIDI) using objective measures. TokenSynth demonstrates the potential of leveraging advanced neural audio codecs and transformers to create powerful and versatile neural synthesizers. The source code, model weights, and audio demos are available at: https://github.com/KyungsuKim42/tokensynth

Figures

Figures reproduced from arXiv: 2502.08939 by the authors.

Figure 1
Figure 1. (a) TokenSynth takes a timbre embedding and MIDI tokens as input. The timbre embedding is extracted from reference audio using [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 28 canonical work pages

  1. [25]

    Instrumentgen: Generating sample-based musical instruments from text,

    S. Nercessian and J. Imort, “Instrumentgen: Generating sample-based musical instruments from text,” arXiv preprint arXiv:2311.04339, 2023

    arXiv 2023

  2. [26]

    Generating Sample-Based Musical Instruments Using Neural Audio Codec Language Models

    S. Nercessian, J. Imort, N. Devis, and F. Blang, “Generating sample- based musical instruments using neural audio codec language models,” arXiv preprint arXiv:2407.15641 , 2024

    work page Pith review arXiv 2024

  3. [16]

    DDSP: differentiable digital signal processing,

    J. H. Engel, L. Hantrakul, C. Gu, and A. Roberts, “DDSP: differentiable digital signal processing,” in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 , OpenReview.net, 2020

    work page 2020

  4. [23]

    Neural music instrument cloning from few samples,

    N. Jonason and B. Sturm, “Neural music instrument cloning from few samples,” in 25th International Conference on Digital Audio Effects (DAFx20in22), Vienna, Austria, September 2022 , 2022

    work page 2022

  5. [1]

    Neural codec language models are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li, et al., “Neural codec language models are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111 , 2023

    arXiv 2023

  6. [2]

    Audiogen: Textually guided audio generation,

    F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. D ´efossez, J. Copet, D. Parikh, Y . Taigman, and Y . Adi, “Audiogen: Textually guided audio generation,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 , Open- Review.net, 2023

    work page 2023

  7. [3]

    Simple and controllable music generation,

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. D ´efossez, “Simple and controllable music generation,” Advances in Neural Information Processing Systems , vol. 36, 2024

    work page 2024

  8. [4]

    Musiclm: Generating music from text,

    A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, et al. , “Musiclm: Generating music from text,” arXiv preprint arXiv:2301.11325 , 2023

    arXiv 2023

Show all 42 references
  1. [5]

    Audioldm: Text-to-audio generation with latent diffusion models,

    H. Liu, Z. Chen, Y . Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “Audioldm: Text-to-audio generation with latent diffusion models,” in International Conference on Machine Learning , pp. 21450–21474, PMLR, 2023

    2023

  2. [6]

    Diffwave: A versatile diffusion model for audio synthesis,

    Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, “Diffwave: A versatile diffusion model for audio synthesis,” in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , OpenReview.net, 2021

    2021

  3. [7]

    Diffusion models: A comprehensive survey of methods and applications,

    L. Yang, Z. Zhang, Y . Song, S. Hong, R. Xu, Y . Zhao, W. Zhang, B. Cui, and M.-H. Yang, “Diffusion models: A comprehensive survey of methods and applications,” ACM Computing Surveys , vol. 56, no. 4, pp. 1–39, 2023

    2023

  4. [8]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  5. [9]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE/ACM Transac- tions on Audio, Speech, and Language Processing, vol. 30, pp. 495–507, 2021

    2021

  6. [10]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” Trans. Mach. Learn. Res. , vol. 2023, 2023

    2023

  7. [11]

    High- fidelity audio compression with improved rvqgan,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High- fidelity audio compression with improved rvqgan,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  8. [12]

    Music controlnet: Multiple time-varying controls for music generation,

    S. Wu, C. Donahue, S. Watanabe, and N. J. Bryan, “Music controlnet: Multiple time-varying controls for music generation,” IEEE ACM Trans. Audio Speech Lang. Process. , vol. 32, pp. 2692–2703, 2024

    2024

  9. [13]

    Smitin: Self-monitored inference-time intervention for generative music transformers,

    J. Koo, G. Wichern, F. G. Germain, S. Khurana, and J. L. Roux, “Smitin: Self-monitored inference-time intervention for generative music transformers,” arXiv preprint arXiv:2404.02252 , 2024

    2024 arXiv

  10. [14]

    Neural audio synthesis of musical notes with wavenet autoencoders,

    J. Engel, C. Resnick, A. Roberts, S. Dieleman, M. Norouzi, D. Eck, and K. Simonyan, “Neural audio synthesis of musical notes with wavenet autoencoders,” in International Conference on Machine Learn- ing, pp. 1068–1077, PMLR, 2017

    2017

  11. [15]

    Gansynth: Adversarial neural audio synthesis,

    J. H. Engel, K. K. Agrawal, S. Chen, I. Gulrajani, C. Donahue, and A. Roberts, “Gansynth: Adversarial neural audio synthesis,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , OpenReview.net, 2019

    2019

  12. [17]

    Sing: Symbol-to-instrument neural generator,

    A. D ´efossez, N. Zeghidour, N. Usunier, L. Bottou, and F. Bach, “Sing: Symbol-to-instrument neural generator,” Advances in neural information processing systems, vol. 31, 2018

    2018

  13. [18]

    Neural music synthesis for flexible timbre control,

    J. W. Kim, R. Bittner, A. Kumar, and J. P. Bello, “Neural music synthesis for flexible timbre control,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 176–180, IEEE, 2019

    2019

  14. [19]

    Neural waveshaping synthesis,

    B. Hayes, C. Saitis, and G. Fazekas, “Neural waveshaping synthesis,” in Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021 (J. H. Lee, A. Lerch, Z. Duan, J. Nam, P. Rao, P. van Kranenburg, and A. Sr...

    2021

  15. [20]

    Differen- tiable wavetable synthesis,

    S. Shan, L. Hantrakul, J. Chen, M. Avent, and D. Trevelyan, “Differen- tiable wavetable synthesis,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 4598–4602, IEEE, 2022

    2022

  16. [21]

    DDX7: differentiable FM synthesis of musical instrument sounds,

    F. Caspe, A. McPherson, and M. Sandler, “DDX7: differentiable FM synthesis of musical instrument sounds,” in Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Bengaluru, India, December 4-8, 2022 (P. Rao, H. A. Murthy, A. Sri...

    2022

  17. [22]

    A review of differentiable digital signal processing for music and speech synthesis,

    B. Hayes, J. Shier, G. Fazekas, A. McPherson, and C. Saitis, “A review of differentiable digital signal processing for music and speech synthesis,” Frontiers in Signal Processing , vol. 3, p. 1284100, 2024

    2024

  18. [24]

    Ganstrument: Adversarial in- strument sound synthesis with pitch-invariant instance conditioning,

    G. Narita, J. Shimizu, and T. Akama, “Ganstrument: Adversarial in- strument sound synthesis with pitch-invariant instance conditioning,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 1–5, IEEE, 2023

    2023

  19. [27]

    Raffel, Learning-based methods for comparing sequences, with appli- cations to audio-to-midi alignment and matching

    C. Raffel, Learning-based methods for comparing sequences, with appli- cations to audio-to-midi alignment and matching . Columbia University, 2016

    2016

  20. [28]

    Show me the instruments: Musical instrument retrieval from mixture audio,

    K. Kim, M. Park, H. Joung, Y . Chae, Y . Hong, S. Go, and K. Lee, “Show me the instruments: Musical instrument retrieval from mixture audio,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 1–5, IEEE, 2023

    2023

  21. [29]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (I...

    2023

  22. [30]

    Classifier-free diffusion guidance,

    J. Ho and T. Salimans, “Classifier-free diffusion guidance,” arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  23. [31]

    Neural discrete representation learning,

    A. Van Den Oord, O. Vinyals, et al. , “Neural discrete representation learning,” Advances in neural information processing systems , vol. 30, 2017

    2017

  24. [32]

    Advances in residual vector quantization: A review,

    C. F. Barnes, S. A. Rizvi, and N. M. Nasrabadi, “Advances in residual vector quantization: A review,” IEEE transactions on image processing , vol. 5, no. 2, pp. 226–262, 1996

    1996

  25. [33]

    Autoregressive image generation using residual quantization,

    D. Lee, C. Kim, S. Kim, M. Cho, and W.-S. Han, “Autoregressive image generation using residual quantization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 11523– 11532, 2022

    2022

  26. [34]

    The audio/mpeg Media Type

    M. Nilsson, “The audio/mpeg Media Type.” RFC 3003, Nov. 2000

    2000

  27. [35]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning , pp. 8748–8763, PMLR, 2021

    2021

  28. [36]

    MT3: multi-task multitrack music transcription,

    J. Gardner, I. Simon, E. Manilow, C. Hawthorne, and J. H. Engel, “MT3: multi-task multitrack music transcription,” in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 , OpenReview.net, 2022

    2022

  29. [37]

    Music mixing style transfer: A contrastive learning approach to disentangle audio effects,

    J. Koo, M. A. Mart ´ınez-Ram´ırez, W.-H. Liao, S. Uhlich, K. Lee, and Y . Mitsufuji, “Music mixing style transfer: A contrastive learning approach to disentangle audio effects,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASS...

    2023

  30. [38]

    Adam: A method for stochastic optimiza- tion,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimiza- tion,” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (Y . Bengio and Y . LeCun, eds.), 2015

    2015

  31. [39]

    PyTorch Lightning,

    W. Falcon and The PyTorch Lightning team, “PyTorch Lightning,” Mar. 2019

    2019

  32. [40]

    Mixed precision training,

    P. Micikevicius, S. Narang, J. Alben, G. F. Diamos, E. Elsen, D. Garc ´ıa, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu, “Mixed precision training,” in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3...

    2018

  33. [41]

    Mir eval: A transparent implementation of common mir metrics.,

    C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, D. P. Ellis, and C. C. Raffel, “Mir eval: A transparent implementation of common mir metrics.,” in ISMIR, vol. 10, p. 2014, 2014

    2014

  34. [42]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning,

    V . W. Liang, Y . Zhang, Y . Kwon, S. Yeung, and J. Y . Zou, “Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 17612–17625, 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.