Pith. sign in

REVIEW 2 major objections 5 minor 45 references

Anysynth clones unseen instruments by attending to raw reference audio and MIDI, without compressing them into embeddings, and gets better with longer prompts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 06:45 UTC pith:3MRWKKAS

load-bearing objection Solid subfield method paper: embedding-free in-context DiT + asymmetric CFG actually move the numbers, with clean ablations and a real prompt-length scaling result. the 2 major comments →

arxiv 2607.11143 v1 pith:3MRWKKAS submitted 2026-07-13 cs.SD

Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance

Chong Jing , Junan Zhang , Jing Yang , Yulun Wu , Fan Fan , Zhizheng Wu This is my paper
classification cs.SD
keywords zero-shot instrument cloningin-context learningflow matchingDiffusion TransformerAsymmetric Hierarchical CFGtimbre transferMIDI-conditioned synthesis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Zero-shot instrument cloning needs a system that can take a short example of an instrument you have never trained on and then play any new MIDI sequence in that same sound. Prior systems squeeze the reference into a fixed embedding such as CLAP, which throws away the short transients and recording quirks that define real timbre. Anysynth instead feeds the uncompressed reference audio and its aligned MIDI straight into a Diffusion Transformer as an in-context prompt, so self-attention can look up acoustic detail while it generates the target. A second design choice, Asymmetric Hierarchical CFG, treats the target MIDI as the structural anchor and the reference only as a later acoustic refinement, avoiding the conflicts that arise when both conditions are guided symmetrically. The result is higher audio quality, closer timbre match, and better note accuracy than embedding-based and token-based baselines, plus a clear scaling law: longer reference prompts steadily improve fidelity, something embedding methods cannot do.

Core claim

Anysynth shows that zero-shot instrument cloning can be performed without any pre-trained timbre embedding: by casting the task as in-context flow matching on a Diffusion Transformer that sees the raw reference mel spectrogram and MIDI as a prefix, self-attention dynamically retrieves the fine-grained acoustic cues needed for faithful rendering, and the model uniquely improves as the prompt lengthens. Asymmetric Hierarchical CFG further raises both note accuracy and timbre fidelity by factoring guidance so that MIDI first anchors structure and the reference then supplies acoustic detail.

What carries the argument

In-context conditional flow matching plus Asymmetric Hierarchical CFG: the reference mel and MIDI are concatenated as an uncompressed prefix so DiT self-attention can retrieve acoustic detail at generation time; at inference the guided velocity is computed by first pulling toward the MIDI-compliant direction then refining with the reference, with MIDI scale larger than reference scale.

Load-bearing premise

That the transformer's self-attention over a raw reference-mel prefix will actually retrieve the fine acoustic cues needed for faithful cloning, and that treating MIDI as the structural parent of timbre guidance correctly models their dependence without leftover conflicts.

What would settle it

Hold the DiT weights fixed and replace the uncompressed reference prefix with a fixed CLAP vector (or scramble the reference-MIDI alignment); if PANNs, MERT and Onset F1 then fall to the levels of the embedding baselines and the prompt-length scaling curve flattens, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes ANYSYNTH, an embedding-free zero-shot instrument cloning system that renders a target MIDI sequence with the timbre of an unseen instrument given only a short aligned reference audio–MIDI pair. Instead of compressing the reference into a fixed-length embedding (e.g., CLAP), the method concatenates the uncompressed reference mel spectrogram as an in-context prefix with the target MIDI and trains a Diffusion Transformer under a conditional flow-matching objective (Sec. 2.2.1–2.2.2, Eq. 2). At inference it applies Asymmetric Hierarchical CFG (Eq. 3), which factorizes guidance by the chain rule so that MIDI acts as a structural anchor and the reference as a conditional acoustic refinement. Experiments on NSynth, Slakh, MAESTRO and GuitarSet report gains over TokenSynth, Control-Transfer-Diffusion and a CLAP-conditioned DiT twin on PANNs, MERT, Onset F1 and Audiobox aesthetics, together with prompt-length scaling that is absent from embedding baselines (Tables 1–3, Figs. 2–4).

Significance. If the results hold, the work supplies a concrete alternative to the dominant embedding-bottleneck design for zero-shot instrument cloning and demonstrates that in-context attention over raw reference audio can recover fine-grained acoustic detail while still following a new note sequence. The prompt-length scaling observation and the asymmetric CFG ablation are useful empirical findings for the broader community working on controllable music synthesis. Strengths include a clean same-backbone ablation isolating in-context conditioning from architecture (ANYSYNTH vs ANYSYNTH-CLAP), a controlled three-way CFG comparison on a single checkpoint (Table 3), and evaluation that spans both synthetic and real-recorded data (MAESTRO, GuitarSet). The contribution is incremental rather than foundational, but it is well-scoped and of clear interest to ISMIR-style venues.

major comments (2)
  1. Evaluation is restricted to 5 s target clips (Sec. 3.1.2) and three instrument families (keyboard, guitar, bass). The central claim of faithful zero-shot cloning and of prompt-length scaling therefore rests on a narrow regime; longer targets or additional families (strings, winds, percussion) could expose attention or memory limits that the current tables do not address. At minimum the paper should either extend the evaluation or explicitly bound the claim to short monophonic/homophonic segments of these families.
  2. Tables 1–3 report only point estimates with no error bars, multiple seeds, or statistical tests. Given that the strongest claims rest on relatively modest absolute gains (e.g., PANNs 0.820 vs 0.779 on NSynth, Onset F1 differences of a few points), the absence of uncertainty quantification leaves open whether the ranking is stable. Reporting standard deviations over seeds or bootstrap intervals would make the superiority claim load-bearing rather than suggestive.
minor comments (5)
  1. Eq. (1) multiplies the onset embedding into the note embedding; a short justification or ablation of the gating choice would help readers understand why additive fusion was rejected.
  2. Figure 4 and Table 2 show essentially flat performance from 8 s to 15 s prompts; the discussion in Sec. 3.3.2 correctly notes diminishing returns but could more clearly state whether this is an artifact of the 5 s target length.
  3. Typographical inconsistencies appear throughout (e.g., “ANYSYNTH” vs “AnySynth”, “V ocos”, “V oicebox”, missing spaces after periods). A careful copy-edit pass is needed.
  4. The demo URL is given; releasing code or model weights would strengthen reproducibility claims that currently rest only on the reported experimental protocol.
  5. Related-work coverage of recent neural-codec and flow-matching music systems is adequate but could briefly situate the onset-gated MIDI encoding relative to prior piano-roll augmentations.

Circularity Check

0 steps flagged

No significant circularity; self-contained empirical ML paper with external metrics and ablations.

full rationale

The paper is a standard empirical systems contribution in neural audio synthesis. Its central claims (superior timbre fidelity, melody adherence, and prompt-length scaling via uncompressed in-context DiT conditioning plus Asymmetric Hierarchical CFG) are supported by held-out evaluations against third-party baselines (TokenSynth, CTD) and an internal CLAP-conditioned twin, using external embedders/transcribers (PANNs, MERT, YourMT3+, Audiobox Aesthetics) and ground-truth audio. The CFG formulation (Eq. 3) is an explicit chain-rule rewrite of the joint score, not a definition that forces the reported metrics; scales αm/αr and drop rates are ordinary hyperparameters. Ablations (Tables 1–3, Fig. 4) isolate the design choices without reducing any “prediction” to a fitted identity or self-citation chain. No uniqueness theorems, ansatz smuggling, or self-definitional loops appear. Score 0 is therefore the correct outcome.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central performance claims rest on standard flow-matching and DiT machinery plus several design choices (onset-gated MIDI, in-context mel prefix, asymmetric CFG scales, restricted instrument families, proxy metrics). No new physical entities are postulated; free parameters are ordinary ML hyperparameters whose values affect reported numbers.

free parameters (4)
  • Asymmetric CFG scales (αm, αr) = (2.0, 1.0)
    Chosen as (2.0, 1.0) with αm>αr; ablation table shows results depend on this choice versus symmetric or combined CFG.
  • Condition dropout rate for CFG training = 0.3
    Independent drop of reference mel and MIDI at rate 0.3; controls how well unconditional and partial-condition scores are learned.
  • DiT capacity and training schedule = 481.30M / 200k steps
    Hidden 1024, depth 25, 16 heads, 481 M params, 200 k steps, LR 1e-4 with 32 k warmup; capacity and optimization choices that enable the reported scores.
  • Flow-matching path parameter σ = σ→0
    Optimal-transport path with σ→0; standard but still a modeling choice that defines the training target velocity.
axioms (5)
  • ad hoc to paper Self-attention over an uncompressed reference-mel prefix can retrieve fine-grained acoustic details (transients, formants, room) without a fixed-length embedding bottleneck.
    Core design premise of Sec. 2.2.1; not independently proven, only supported by the CLAP ablation.
  • ad hoc to paper Joint posterior over MIDI and reference timbre factorizes by the chain rule p(cr|cm,xt)p(cm|xt), justifying asymmetric hierarchical CFG (Eq. 3).
    Inspired by InstructPix2Pix; treated as the natural semantic-acoustic hierarchy of music (Sec. 2.2.3).
  • domain assumption Standard continuous-time flow matching with OT path and masked L1 velocity loss is a valid generative objective for mel spectrograms.
    Taken from the flow-matching / Voicebox / F5-TTS literature and used without re-derivation (Eq. 2).
  • domain assumption PANNs cosine, MERT patch cosine, Onset F1 (via YourMT3+), and Audiobox CE/PQ are adequate proxies for timbre fidelity, melody adherence, and perceptual quality.
    Evaluation section relies on these external models as ground truth for the claimed outperformance.
  • domain assumption Training on rendered NSynth+Slakh keyboard/guitar/bass and evaluating on MAESTRO/GuitarSet constitutes a meaningful zero-shot instrument-cloning test.
    Instrument-family restriction and synthetic-to-real transfer are assumed sufficient for the generalization claims.
invented entities (2)
  • Asymmetric Hierarchical CFG no independent evidence
    purpose: Decouple MIDI (semantic) and reference-timbre (acoustic) guidance to avoid gradient conflicts and improve note accuracy plus timbre fidelity.
    New inference-time formulation (Eq. 3) specific to this paper; independent evidence is only the internal ablation table.
  • Onset-gated MIDI embedding (en + eo ⊙ en) no independent evidence
    purpose: Sharpen note boundaries so rapidly repeated pitches are not merged into sustained events.
    Ad-hoc encoding design in Sec. 2.2.1; no external validation beyond the overall system results.

pith-pipeline@v1.1.0-grok45 · 16671 in / 3665 out tokens · 40432 ms · 2026-07-14T06:45:02.822359+00:00 · methodology

0 comments
read the original abstract

Zero-shot instrument cloning aims to render an arbitrary [Target MIDI] sequence with the acoustic identity of an unseen instrument given only a short [Reference Audio, Reference MIDI] pair. Existing methods rely on pre-trained embeddings (e.g., CLAP) that compress the reference audio into a fixed-length vector, discarding fine-grained acoustic cues essential for faithful timbre reconstruction. We present Anysynth, an embedding-free neural synthesizer based on in-context flow matching. By conditioning a Diffusion Transformer (DiT) directly on the uncompressed reference audio and target MIDI, our model allows self-attention to dynamically retrieve acoustic details at generation time. Experiments show that \tool outperforms embedding-based and auto-regressive baselines in audio quality, timbre similarity, and melody adherence. Notably, the model exhibits prompt-length scaling: longer reference prompts yield steadily better timbre fidelity, a property absent in embedding-based systems. To optimize controllability, we further propose Asymmetric Hierarchical CFG, which structurally decouples MIDI and reference-timbre guidance based on their natural semantic-acoustic dependency. This asymmetric formulation avoids gradient conflicts and improves both note accuracy and timbre fidelity, pushing the boundary of expressive, zero-shot instrument cloning. Demo audios are available at https://anysynth-demo.github.io/

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 12 linked inside Pith

  1. [1]

    An r&b electric piano solo

    INTRODUCTION Recent autoregressive and diffusion-based models have sub- stantially improved music audio generation [1–12]. As syn- thesis quality increases, the next challenge is fine-grained controllability: rather than generating plausible audio in coarse granularity (e.g., conditioned on text like"An r&b electric piano solo."), musicians require system...

    Pith/arXiv arXiv 2026

  2. [2]

    ANYSYNTHFRAMEWORK 2.1 Task Formulation Let xref be a short reference audio clip of P frames, mref ∈R P×K its time-aligned MIDI piano roll ( K=128 pitch bins), and mtar ∈R L×K a target MIDI sequence of L frames to be rendered. Zero-shot instrument cloning asks for target audio ˆxtar that (i) follows the pitch, timing, and dynamics prescribed by mtar, and (...

    2026

  3. [3]

    semantic-then- acoustic

    EXPERIMENTS 3.1 Experimental Setup 3.1.1 Implementation Details Training DatasetsWe construct our training corpus by combining and rendering data from two primary sources: NSynth and Slakh. Here we consider three primary instru- ment classes–keyboard, guitar and bass. Proceedings of the 27th ISMIR Conference, Abu Dhabi, UAE, November 08–12, 2026 • NSynth[...

    arXiv 2026

  4. [4]

    By directly conditioning a Diffusion Trans- former on uncompressed reference audio and target MIDI, ANYSYNTHbypasses the semantic bottleneck of traditional embedding methods

    CONCLUSION In this paper, we introduced ANYSYNTH, reformulat- ing zero-shot instrument cloning as an in-context learn- ing problem. By directly conditioning a Diffusion Trans- former on uncompressed reference audio and target MIDI, ANYSYNTHbypasses the semantic bottleneck of traditional embedding methods. Experiments demonstrate it signif- icantly outperf...

  5. [5]

    All core scientific ideas, novel algorithmic designs, experimen- tal methodologies, and data analyses are entirely the work of human

    AI USAGE STATEMENT Generative AI tools are employed for language polishing, structural refinement of the paper writing, and assistance with routine coding tasks (e.g., standard utility scripts). All core scientific ideas, novel algorithmic designs, experimen- tal methodologies, and data analyses are entirely the work of human. The authors have reviewed al...

  6. [6]

    Simple and control- lable music generation,

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Syn- naeve, Y . Adi, and A. Défossez, “Simple and control- lable music generation,”Advances in neural informa- tion processing systems, vol. 36, pp. 47 704–47 720, 2023

    2023

  7. [7]

    Musiclm: Generat- ing music from text,

    A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, Proceedings of the 27th ISMIR Conference, Abu Dhabi, UAE, November 08–12, 2026 TokenSynth CTD AnySynth (CLAP) AnySynth 3 8 15 Prompt length (s) 0.750 0.800 0.850PANNs ↑ (a) PANNs 3 8 15 Prompt length (s) 0.300 0.350 0.400 0.450 0.500MERT ↑ (b) MERT 3 8 15 Prompt...

    Pith/arXiv arXiv 2026

  8. [8]

    Noise2music: Text-conditioned music generation with diffusion models,

    Q. Huang, D. S. Park, T. Wang, T. I. Denk, A. Ly, N. Chen, Z. Zhang, Z. Zhang, J. Yu, C. Franket al., “Noise2music: Text-conditioned music generation with diffusion models,”arXiv preprint arXiv:2302.03917, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Yue: Scaling open foundation models for long-form music genera- tion,

    R. Yuan, H. Lin, S. Guo, G. Zhang, J. Pan, Y . Zang, H. Liu, Y . Liang, W. Ma, X. Duet al., “Yue: Scaling open foundation models for long-form music genera- tion,”arXiv preprint arXiv:2503.08638, 2025

    arXiv 2025

  10. [10]

    Musicflow: Cascaded flow matching for text guided music generation,

    K. Prajwal, B. Shi, M. Le, A. Vyas, A. Tjandra, M. Luthra, B. Guo, H. Wang, T. Afouras, D. Kantet al., “Musicflow: Cascaded flow matching for text guided music generation,” inInternational Conference on Ma- chine Learning. PMLR, 2024, pp. 41 052–41 063

    2024

  11. [11]

    Vevo2: A unified and controllable framework for speech and singing voice generation,

    X. Zhang, J. Zhang, Y . Wang, C. Wang, Y . Chen, D. Jia, Z. Chen, and Z. Wu, “Vevo2: A unified and controllable framework for speech and singing voice generation,” IEEE Transactions on Audio, Speech and Language Processing, 2026

    2026

  12. [12]

    Anyaccomp: Generalizable accompaniment generation via quantized melodic bottleneck,

    J. Zhang, Y . Zhang, X. Zhang, and Z. Wu, “Anyaccomp: Generalizable accompaniment generation via quantized melodic bottleneck,”arXiv preprint arXiv:2509.14052, 2025

    arXiv 2025

  13. [13]

    Ace-step: A step towards music generation foundation model,

    J. Gong, S. Zhao, S. Wang, S. Xu, and J. Guo, “Ace-step: A step towards music generation foundation model,” arXiv preprint arXiv:2506.00045, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Ace-step 1.5: Pushing the bound- aries of open-source music generation,

    J. Gong, Y . Song, W. Zhao, S. Wang, S. Xu, J. Guo, and X. Yang, “Ace-step 1.5: Pushing the bound- aries of open-source music generation,”arXiv preprint arXiv:2602.00744, 2026

    arXiv 2026

  15. [15]

    Songbloom: Coherent song generation via interleaved autoregressive sketching and diffusion refinement,

    C. Yang, S. Wang, H. Chen, W. Tan, J. Yu, and H. Li, “Songbloom: Coherent song generation via interleaved autoregressive sketching and diffusion refinement,” in The Thirty-ninth Annual Conference on Neural Infor- mation Processing Systems, 2025

    2025

  16. [16]

    Levo: High- quality song generation with multi-preference align- ment,

    S. Lei, Y . Xu, H. Zhang, H. Chen, Y . Zhang, C. Yang, H. Zhu, S. Wang, Z. Wu, D. Yuet al., “Levo: High- quality song generation with multi-preference align- ment,” inThe Thirty-ninth Annual Conference on Neu- ral Information Processing Systems, 2025

    2025

  17. [17]

    Heartmula: A family of open sourced music foundation models,

    D. Yang, Y . Xie, Y . Yin, Z. Wang, X. Yi, G. Zhu, X. Weng, Z. Xiong, Y . Ma, D. Conget al., “Heartmula: A family of open sourced music foundation models,” arXiv preprint arXiv:2601.10547, 2026

    Pith/arXiv arXiv 2026

  18. [18]

    Com- bining audio control and style transfer using latent dif- fusion,

    N. Demerlé, P. Esling, G. Doras, and D. Genova, “Com- bining audio control and style transfer using latent dif- fusion,” inProc. ISMIR, 2024

    2024

  19. [19]

    Flowsynth: Instru- ment generation through distributional flow matching and test-time search,

    Q. Yang, R. Leistikow, and Y . Zang, “Flowsynth: Instru- ment generation through distributional flow matching and test-time search,”arXiv preprint arXiv:2510.21667, 2025

    arXiv 2025

  20. [20]

    Token- synth: A token-based neural synthesizer for instrument cloning and text-to-instrument,

    K. Kim, J. Koo, S. Lee, H. Joung, and K. Lee, “Token- synth: A token-based neural synthesizer for instrument cloning and text-to-instrument,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  21. [21]

    Neural codec language model for controllable timbre transfer in music synthesis,

    S. Liu, T. Liu, D. Dalakoti, A. Suresh, Y . Teng, X. Liu, A. Roy, R. Bhatia, D. Hatadi, and P. Ghuman, “Neural codec language model for controllable timbre transfer in music synthesis,” inPMLR, 2026, pp. 1–13

    2026

  22. [22]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” inICASSP 2023-2023 IEEE Interna- tional Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2023, pp. 1–5. Proceedings of the 27th ISMIR Conference, Abu Dh...

    2023

  23. [23]

    Mulan: A joint embedding of music audio and natural language,

    Q. Huang, A. Jansen, J. Lee, R. Ganti, J. Y . Li, and D. P. Ellis, “Mulan: A joint embedding of music audio and natural language,”arXiv preprint arXiv:2208.12415, 2022

    Pith/arXiv arXiv 2022

  24. [24]

    Neural codec lan- guage models are zero-shot text to speech synthesizers,

    S. Chen, C. Wang, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Liet al., “Neural codec lan- guage models are zero-shot text to speech synthesizers,” IEEE Transactions on Audio, Speech and Language Processing, vol. 33, pp. 705–718, 2025

    2025

  25. [25]

    MIDI-V ALLE: Improving expressive piano performance synthesis through neural codec lan- guage modelling,

    J. Tang, X. Wang, Z. Zhang, J. Yamagish, G. Wiggins, and G. Fazekas, “MIDI-V ALLE: Improving expressive piano performance synthesis through neural codec lan- guage modelling,” inProc. ISMIR, 2025

    2025

  26. [26]

    V oicebox: Text-guided multilingual universal speech generation at scale,

    M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V . Manohar, Y . Adi, J. Mahadeokar et al., “V oicebox: Text-guided multilingual universal speech generation at scale,”Advances in neural infor- mation processing systems, vol. 36, pp. 14 005–14 034, 2023

    2023

  27. [27]

    AnyEnhance: A unified generative model with prompt-guidance and self-critic for voice enhancement,

    J. Zhang, J. Yang, Z. Fang, Y . Wang, Z. Zhang, Z. Wang, F. Fan, and Z. Wu, “AnyEnhance: A unified generative model with prompt-guidance and self-critic for voice enhancement,”IEEE Transactions on Audio, Speech and Language Processing, 2025

    2025

  28. [28]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,

    Y . Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. JianZhao, K. Yu, and X. Chen, “F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,” inProc. ACL, 2025, pp. 6255–6271

    2025

  29. [29]

    Maskgct: Zero-shot text-to-speech with masked generative codec transformer,

    Y . Wang, H. Zhan, L. Liu, R. Zeng, H. Guo, J. Zheng, Q. Zhang, X. Zhang, S. Zhang, and Z. Wu, “Maskgct: Zero-shot text-to-speech with masked generative codec transformer,” inThe Thirteenth International Confer- ence on Learning Representations, 2025

    2025

  30. [30]

    Metis: A foundation speech generation model with masked generative pre-training,

    Y . Wang, J. Zheng, J. Zhang, X. Zhang, H. Liao, and Z. Wu, “Metis: A foundation speech generation model with masked generative pre-training,”arXiv preprint arXiv:2502.03128, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Compositional visual generation with composable dif- fusion models,

    N. Liu, S. Li, Y . Du, A. Torralba, and J. B. Tenenbaum, “Compositional visual generation with composable dif- fusion models,” inECCV. Springer, 2022, pp. 423– 439

    2022

  32. [32]

    Instruct- pix2pix: Learning to follow image editing instructions,

    T. Brooks, A. Holynski, and A. A. Efros, “Instruct- pix2pix: Learning to follow image editing instructions,” inProc. CVPR, 2023, pp. 18 392–18 402

    2023

  33. [33]

    Scalable diffusion models with transformers,

    W. Peebles and S. Xie, “Scalable diffusion models with transformers,” inProc. ICCV, 2023, pp. 4195–4205

    2023

  34. [34]

    V ocos: Closing the gap between time- domain and fourier-based neural vocoders for high- quality audio synthesis,

    H. Siuzdak, “V ocos: Closing the gap between time- domain and fourier-based neural vocoders for high- quality audio synthesis,” inICLR, 2024

    2024

  35. [35]

    Ro- former: Enhanced transformer with rotary position em- bedding,

    J. Su, M. Ahmed, Y . Lu, S. Pan, W. Bo, and Y . Liu, “Ro- former: Enhanced transformer with rotary position em- bedding,”Neurocomputing, vol. 568, p. 127063, 2024

    2024

  36. [36]

    Stemgen: A music generation model that listens,

    J. D. Parker, J. Spijkervet, K. Kosta, F. Yesiler, B. Kuznetsov, J.-C. Wang, M. Avent, J. Chen, and D. Le, “Stemgen: A music generation model that listens,” in Proc. ICASSP. IEEE, 2024, pp. 1116–1120

    2024

  37. [37]

    Neural audio synthesis of musical notes with wavenet autoencoders,

    J. Engel, C. Resnick, A. Roberts, S. Dieleman, D. Eck, K. Simonyan, and M. Norouzi, “Neural audio synthesis of musical notes with wavenet autoencoders,” 2017. [Online]. Available: https://arxiv.org/abs/1704.01279

    Pith/arXiv arXiv 2017

  38. [38]

    Cutting music source separation some slakh: A dataset to study the impact of training data quality and quantity,

    E. Manilow, G. Wichern, P. Seetharaman, and J. L. Roux, “Cutting music source separation some slakh: A dataset to study the impact of training data quality and quantity,” 2019. [Online]. Available: https://arxiv.org/abs/1909.08494

    Pith/arXiv arXiv 2019

  39. [39]

    Cosyvoice 2: Scalable streaming speech synthesis with large language models,

    Z. Du, Y . Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y . Yang, C. Gao, H. Wanget al., “Cosyvoice 2: Scalable streaming speech synthesis with large language models,” arXiv preprint arXiv:2412.10117, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Guitarset: A dataset for guitar transcription

    Q. Xi, R. M. Bittner, J. Pauwels, X. Ye, and J. P. Bello, “Guitarset: A dataset for guitar transcription.” inISMIR, 2018, pp. 453–460

    2018

  41. [41]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition,

    Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,”IEEE/ACM Transactions on Audio, Speech, and Language Process- ing, vol. 28, pp. 2880–2894, 2020

    2020

  42. [42]

    Yourmt3+: Multi-instrument music transcription with enhanced transformer architectures and cross-dataset stem augmentation,

    S. Chang, E. Benetos, H. Kirchhoff, and S. Dixon, “Yourmt3+: Multi-instrument music transcription with enhanced transformer architectures and cross-dataset stem augmentation,” in2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2024, pp. 1–6

    2024

  43. [43]

    MERT: Acoustic music understanding model with large-scale self-supervised training,

    Y . Li, R. Yuan, G. Zhang, Y . Ma, X. Chen, H. Yin, C. Xiao, C. Lin, A. Ragni, E. Benetoset al., “MERT: Acoustic music understanding model with large-scale self-supervised training,” inICLR, 2024

    2024

  44. [44]

    Meta Audiobox Aesthetics: Unified automatic qual- ity assessment for speech, music, and sound,

    A. Tjandra, Y .-C. Wu, B. Guo, J. Hoffman, B. Ellis, A. Vyas, B. Shi, S. Chen, M. Le, N. Zacharovet al., “Meta Audiobox Aesthetics: Unified automatic qual- ity assessment for speech, music, and sound,”arXiv preprint arXiv:2502.05139, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Umap: Uniform manifold approximation and projection for dimension reduction,

    L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,”arXiv preprint arXiv:1802.03426, 2018

    Pith/arXiv arXiv 2018