Pith. sign in

REVIEW 4 major objections 8 minor 61 references

Fx-Encoder++: Extracting Instrument-Wise Audio Effects Representations from Mixtures

T0 review · 4 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fx-Encoder++ extracts instrument-wise audio effects from full music mixes.

desk verdict First credible attempt at instrument-wise effects extraction from mixtures, with real gains at mixture level; the training-query provenance needs an explicit statement before the extractor claim is fully sealed. read the letter →

arxiv 2507.02273 v1 pith:VLI5JCKZ submitted 2025-07-03 cs.SD eess.AS

Yen-Tung Yeh , Junghyun Koo , Marco A. Martínez-Ramírez , Wei-Hsiang Liao , Yi-Hsuan Yang , Yuki Mitsufuji This is my paper
classification cs.SDeess.AS
keywords audioeffectsrepresentationcontrastivelearninginstrument-wiseextractionmusicmixturesCLAPqueriestext-guidedretrievalintelligentproductionparametermatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Fx-Encoder++, a model that reads audio effects (EQ, reverb, compression, and similar processing) separately for each instrument directly from a complete music mix. Its central claim is that a query-conditioned extractor can turn a mixture-level embedding of the whole track into an embedding of the effects applied to a single queried instrument, using either an audio clip or a text prompt such as "this is the sound of drums." If this works, intelligent music production systems such as automatic mixers could know how each stem was processed without running source separation first. The paper reports that the model beats previous effects-representation methods on mixture-level retrieval and, uniquely, succeeds at instrument-wise retrieval from the full mix.

What carries the argument

The extractor is a 3-layer MLP that takes two inputs: a mixture-level effects embedding from a PANN-based encoder and a query embedding from a pretrained CLAP model. The CLAP encoder maps both audio and text into a shared space, so the same extractor can be conditioned by audio during training and by text at inference. The instrument-aware contrastive loss places positive pairs as the same instrument under identical effects across different mixtures and negatives as the same instrument under different effects; it is combined with the mixture-level SimCLR-style loss in a weighted sum (λmix = 0.8, λinst = 0.2) with the instrument objective annealed in after a curriculum. Supporting machinery includes Fx-normalization to strip pre-existing effects from training audio, consistent instrumentation within a batch to block timbre shortcuts, and hand-crafted hard negatives that share content but differ in effects.

What would settle it

Take a trained Fx-Encoder++, give it a text query for an instrument that is present in the mix, and compare the extracted embedding against the ground-truth isolated stem's effects embedding. Then replace the query with a text label of an instrument that is absent (e.g., "piano" in a mix with only drums and bass) and check whether the extracted embedding still correlates strongly with the true effects of any present instrument; if it does, the extractor is ignoring the query and the reported text-query performance would not be attributable to query conditioning.

Watch

Extended reading notes

Core claim

Fx-Encoder++ claims to be the first model to extract instrument-wise audio effects representations from music mixtures without source separation. On the paper's own terms: given a mixture x and an instrument query q (an audio example or text description), the encoder E(x,q) produces an embedding of the effects applied to that instrument, while E(x,∅) produces a mixture-level effects embedding. Training uses a contrastive objective with pairs of mixtures that share the same effect chains but differ in musical content, and positive instrument-level pairs are formed from the same instrument type across those mixtures. Evaluated on audio effects retrieval on MUSDB and MedleyDB, the model consistently outperforms prior effects-specific encoders at the mixture level and attains non-trivial instrument-wise retrieval with both audio and text queries, surpassing pipelines that rely on universal source separation.

Load-bearing premise

The extractor's instrument-wise accuracy depends on CLAP query embeddings being a clean, content-independent proxy for "which instrument's effects to pull out" — if a training-time audio query came from the same segment the extractor sees in the mixture, the model could learn to copy content rather than isolate effects, and that shortcut would have to survive the switch to text queries at inference.

Editorial extensions

If this is right

  • Automatic mixing systems could receive per-stem effects estimates directly from a full mix, removing the need for separate source separation stages that introduce artifacts.
  • Users could query a mix with plain-text prompts like "this is the sound of bass" to retrieve which effects processing a particular instrument underwent.
  • The instrument-wise embeddings could serve as targets for effect-style transfer or preset matching on individual stems within a finished song.
  • Because the extractor works on the mixture embedding, it could be retrained or fine-tuned for different instrument taxonomies without changing the underlying encoder.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If text-query generalization holds beyond the tested four-instrument sets, the same design might extend to effect-aware source separation: the extractor's instrument-specific effects representation could guide a separator toward the queried instrument's processed signal.
  • The paper leaves open whether the extractor learns a genuine per-instrument decomposition of the effects embedding or a soft selection over effect dimensions; a probing experiment that masks out one effect type at a time would distinguish these.
  • A testable extension would be to train with audio queries drawn from a different segment than the mixture content; if performance holds, that would confirm the shortcut concern is not realized and strengthen the claimed content-independence of the conditioning signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper presents Fx-Encoder++, a contrastively trained encoder that produces both mixture-level and instrument-wise audio effects embeddings. A proposed 'extractor' module takes a mixture embedding E(M_i) and an instrument query (CLAP audio or text embedding) and outputs an instrument-specific effects embedding. Training uses MoisesDB with Fx-normalization, consistent instrument composition per batch, and hard negatives; the objective combines mixture-level and instrument-level SimCLR losses (Eq. 4). Evaluation on MUSDB18 and MedleyDB covers retrieval (R@K) and effects parameter matching (Ld), with comparisons to general-purpose and effects-specific baselines, including source-separation-based pipelines. The authors report consistent mixture-level retrieval gains and a novel instrument-wise extraction capability.

Significance. If the central claim holds, Fx-Encoder++ would be the first model to extract per-instrument effects representations directly from full music mixtures without source separation, which is highly relevant to automatic mixing and intelligent production. The paper's strengths include a clear problem definition, the use of real multitrack datasets for evaluation, the systematic comparison against separation-based pipelines, and the public release of code. However, the main claim depends on the extractor genuinely using the mixture embedding; the unspecified provenance of the training-time query leaves open a shortcut solution in which the query alone carries the effects information. The lack of uncertainty quantification in the reported metrics further tempers the strength of the comparative claims.

major comments (4)
  1. [3.2, Eq. (3)] The training-time query signal Q_m_i is not specified. If the CLAP embedding is computed from the same wet segment that later appears in mixture M_i, the instrument-wise contrastive loss in Eq. (3) can be minimized by mapping Q directly to the instrument-effects embedding while ignoring E(M_i) entirely; the reported 'extractor' would then be a query encoder, not a mixture extractor. The paper never states whether the audio query is derived from the same segment, a different segment, a different track, or a synthesized text description. The remark in Sec. 5.1 that audio queries have 'different content and Fx than the target' appears to describe the evaluation setup, not the training setup. Please specify the training-time query source explicitly and provide an ablation that removes E(M_i) from Eq. (2) or corrupts it (e.g., shuffling mixture embeddings across the batch) to demonstrate that the extractor relies on the mixture embedding rather than on the query alone.
  2. [3.3] The query-dropout mechanism is described as applying a high dropout rate (0.75 to 0.95) but it is not stated what is dropped (the query embedding itself, the CLAP token, or the entire query channel) nor whether text queries are seen during training. Since inference uses text queries in several experiments, the modality gap between audio and text CLAP embeddings is load-bearing for the E(x, q_text) results in Table 3. Please clarify the dropout operation and whether any text-conditioned samples are included in training; if not, report text-query performance as a zero-shot transfer result and discuss how the dropout rate was chosen.
  3. [Tables 2–3, Fig. 2] All retrieval scores are reported as point estimates with no error bars, confidence intervals, or multiple-seed runs. Given the small absolute differences between Fx-Encoder++ and AFx-Rep on several MedleyDB rows (e.g., Table 2, Mixture R@1: 5.6 vs 4.2) and the absence of variance information, it is not possible to determine which differences are reliable. Please add standard deviations across at least three seeds or bootstrap confidence intervals, and state the number of trials used for the Ld parameter-matching results.
  4. [4.2 and Abstract] The parameter matching results in Table 2 (Ld column) do not show consistent mixture-level improvement: VGGish wins on drums in MUSDB, AFx-Rep wins on vocals, and no model dominates. The conclusion acknowledges this, but the abstract's phrase 'outperforms previous approaches at mixture level' is too broad because it conflates retrieval and parameter matching. Please restrict the abstract and summary claims to 'audio effects retrieval' or explicitly qualify that mixture-level improvement is shown for retrieval while parameter matching remains mixed.
minor comments (8)
  1. [3.2, Eq. (2)] The extractor input combination is not specified; state whether Q_m_i is concatenated with E(M_i), multiplied, or otherwise combined before the MLP.
  2. [Table 3] The column headers list 'Target Instrument (Oracle)' and 'USS(m) / MSS(m)' but the table body mixes two numbers with slashes for some columns; add a caption that explains the slash notation and which rows apply to which protocol.
  3. [Figure 1] The text query path is drawn only for inference; indicate in the caption or main text whether text queries are used during training as well.
  4. [2.2] The sentence 'Tone Embedding [16] and OpenAMP [17] focused specifically on guitar tones but was limited to isolated guitar recordings' uses singular 'was' for two subjects; it should be 'were limited.'
  5. [3.1] The Fx probability scheduling reference [15] is not described; a one-sentence explanation of how the probability is scheduled would improve reproducibility.
  6. [5.1] The statement that 'the query audio has different content and Fx than the target' is directly relevant to the training-query provenance question; please move or expand this explanation into the method section where the query source is defined.
  7. [4.1] The evaluation framework mentions a candidate pool of 500, but it is not stated how the candidates are sampled (e.g., random effect configurations per instrument, or shared across instruments) nor how many queries are used per condition; please provide these details.
  8. [References] MoisesDB [46] is cited but the train/validation split used is not described; state the split and confirm that no evaluation tracks from MUSDB or MedleyDB appear in the training set.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the instrument-wise extractor claim is tested on held-out MUSDB/MedleyDB data against external baselines, and the cited prior work supplies components, not the conclusion.

full rationale

The paper's central claim — that a contrastively trained encoder plus a query-conditioned extractor can produce instrument-wise audio effects embeddings from mixtures — is not derived from a fitted formula or from the definitions of its own outputs. Eq. (2) defines the extractor output as z_m_i = extractor(Q_m_i, E(M_i)) and Eq. (3) is a standard contrastive loss over triplets; neither equation forces the result by construction. The evaluation (Tables 2 and 3) is external: retrieval and parameter-matching are run on held-out MUSDB18 and MedleyDB recordings against general-purpose (CLAP, PANN, VGGish) and effects-specific (FX-Encoder, AFx-Rep) baselines, including USS/MSS separation pipelines. Borrowing the SimCLR contrastive objective from FX-Encoder [15] and Fx-Normalization preprocessing from [24] are author-overlapping citations, but they supply training machinery rather than the instrument-wise extraction conclusion; no uniqueness theorem or prior claim is invoked to forbid alternatives. One ambiguity is the provenance of training audio queries in Sec. 3.2: if the CLAP query were the same wet segment that enters the mixture, Eq. (3) could in principle be minimized by reading effects from the query alone while ignoring E(M_i). However, the paper does not define it that way, Sec. 5.1 states that the audio queries used at evaluation have different content and Fx than the target, and the reported text-query results (E(x, q_text), Table 3) are comparable to audio-query results, which would be unlikely if the encoder had learned a query-only shortcut. This is an empirical/design ambiguity, not a by-construction equivalence, so per the hard rules it is not counted as circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical entities, forces, or latent dimensions are postulated; the extractor is a learned MLP component, not an invented entity. The central claim depends on several domain assumptions about preprocessing and query encoders, plus a small set of hand-chosen hyperparameters. These are reasonable for an empirical ML paper but are worth stating explicitly.

free parameters (4)
  • loss weighting lambda_mix / lambda_inst = 0.8 / 0.2 (final); 1.0 / 0.0 initially, linearly interpolated
    Hand-chosen objective balance that controls the trade-off between mixture-level and instrument-level contrastive losses.
  • contrastive temperature tau = 0.1
    Set following SimCLR and FX-Encoder; influences the hardness of the contrastive distribution.
  • query dropout rate = 0.75 to 0.95
    Applied to CLAP query embeddings during training to force generalization across audio and text modalities.
  • number of effects in chain k and instrument count = randomly sampled (1-4 instruments, 1-8 effects)
    The pipeline samples effect chains with random order, number, types, and parameters; these distributions affect task difficulty and model robustness.
assumptions (5)
  • domain assumption Fx-Normalization adequately removes pre-existing effects from MoisesDB stems so that the contrastive objective reflects only the applied effects.
    Section 3.1 relies on this to ensure identical effect chains make equal contributions across samples; if preprocessing is imperfect, the model may cluster on residual stem characteristics.
  • domain assumption CLAP embeddings provide an instrument-discriminative and robust query space, especially with heavy dropout applied.
    Section 3.2 uses CLAP for both audio and text queries, assuming the text-audio alignment is strong enough for the extractor to transfer from audio-conditioned training to text-conditioned inference.
  • domain assumption PANN is a sufficient backbone for encoding effects information from log-mel spectrograms.
    Section 3.3 selects the PANN architecture following prior effects work; the choice assumes that frame-level spectral features capture effect timbres.
  • domain assumption The effect processor implementations (dasp-pytorch, torchcomp, in-house IRs) are accurate enough that models trained on them transfer to MUSDB and MedleyDB.
    The training and evaluation pipelines both use these DSP implementations; any systematic inaccuracy in the effect models would be shared and could limit real-world transfer.
  • domain assumption Consistent instrument composition within a training batch forces the model to attend to effects rather than timbre, and this continues to hold at inference for arbitrary mixtures.
    Section 3.1 controls instrumentation per batch; the paper assumes that the learned representation generalizes beyond the batch-level control to mixtures with unseen instrument combinations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fx-Encoder++: Extracting Instrument-Wise Audio Effects Representations from Mixtures." pith.science (2026). https://pith.science/paper/VLI5JCKZ

@misc{pith2026250702273,
  author       = {Pith},
  title        = {Pith review of: Fx-Encoder++: Extracting Instrument-Wise Audio Effects Representations from Mixtures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VLI5JCKZ}},
  note         = {Machine review of arXiv:2507.02273}
}
read the original abstract

General-purpose audio representations have proven effective across diverse music information retrieval applications, yet their utility in intelligent music production remains limited by insufficient understanding of audio effects (Fx). Although previous approaches have emphasized audio effects analysis at the mixture level, this focus falls short for tasks demanding instrument-wise audio effects understanding, such as automatic mixing. In this work, we present Fx-Encoder++, a novel model designed to extract instrument-wise audio effects representations from music mixtures. Our approach leverages a contrastive learning framework and introduces an "extractor" mechanism that, when provided with instrument queries (audio or text), transforms mixture-level audio effects embeddings into instrument-wise audio effects embeddings. We evaluated our model across retrieval and audio effects parameter matching tasks, testing its performance across a diverse range of instruments. The results demonstrate that Fx-Encoder++ outperforms previous approaches at mixture level and show a novel ability to extract effects representation instrument-wise, addressing a critical capability gap in intelligent music production systems.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 54 canonical work pages

  1. [1]

    Fx- Encoder++: Extracting Instrument-Wise Audio Effects Representations from Mixtures

    INTRODUCTION Recent advances in deep learning have enabled significant progress in general-purpose audio representations [1–5], which have proven effective across diverse applications. However, they inadequately capture the nuanced charac- teristics of audio effects processing [6–8], since they prior- itize semantic content recognition over subtle audio e...

    arXiv 2025

  2. [2]

    VGGish [1] and PANN

    RELA TED WORKS 2.1 General-purpose Audio Representation General-purpose audio representations have emerged to support various downstream tasks. VGGish [1] and PANN

    work page

  3. [3]

    More recently, CLAP

    employ CNNs trained on AudioSet [27] for audio clas- sification and pattern recognition. More recently, CLAP

    work page

  4. [4]

    Neural audio compression models [4, 32, 33] rep- resent another category, utilizing V AEs to reconstruct per- ceptual features while minimizing bitrate

    uses a transformer-based architecture with contrastive learning to align audio and text, enabling applications in text-to-music generation [28, 29] and audio separation [30, 31]. Neural audio compression models [4, 32, 33] rep- resent another category, utilizing V AEs to reconstruct per- ceptual features while minimizing bitrate. Despite their success acr...

    work page

  5. [5]

    drums” and “bass

    RESULTS 5.1 Audio Effects Retrieval Mixture-level Retrieval . Table 2 shows Fx-Encoder++ substantially outperforms both general-purpose models and other effect-specific models on MUSDB dataset. Our model particularly excels with “drums” and “bass” . How- ever, performance on “vocals” is comparatively lower than other instruments (drops approximately 10%),...

    work page

  6. [6]

    drums+bass

    METHOD Our goal is to develop an encoder, denoted as E(x, q), that encodes an audio effects embedding from music mix- tures x. When conditioned with an instrument query q, the encoder extracts effects representations specific to that instrument within the mixture; without conditioning (i.e., q = ∅), the encoder produces representations that charac- terize...

    work page

  7. [7]

    this is the sound of {target instrument}

    EV ALUA TION METHOD 4.1 Audio Effects Retrieval To evaluate our model, we conduct audio effects re- trieval experiments using a controllable effects pipeline 1 https://github.com/SonyResearch/Fx-Encoder_ PlusPlus Type Model MUSDB18 [25] Drums Bass V ocals Other Mixture R@1 R@5 R@10 L d R@1 R@5 R@10 L d R@1 R@5 R@10 L d R@1 R@5 R@10 L d R@1 R@5 R@10 GP CLA...

    work page

  8. [8]

    Text2fx: Harnessing clap embeddings for text-guided audio ef- fects,

    A. Chu, P. O’Reilly, J. Barnett, and B. Pardo, “Text2fx: Harnessing clap embeddings for text-guided audio ef- fects,” in Proc. International Conference on Acoustics, Speech, and Signal Processing (ICASSP) , 2025

    work page 2025

Show all 61 references
  1. [9]

    Our approach outperforms existing meth- ods in audio effects retrieval at the mixture level while en- abling instrument-wise effects embedding extraction

    CONCLUSION We introduced Fx-Encoder++, the first model that ex- tracts instrument-wise audio effects information from mu- sic mixtures. Our approach outperforms existing meth- ods in audio effects retrieval at the mixture level while en- abling instrument-wise effects embeddin...

  2. [10]

    ACKNOWLEDGEMENTS Yen-Tung thanks National Science and Technology Coun- cil for supporting his PhD study

  3. [11]

    CNN architectures for large-scale audio classification,

    S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold et al. , “CNN architectures for large-scale audio classification,” in Proc. International Conference on Acoustics, Speech, and Signal Process- ing (ICAS...

    2017

  4. [12]

    PANNs: Large-scale pretrained au- dio neural networks for audio pattern recognition,

    Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumbley, “PANNs: Large-scale pretrained au- dio neural networks for audio pattern recognition,” IEEE/ACM Trans. Audio, Speech, Lang. Process. , vol. 28, 2020

    2020

  5. [13]

    Large-scale contrastive language- audio pretraining with feature fusion and keyword- to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language- audio pretraining with feature fusion and keyword- to-caption augmentation,” in Proc. International Con- ference on Acoustics, Speech, and Signal Processing (ICASSP), 2023

    2023

  6. [14]

    High fidelity neural audio compression,

    A. Défossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” Trans. Machine Learning Research, 2023

    2023

  7. [15]

    Beats: Audio pre-training with acoustic tokenizers,

    S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, and F. Wei, “Beats: Audio pre-training with acoustic tokenizers,” in Proc. ICML, 2022

    2022

  8. [16]

    ST-ITO: Con- trolling audio effects for style transfer with inference- time optimization,

    C. J. Steinmetz, S. Singh, M. Comunità, I. Ibnyahya, S. Yuan, E. Benetos, and J. D. Reiss, “ST-ITO: Con- trolling audio effects for style transfer with inference- time optimization,” in Proc. International Society for Music Information Retrieval (ISMIR) , 2024

    2024

  9. [17]

    Extraction Level

    focused specifically on guitar tones but was limited to isolated guitar recordings. Recently, AFx-Rep [6], a classification-based model designed for inference-time ef- fects optimization. Its training objective is to classify which single effect is applied between two given au...

  10. [18]

    Leveraging neural representations for audio manipulation,

    S. H. Hawley and C. J. Steinmetz, “Leveraging neural representations for audio manipulation,” in 154th Con- vention of the Audio Engineering Society , 2023

    2023

  11. [19]

    De Man, R

    B. De Man, R. Stables, and J. D. Reiss, Intelligent mu- sic production. Focal Press, 2019

    2019

  12. [20]

    Au- tomatic multitrack mixing with a differentiable mixing console of neural audio effects,

    C. J. Steinmetz, J. Pons, S. Pascual, and J. Serrà, “Au- tomatic multitrack mixing with a differentiable mixing console of neural audio effects,” in Proc. International Conference on Acoustics, Speech, and Signal Process- ing (ICASSP), 2021

    2021

  13. [21]

    A deep learning approach to intelligent drum mixing with the wave-u-net,

    M. Martinez Ramirez, D. Stoller, and D. Moffat, “A deep learning approach to intelligent drum mixing with the wave-u-net,” Journal of the Audio Engineering So- ciety, vol. 69, 2021

    2021

  14. [22]

    Style transfer of audio effects with differentiable signal pro- cessing,

    C. J. Steinmetz, N. J. Bryan, and J. D. Reiss, “Style transfer of audio effects with differentiable signal pro- cessing,” J. Audio Eng. Soc, vol. 70, 2022

    2022

  15. [23]

    End-to-end music re- mastering system using self-supervised and adversarial training,

    J. Koo, S. Paik, and K. Lee, “End-to-end music re- mastering system using self-supervised and adversarial training,” in Proc. International Conference on Acous- tics, Speech, and Signal Processing (ICASSP) , 2022

    2022

  16. [24]

    Diff-MST: Differentiable mixing style transfer,

    S. S. Vanka, C. Steinmetz, J.-B. Rolland, J. Reiss, and G. Fazekas, “Diff-MST: Differentiable mixing style transfer,” in Proc. International Society for Music In- formation Retrieval (ISMIR), 2024

    2024

  17. [25]

    Music mixing style transfer: A contrastive learning approach to disentan- gle audio effects,

    J. Koo, M. A. Martínez-Ramírez, W.-H. Liao, S. Uh- lich, K. Lee, and Y . Mitsufuji, “Music mixing style transfer: A contrastive learning approach to disentan- gle audio effects,” inProc. International Conference on Acoustics, Speech, and Signal Processing (ICASSP) , 2023

    2023

  18. [26]

    Towards zero-shot amplifier modeling: One-to-many amplifier modeling via tone embedding control,

    Y .-H. Chen, Y .-T. Yeh, Y .-C. Cheng, J.-T. Wu, Y .-H. Ho, J.-S. R. Jang, and Y .-H. Yang, “Towards zero-shot amplifier modeling: One-to-many amplifier modeling via tone embedding control,” inProc. International So- ciety for Music Information Retrieval (ISMIR) , 2024

    2024

  19. [27]

    Open-Amp: Synthetic data framework for audio effect foundation models,

    A. Wright, A. Carson, and L. Juvela, “Open-Amp: Synthetic data framework for audio effect foundation models,” in Proc. International Conference on Acous- tics, Speech, and Signal Processing (ICASSP) , 2025

    2025

  20. [28]

    Universal source separation with weakly labelled data,

    Q. Kong, K. Chen, H. Liu, X. Du, T. Berg-Kirkpatrick, S. Dubnov, and M. D. Plumbley, “Universal source separation with weakly labelled data,” arXiv preprint arXiv:2305.07447, 2023

    2023 arXiv

  21. [29]

    Mu- sic source separation in the waveform domain,

    A. Défossez, N. Usunier, L. Bottou, and F. Bach, “Mu- sic source separation in the waveform domain,” arXiv preprint arXiv:1911.13254, 2019

    1911 arXiv

  22. [30]

    Music separation en- hancement with generative modeling,

    N. Schaffer, B. Cogan, E. Manilow, M. Morrison, P. Seetharaman, and B. Pardo, “Music separation en- hancement with generative modeling,” in Proc. Inter- national Society for Music Information Retrieval (IS- MIR), 2022

    2022

  23. [31]

    Reverse engineering of a recording mix with differentiable digital signal pro- cessing,

    J. T. Colonel and J. Reiss, “Reverse engineering of a recording mix with differentiable digital signal pro- cessing,” The Journal of the Acoustical Society of America, vol. 150, no. 1, pp. 608–619, 2021

    2021

  24. [32]

    Search- ing for music mixing graphs: A pruning approach,

    S. Lee, M. A. Martínez-Ramírez, W.-H. Liao, S. Uh- lich, G. Fabbro, K. Lee, and Y . Mitsufuji, “Search- ing for music mixing graphs: A pruning approach,” in 27th International Conference on Digital Audio Effects (DAFx), 2024

    2024

  25. [33]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in Proc. ICML, 2020

    2020

  26. [34]

    Automatic music mixing with deep learning and out-of-domain data,

    M. A. Martínez-Ramírez, W.-H. Liao, G. Fabbro, S. Uhlich, C. Nagashima, and Y . Mitsufuji, “Automatic music mixing with deep learning and out-of-domain data,” in Proc. International Society for Music Infor- mation Retrieval (ISMIR), 2022

    2022

  27. [35]

    MUSDB18-HQ - an uncompressed version of MUSDB18,

    Z. Rafii, A. Liutkus, F.-R. Stöter, S. I. Mimilakis, and R. Bittner, “MUSDB18-HQ - an uncompressed version of MUSDB18,” Aug. 2019. [Online]. Available: https://doi.org/10.5281/zenodo.3338373

    2019 doi

  28. [36]

    MedleyDB: A multitrack dataset for annotation-intensive mir research

    R. M. Bittner, J. Salamon, M. Tierney, M. Mauch, C. Cannam, and J. P. Bello, “MedleyDB: A multitrack dataset for annotation-intensive mir research.” in Proc. International Society for Music Information Retrieval (ISMIR), 2014

    2014

  29. [37]

    Audio Set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio Set: An ontology and human-labeled dataset for audio events,” inProc. International Conference on Acoustics, Speech, and Signal Processing (ICASSP) , 2017

    2017

  30. [38]

    AudioLDM 2: Learning holistic audio generation with self-supervised pretraining,

    H. Liu, Y . Yuan, X. Liu, X. Mei, Q. Kong, Q. Tian, Y . Wang, W. Wang, Y . Wang, and M. D. Plumb- ley, “AudioLDM 2: Learning holistic audio generation with self-supervised pretraining,” IEEE/ACM Trans- actions on Audio, Speech, and Language Processing , 2024

    2024

  31. [39]

    Simple and control- lable music generation,

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Syn- naeve, Y . Adi, and A. Défossez, “Simple and control- lable music generation,” in Proc. NeurIPS, 2023

    2023

  32. [40]

    Sep- arate anything you describe,

    X. Liu, Q. Kong, Y . Zhao, H. Liu, Y . Yuan, Y . Liu, R. Xia, Y . Wang, M. D. Plumbley, and W. Wang, “Sep- arate anything you describe,”IEEE/ACM Trans. Audio, Speech, Lang. Process., 2024

    2024

  33. [41]

    Leveraging audio-only data for text-queried target sound extraction,

    K. Saijo, J. Ebbers, F. G. Germain, S. Khurana, G. Wiehern, and J. L. Roux, “Leveraging audio-only data for text-queried target sound extraction,” in Proc. International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2025

    2025

  34. [42]

    High-fidelity audio compression with im- proved rvqgan,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with im- proved rvqgan,” in Proc. NeurIPS, 2023

    2023

  35. [43]

    Music2latent: Consistency autoencoders for latent audio compres- sion,

    M. Pasini, S. Lattner, and G. Fazekas, “Music2latent: Consistency autoencoders for latent audio compres- sion,” in Proc. International Society for Music Infor- mation Retrieval (ISMIR), 2024

    2024

  36. [44]

    Reverb conversion of mixed vocal tracks using an end-to-end convolutional deep neural network,

    J. Koo, S. Paik, and K. Lee, “Reverb conversion of mixed vocal tracks using an end-to-end convolutional deep neural network,” in Proc. International Confer- ence on Acoustics, Speech, and Signal Processing (ICASSP), 2021

    2021

  37. [45]

    Automatic dj transitions with differentiable audio effects and genera- tive adversarial networks,

    B.-Y . Chen, W.-H. Hsu, W.-H. Liao, M. A. M. Ramírez, Y . Mitsufuji, and Y .-H. Yang, “Automatic dj transitions with differentiable audio effects and genera- tive adversarial networks,” in Proc. International Con- ference on Acoustics, Speech, and Signal Processing (ICASSP), 2022

    2022

  38. [46]

    Differentiable artifi- cial reverberation,

    S. Lee, H.-S. Choi, and K. Lee, “Differentiable artifi- cial reverberation,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 30, 2022

    2022

  39. [47]

    Can contrastive learning avoid shortcut solutions?

    J. Robinson, L. Sun, K. Yu, K. Batmanghelich, S. Jegelka, and S. Sra, “Can contrastive learning avoid shortcut solutions?” in Proc. NeurIPS, 2021

    2021

  40. [48]

    Contrastive learning with hard negative samples,

    J. Robinson, C.-Y . Chuang, S. Sra, and S. Jegelka, “Contrastive learning with hard negative samples,” in Proc. International Conference on Learning Represen- tations (ICLR), 2021

    2021

  41. [49]

    Film: Visual reasoning with a general conditioning layer,

    E. Perez, F. Strub, H. De Vries, V . Dumoulin, and A. Courville, “Film: Visual reasoning with a general conditioning layer,” in Proc. AAAI, 2018

    2018

  42. [50]

    Con- trastive localized language-image pre-training,

    H.-Y . Chen, Z. Lai, H. Zhang, X. Wang, M. Eichner, K. You, M. Cao, B. Zhang, Y . Yang, and Z. Gan, “Con- trastive localized language-image pre-training,” arXiv preprint arXiv:2410.02746, 2024

    2024 arXiv

  43. [51]

    dasp-pytorch,

    C. J. Steinmetz, “dasp-pytorch,” [Online] https:// github.com/csteinmetz1/dasp-pytorch/

  44. [52]

    Differentiable all-pole filters for time-varying audio systems,

    C.-Y . Yu, C. Mitcheltree, A. Carson, S. Bilbao, J. D. Reiss, and G. Fazekas, “Differentiable all-pole filters for time-varying audio systems,” in International Con- ference on Digital Audio Effects (DAFx) , 2024

    2024

  45. [53]

    pyloudnorm: A simple yet flexible loudness meter in python,

    C. J. Steinmetz and J. Reiss, “pyloudnorm: A simple yet flexible loudness meter in python,” in Audio Engi- neering Society Convention 150, 2021

    2021

  46. [54]

    Adam: A method for stochas- tic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochas- tic optimization,” in Proc. International Conference on Learning Representations (ICLR), 2014

    2014

  47. [55]

    Accurate, Large Minibatch SGD: Training im- agenet in 1 hour,

    P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y . Jia, and K. He, “Accurate, Large Minibatch SGD: Training im- agenet in 1 hour,” arXiv preprint arXiv:1706.02677 , 2017

    2017 arXiv

  48. [56]

    MoisesDB: A dataset for source separation beyond 4- stems,

    I. Pereira, F. Araújo, F. Korzeniowski, and R. V ogl, “MoisesDB: A dataset for source separation beyond 4- stems,” in Proc. International Society for Music Infor- mation Retrieval (ISMIR), 2023, pp. 619–626

    2023

  49. [57]

    Hybrid spectrogram and waveform source separation,

    A. Défossez, “Hybrid spectrogram and waveform source separation,” arXiv preprint arXiv:2111.03600 , 2021

    2021 arXiv

  50. [58]

    auraloss: Audio fo- cused loss functions in PyTorch,

    C. J. Steinmetz and J. D. Reiss, “auraloss: Audio fo- cused loss functions in PyTorch,” in Digital Music Research Network One-day Workshop (DMRN+15) , 2020

    2020

  51. [59]

    Deep learning for black-box modeling of audio ef- fects,

    M. A. Martínez Ramírez, E. Benetos, and J. D. Reiss, “Deep learning for black-box modeling of audio ef- fects,” Applied Sciences, vol. 10, 2020

    2020

  52. [60]

    Hyper recur- rent neural network: Condition mechanisms for black- box audio effect modeling,

    Y .-T. Yeh, W.-Y . Hsiao, and Y .-H. Yang, “Hyper recur- rent neural network: Condition mechanisms for black- box audio effect modeling,” in International Confer- ence on Digital Audio Effects (DAFx) , 2024

    2024

  53. [61]

    Efficient neural net- works for real-time modeling of analog dynamic range compression,

    C. J. Steinmetz and J. D. Reiss, “Efficient neural net- works for real-time modeling of analog dynamic range compression,” 152nd Convention of the Audio Engi- neering Society, 2021

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.