pith. sign in

arxiv: 2509.15151 · v4 · pith:JE442YB2new · submitted 2025-09-18 · 💻 cs.SD · cs.AI

Exploring How Audio Effects Alter Emotion with Foundation Models

Stelios Katsis , Vassilis Lyberatos , Spyridon Kantarelis , Edmund Dervakos , Giorgos Stamou This is my paper

Pith reviewed 2026-05-22 12:29 UTC · model grok-4.3

classification 💻 cs.SD cs.AI
keywords audio effectsfoundation modelsemotionmusicprobing methodssound designaffective computingtimbre
0
0 comments X

The pith

Foundation models can be probed to reveal how audio effects reshape the emotional content of music.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper investigates whether large pretrained foundation models can serve as a tool to measure the emotional impact of common audio effects such as reverberation, distortion, modulation, and dynamic range compression. It applies various probing techniques to the models' internal embeddings to identify nonlinear associations between these processing choices and estimated affective responses. The goal is to move past simple low-level audio feature analysis toward a more systematic account of how sound design decisions influence listener perception in music.

Core claim

Foundation models pretrained on multimodal data encode rich associations between musical structure, timbre, and affective meaning. These associations provide a framework for examining the emotional consequences of audio effects by applying probing methods to the models' embeddings, which uncovers patterns linked to specific effects and allows evaluation of how robustly the models capture affective information.

What carries the argument

Embeddings extracted from foundation audio models, combined with probing methods that map those embeddings to emotion estimates.

If this is right

  • Specific audio effects become linked to repeatable shifts in estimated emotion through the model embeddings.
  • The approach supplies a quantitative way to compare the emotional influence of different production techniques.
  • Foundation models can be assessed for how well they preserve affective meaning after audio processing is applied.
  • Insights into music cognition and production practices follow from the uncovered patterns between effects and affect.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same probing pipeline could be tested on speech or environmental sound to see whether audio-effect emotion links generalize beyond music.
  • Comparing results across multiple foundation models would indicate which pretraining regimes best capture timbre-related affective changes.
  • Real-time versions of these probes might be integrated into digital audio workstations to suggest effects that target a chosen emotional outcome.

Load-bearing premise

The probing methods applied to the model embeddings can isolate emotional changes caused by the audio effects themselves rather than simply reflecting patterns already present in the pretraining data or in the choice of emotion labels.

What would settle it

Run the same probing pipeline on pairs of audio clips that differ only by the presence or absence of a given audio effect, then check whether the extracted emotion shifts match independent human listener ratings of those same pairs.

Figures

Figures reproduced from arXiv: 2509.15151 by Edmund Dervakos, Giorgos Stamou, Spyridon Kantarelis, Stelios Katsis, Vassilis Lyberatos.

Figure 1
Figure 1. Figure 1: Radar plots of emotion predictions for CLAP, Qwen, and MERT across three audio effects for the EMOPIA dataset. Each level in the plots depicts the distribution of emotions, normalised based on the greatest value in a given plot. To investigate and interpret musical emotion, we employed three state-of-the-art foundation models, each offering unique capabil￾ities. First, MERT-v1-330M [13] is a 330M-parameter… view at source ↗
Figure 2
Figure 2. Figure 2: UMAP visualization of foundation model embeddings for the EMOPIA dataset, showing trajectories generated for each input identity after applying audio FX with the intensity ranging from 1 to 10. detuning the signal with rate, depth, and feedback, producing sub￾tle modulation that can affect perceived pitch and texture. Phaser applies sweeping frequency modulation via rate, depth, and feed￾back, generating m… view at source ↗
Figure 3
Figure 3. Figure 3: UMAP visualization of foundation model embeddings for the witheFlow dataset, showing trajectories generated for each input after applying real-world scenario audio FX [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
read the original abstract

Audio effects (FX) such as reverberation, distortion, modulation, and dynamic range processing play a pivotal role in shaping emotional responses during music listening. While prior studies have examined links between low-level audio features and affective perception, the systematic impact of audio FX on emotion remains underexplored. This work investigates how foundation models - large-scale neural architectures pretrained on multimodal data - can be leveraged to analyze these effects. Such models encode rich associations between musical structure, timbre, and affective meaning, offering a powerful framework for probing the emotional consequences of sound design techniques. By applying various probing methods to embeddings from deep learning models, we examine the complex, nonlinear relationships between audio FX and estimated emotion, uncovering patterns tied to specific effects and evaluating the robustness of foundation audio models. Our findings aim to advance understanding of the perceptual impact of audio production practices, with implications for music cognition, performance, and affective computing.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper claims that foundation models pretrained on multimodal data encode rich associations between musical structure, timbre, and affective meaning, and that applying various probing methods to their embeddings can uncover complex nonlinear relationships between audio effects (reverb, distortion, modulation, dynamic range processing) and estimated emotion, thereby advancing understanding of the perceptual impact of audio production practices.

Significance. If the central claim holds after proper controls and validation, the work would offer a scalable framework for probing emotional consequences of sound design using large-scale pretrained models, with potential implications for music cognition, performance, and affective computing. The approach leverages existing foundation models rather than training new ones from scratch, which is a methodological strength if the isolation of FX effects can be demonstrated.

major comments (2)
  1. [Abstract] Abstract: The abstract describes an exploratory pipeline but supplies no quantitative results, validation details, or error analysis, making it impossible to assess whether observed differences in embeddings actually support the claimed relationships between specific FX and emotion.
  2. [Methods] Methods (probing pipeline description): The approach relies on pretrained models and probing rather than direct fitting, but lacks explicit controls (e.g., training-set decontamination, synthetic FX-free baselines, or direct comparison to human ratings on identical FX pairs) needed to separate FX-induced emotional changes from correlations already present in the pretraining corpus or biases in the downstream emotion estimator.
minor comments (1)
  1. [Abstract] The abstract uses the phrase 'various probing methods' without naming them or citing prior work on embedding probing for audio; this should be expanded with concrete examples and references in the main text.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback. Their comments help clarify how to better present our exploratory findings and strengthen the methodological description. We respond to each major comment below and indicate the revisions made to the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The abstract describes an exploratory pipeline but supplies no quantitative results, validation details, or error analysis, making it impossible to assess whether observed differences in embeddings actually support the claimed relationships between specific FX and emotion.

    Authors: We agree that the original abstract was too high-level. We have revised it to summarize the main quantitative outcomes of the probing experiments, including observed shifts in emotion embeddings for each FX category, brief validation metrics, and notes on the error analysis performed. These additions should allow readers to evaluate the support for the reported relationships. revision: yes

  2. Referee: [Methods] Methods (probing pipeline description): The approach relies on pretrained models and probing rather than direct fitting, but lacks explicit controls (e.g., training-set decontamination, synthetic FX-free baselines, or direct comparison to human ratings on identical FX pairs) needed to separate FX-induced emotional changes from correlations already present in the pretraining corpus or biases in the downstream emotion estimator.

    Authors: We acknowledge the importance of explicit controls. The revised methods section now details the synthetic FX-free baselines used for comparison and discusses potential biases in the downstream emotion estimator. Full training-set decontamination is not feasible for large foundation models; we have added a justification based on the broad, multimodal nature of pretraining. Direct human ratings on matched FX pairs were outside the scope of this work and would require a separate perceptual study, which we now note as a limitation. revision: partial

Circularity Check

0 steps flagged

No circularity detected in derivation or claims

full rationale

The paper describes an empirical probing study that applies standard methods to embeddings from externally pretrained foundation models. No equations, parameter fits, or derivations are shown that reduce by construction to the target emotion estimates or FX effects. The approach relies on off-the-shelf models and probing techniques rather than self-defining quantities or load-bearing self-citations. Central claims rest on the independent representational power of the foundation models, which are external to this work. This is a normal non-circular finding for an exploratory analysis paper.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The work rests on the assumption that foundation models already encode useful affective associations and that probing can surface them; no new free parameters, axioms, or invented entities are introduced in the abstract.

axioms (1)
  • domain assumption Foundation models pretrained on multimodal data encode associations between musical structure, timbre, and affective meaning
    Invoked in the abstract as the basis for using these models to probe emotional consequences.

pith-pipeline@v0.9.0 · 5696 in / 1182 out tokens · 36822 ms · 2026-05-22T12:29:51.792965+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

31 extracted references · 31 canonical work pages · 4 internal anchors

  1. [1]

    sublimity,

    INTRODUCTION The relationship between sound and human emotion has been exten- sively studied across multiple domains, particularly within affective computing and music cognition. Prior research has demonstrated how low-level audio features—such as timbre, tempo, pitch, and rhythm—correlate with emotional responses in listeners [1, 2, 3, 4, 5]. Parallel in...

    work page

  2. [2]

    Exploring How Audio Effects Alter Emotion with Foundation Models

    MA TERIAL In this study, we employed a diverse set of datasets and models, span- ning both deep and shallow architectures, along with several com- monly used audio FXs with varying parameter settings. The code, analytical details, and complete experimental results are available in our GitHub repository 1. We utilized threedatasetscapturing both categorica...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  3. [3]

    First, we analyzed performance changes to assess how audio ma- nipulations impact model accuracy

    EXPERIMENTS Using the above material, we conducted four experiments to investi- gate the relationships between audio effects and estimated emotions. First, we analyzed performance changes to assess how audio ma- nipulations impact model accuracy. Second, we examined shifts in emotion predictions to identify changes in predicted emotional la- bels or dimen...

    work page

  4. [4]

    Distortion and phaser, in particular, strongly increaseAngerwhile reducingCalmness, whereas chorus and delay introduce more variability in predictions

    CONCLUSIONS AND FUTURE WORK Our study demonstrates that audio effects substantially alter es- timated emotion in music. Distortion and phaser, in particular, strongly increaseAngerwhile reducingCalmness, whereas chorus and delay introduce more variability in predictions. Analysis of embedding-space trajectories shows that the magnitude and struc- ture of ...

    work page

  5. [5]

    Exploring relationships between audio features and emotion in mu- sic,

    Cyril Laurier, Olivier Lartillot, Tuomas Eerola, and Petri Toiviainen, “Exploring relationships between audio features and emotion in mu- sic,” inProceedings of the 7th Triennial Conference of European Soci- ety for the Cognitive Sciences of Music. Citeseer, 2009, pp. 260–264

    work page 2009

  6. [6]

    Audio fea- tures for music emotion recognition: a survey,

    Renato Panda, Ricardo Malheiro, and Rui Pedro Paiva, “Audio fea- tures for music emotion recognition: a survey,”IEEE Transactions on Affective Computing, vol. 14, no. 1, pp. 68–88, 2020

    work page 2020

  7. [7]

    Perceptual musical features for interpretable audio tag- ging,

    Vassilis Lyberatos, Spyridon Kantarelis, Edmund Dervakos, and Gior- gos Stamou, “Perceptual musical features for interpretable audio tag- ging,” in2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW). IEEE, 2024, pp. 878– 882

    work page 2024

  8. [8]

    Challenges and perspectives in interpretable music auto- tagging using perceptual features,

    Vassilis Lyberatos, Spyridon Kantarelis, Edmund Dervakos, and Gior- gos Stamou, “Challenges and perspectives in interpretable music auto- tagging using perceptual features,”IEEE Access, vol. 13, pp. 60720– 60732, 2025

    work page 2025

  9. [9]

    Music inter- pretation and emotion perception: A computational and neurophysio- logical investigation,

    Vassilis Lyberatos, Spyridon Kantarelis, Ioanna Zioga, Christina Anag- nostopoulou, Giorgos Stamou, and Anastasia Georgaki, “Music inter- pretation and emotion perception: A computational and neurophysio- logical investigation,” 2025

    work page 2025

  10. [10]

    Effect of sound sequence on soundscape emotions,

    Zhihui Han, Jian Kang, and Qi Meng, “Effect of sound sequence on soundscape emotions,”Applied Acoustics, vol. 207, pp. 109371, 2023

    work page 2023

  11. [11]

    Reverberation time and musical emotion in recorded music listening,

    Hannah Wilkie and Peter Harrison, “Reverberation time and musical emotion in recorded music listening,”Music Perception: An Interdis- ciplinary Journal, vol. 42, no. 4, pp. 329–344, 2025

    work page 2025

  12. [12]

    Comparing the acoustic expression of emotion in the speaking and the singing voice,

    Klaus R Scherer, Johan Sundberg, Lucas Tamarit, and Gl ´aucia L Sa- lom˜ao, “Comparing the acoustic expression of emotion in the speaking and the singing voice,”Computer Speech & Language, vol. 29, no. 1, pp. 218–235, 2015

    work page 2015

  13. [13]

    The emo- tional characteristics of the violin with different pitches, dynamics, and vibrato,

    Wenyi Song, Anh-Dung Dinh, and Andrew Brian Horner, “The emo- tional characteristics of the violin with different pitches, dynamics, and vibrato,” inProceedings of Meetings on Acoustics. Acoustical Society of America, 2024, vol. 55, p. 035004

    work page 2024

  14. [14]

    In- vestigating the sensitivity of pre-trained audio embeddings to common effects,

    Victor Deng, Changhong Wang, Gael Richard, and Brian McFee, “In- vestigating the sensitivity of pre-trained audio embeddings to common effects,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    work page 2025

  15. [15]

    What all do audio transformer models hear? probing acoustic rep- resentations for language delivery and its structure,

    Jui Shah, Yaman Kumar Singla, Changyou Chen, and Rajiv Ratn Shah, “What all do audio transformer models hear? probing acoustic rep- resentations for language delivery and its structure,”arXiv preprint arXiv:2101.00387, 2021

    work page arXiv 2021

  16. [16]

    Musiclm: Generating music from text,

    Andrea Agostinelli, Timo I. Denk, Zal ´an Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Chris- tian Frank, “Musiclm: Generating music from text,” 2023

    work page 2023

  17. [17]

    Mert: Acoustic music understanding model with large-scale self-supervised training,

    Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Bene- tos, et al., “Mert: Acoustic music understanding model with large-scale self-supervised training,”arXiv preprint arXiv:2306.00107, 2023

    work page arXiv 2023

  18. [18]

    Qwen Technical Report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al., “Qwen tech- nical report,”arXiv preprint arXiv:2309.16609, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  19. [19]

    Clap: Learning audio concepts from natural language supervision,

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang, “Clap: Learning audio concepts from natural language supervision,” inProceedings of the 2022 International Conference on Machine Learning (ICML), 2022

    work page 2022

  20. [20]

    Audio explanation synthesis with generative foundation models,

    Alican Akman, Qiyang Sun, and Bj ¨orn W Schuller, “Audio explanation synthesis with generative foundation models,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2025, pp. 1–5

    work page 2025

  21. [21]

    Emopia: A multi-modal pop piano dataset for emotion recognition and emotion-based music generation,

    Hsiao-Tzu Hung, Joann Ching, Seungheon Doh, Nabin Kim, Juhan Nam, and Yi-Hsuan Yang, “Emopia: A multi-modal pop piano dataset for emotion recognition and emotion-based music generation,”arXiv preprint arXiv:2108.01374, 2021

    work page arXiv 2021

  22. [22]

    Bench- marking music emotion recognition systems,

    Anna Alajanki, Yi-Hsuan Yang, and Mohammad Soleymani, “Bench- marking music emotion recognition systems,”PloS one, pp. 835–838, 2016

    work page 2016

  23. [23]

    A circumplex model of affect.,

    James A Russell, “A circumplex model of affect.,”Journal of person- ality and social psychology, vol. 39, no. 6, pp. 1161, 1980

    work page 1980

  24. [24]

    Assessing aesthetic music-evoked emotions in a minute or less: A comparison of the gems-45 and the gems-9,

    Peer-Ole Jacobsen, Hannah Strauss, Julia Vigl, Eva Zangerle, and Mar- cel Zentner, “Assessing aesthetic music-evoked emotions in a minute or less: A comparison of the gems-45 and the gems-9,”Musicae Sci- entiae, p. 10298649241256252, 2024

    work page 2024

  25. [25]

    Large-scale contrastive language- audio pretraining with feature fusion and keyword-to-caption augmen- tation,

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Kirkpatrick, and Shlomo Dubnov, “Large-scale contrastive language- audio pretraining with feature fusion and keyword-to-caption augmen- tation,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    work page 2023

  26. [26]

    Qwen2-Audio Technical Report

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al., “Qwen2- audio technical report,”arXiv preprint arXiv:2407.10759, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  27. [27]

    Xgboost: A scalable tree boosting system,

    Tianqi Chen and Carlos Guestrin, “Xgboost: A scalable tree boosting system,” inProceedings of the 22nd acm sigkdd international confer- ence on knowledge discovery and data mining, 2016, pp. 785–794

    work page 2016

  28. [28]

    Objective study of the performance degradation in emotion recognition through the amr-wb+ codec.,

    Aaron Albin and Elliot Moore, “Objective study of the performance degradation in emotion recognition through the amr-wb+ codec.,” in INTERSPEECH, 2015, pp. 1319–1323

    work page 2015

  29. [29]

    Black-box adversar- ial attacks through speech distortion for speech emotion recognition,

    Jinxing Gao, Diqun Yan, and Mingyu Dong, “Black-box adversar- ial attacks through speech distortion for speech emotion recognition,” EURASIP Journal on Audio, Speech, and Music Processing, vol. 2022, no. 1, pp. 20, 2022

    work page 2022

  30. [30]

    Reverb and noise as real- world effects in speech recognition models: A study and a proposal of a feature set,

    Valerio Cesarini and Giovanni Costantini, “Reverb and noise as real- world effects in speech recognition models: A study and a proposal of a feature set,”Applied Sciences, vol. 14, no. 23, pp. 11446, 2024

    work page 2024

  31. [31]

    UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction

    Leland McInnes, John Healy, and James Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,”arXiv preprint arXiv:1802.03426, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018