REVIEW 3 major objections 4 minor 22 references
Go witheFlow: Real-time Emotion Driven Audio Effects Modulation
T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read witheFlow maps a performer's EEG attention, ECG stress, and audio emotion into real-time gain changes on DAW effect chains, keeping the musician in creative control.
desk verdict Honest, clearly-described proof-of-concept for bio-signal-driven audio FX; the load-bearing risk is the unvalidated audio emotion regressor, but the paper itself flags it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing components are the Baevsky Stress Index (a heart-rate-variability metric computed from the mode and range of RR intervals), the attention and relaxation ratios derived from EEG alpha and beta power over four-second windows, and the valence-arousal regressor built by replacing the final layer of a PANNs CNN10 with a two-output linear head trained on DEAM. The mixing logic is a transparent piecewise map: the input domain (stress, attention, valence, arousal) is partitioned into regions, each paired with a Python gain function; user-editable YAML files define the region conditions and functions. This formalization is what makes the system interpretable and prepares it for a fut
What would settle it
Collect solo recordings with human valence-arousal ratings, run the system's regressor on them, and check correlation with the ratings; if predictions are uncorrelated or systematically shifted, the V-A-distance rules in all four rulesets are being driven by unreliable features. A simpler laboratory test is to feed synthetic V-A values into the mixing logic and verify that gain changes match the specified quadrant rules exactly.
Extended reading notes
Core claim
The central claim is that a laptop-runnable system can close a feedback loop between a performer's measurable internal state and their sonic output without altering musical content. Signals from commercial EEG and ECG sensors are converted into attention/relaxation and stress values; a PANNs CNN10 retrained on DEAM regresses the dry audio's valence and arousal; a rule-based mixing logic, formalized as a piecewise function over a partitioned input domain, translates these values into gain changes on multiple effects channels. The gain logic is organized by stress/attention quadrants: high stress pushes the mix toward V-A positions far from the dry signal, while low stress keeps it close, with
Load-bearing premise
The load-bearing premise is that the audio emotion regressor — PANNs CNN10 trained on DEAM — gives trustworthy valence-arousal estimates for dry solo performance audio in real time; the paper itself notes that emotion datasets typically contain full productions rather than solo performances.
Editorial extensions
If this is right
- Live performers can shape their effect mix hands-free, with gain decisions driven by measurable internal state rather than manual knob-twiddling.
- Because the rules are editable YAML/Python files, musicians can inspect exactly why a gain changed and customize or reverse the behavior, supporting trust and agency.
- Local, laptop-based processing keeps biosignal data private and avoids cloud latency, making the approach practical in real performance settings.
- The piecewise-rule formalization points directly to a learnable mixing logic, where decision trees could refine user-defined rules while staying interpretable.
- The system's reliance on solo performance audio makes annotated solo-performance datasets with real-time labels a critical next resource.
Reading between the lines
- If the DEAM-trained regressor is systematically biased on solo instrument audio, every biosignal ruleset inherits that bias, since all rules compare the dry signal's V-A position with each effect channel's V-A position; a validation study on solo recordings would settle this.
- The architecture is feature-agnostic, so the same rule-based loop could be closed with video, motion, or other physiological signals without redesigning the mixing logic.
- The closed loop between internal state and sound raises a testable safety question: whether the system's own audio changes could feed back into the performer's physiology over long sessions; the foot-pedal override mitigates but does not measure this.
- A quantitative evaluation framework for emotion-driven mixing systems, which the paper identifies as missing, could be built from the V-A-distance principle underlying the rules.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces witheFlow, a proof-of-concept system for real-time audio effects modulation driven by a combination of biosignal features (EEG-derived attention/relaxation, ECG-derived Baevsky Stress Index) and audio-based valence-arousal estimates. It describes the system architecture, DAW routing, a customizable YAML rule-based mixing logic, artifact handling, and reports anecdotal positive feedback from collaborating musicians. The central contribution is the integration and open-source implementation rather than a formal empirical evaluation; the paper explicitly states that a quantitative evaluation framework is future work.
Significance. If the system's underlying emotion estimates are trustworthy, witheFlow is a relevant contribution to real-time co-creative performance systems: it is lightweight, local, interpretable, and provides performer control via MIDI override. The architecture is coherent, the formulas for attention/relaxation and Baevsky SI are standard, and the decision to use transparent rule-based mixing rather than an opaque end-to-end model is well motivated. The paper's strengths include its open-source commitment and its careful ethical discussion of biosignal privacy and psychological safety. The main risk is that the audio V-A regressor, which is load-bearing for all four rulesets, is not validated on the solo dry instrumental material that the system actually processes.
major comments (3)
- [§2, Audio Emotion Regressor; Table 1] The audio V-A regressor is load-bearing for every rule in Table 1, since each rule compares the dry signal's V-A position with each effect channel's V-A position. The manuscript concedes that DEAM contains full music productions rather than solo performances, but reports no validation of the regressor on DEAM's test split or on solo/instrumental material. If the V-A estimates are systematically biased for dry solo input, all four rulesets are corrupted and the 'emotion-driven' behavior reduces to an arbitrary effect switcher. Please add quantitative validation (e.g., DEAM test-set metrics and a small solo-performance evaluation) or explicitly restrict the system claims to the design/proof-of-concept level.
- [§3, Discussion; §5, Conclusion] The abstract and conclusion state that the system 'enhances' music performance, but the only supporting evidence is the statement in Section 3 that musicians 'have reported a generally positive experience.' No details are given on the number of musicians, instruments, tasks, instructions, or data collection. Either report a systematic qualitative evaluation (even a small-N study with a clear protocol) or reframe the claims to say that the system is intended to enhance and that only anecdotal feedback has been collected so far. The paper's own Section 3 says formal evaluation is future work, so the current wording overstates the evidence.
- [§2, Mixing Logic; Table 1] Table 1 defines behavior in terms of 'High/Low' stress, attention, valence, and arousal, but the threshold values and calibration procedure are not specified. This makes the default behavior non-reproducible from the text alone. Please provide the default threshold values (or point to the exact configuration lines in the GitHub repository) and describe how thresholds are calibrated for individual performers.
minor comments (4)
- [Title/Abstract] The system name is typeset inconsistently: 'Go witheFlow', 'witheFlow', and 'witheflow' (in the YouTube link). Please standardize.
- [§2, Audio Routing] Typo: 'The performer creates up toneffects chains' should read 'up to n effects chains' (or similar).
- [§2, Audio Emotion Regressor] Use a space in '30kHz' and state the original PANNs input sampling rate for comparison.
- [§4, Ethical considerations] The institution is named 'National and Technical University of Athens' here but 'National Technical University of Athens' in the affiliation. Please make this consistent.
Circularity Check
No material circularity: the system is an openly described proof-of-concept whose components come from external sources and whose rules are disclosed design choices, not derived predictions.
full rationale
The paper does not derive any result from its own output. The two feature extractors are external: the audio emotion regressor is a PANNs CNN10 trained on the DEAM dataset, and the biosignal features (attention/relaxation, Baevsky SI) are standard definitions with external citations. The mixing logic is presented as hand-written, customizable rules, with the paper explicitly stating 'the mixing logic is fully customizable' and that the functions 'can easily be customized by the end user' — these are design choices, not predictions fitted to data. The formalization as a piecewise function over a partitioned domain is a description of the implemented rule set, not a derivation that presupposes its conclusion. The only self-citations ([7], [8]) are used for framing and for a preliminary dataset analysis; neither is load-bearing for the system's claimed functionality. The paper's own concession that 'datasets typically contain full music productions and not solo performances' and its statement that 'a formal framework for quantitative evaluation' remains future work identify a correctness/validation risk, but not circularity: the audio regressor is trained externally and no evaluation on solo material is claimed. Overall, the claim is a working prototype with disclosed components and no hidden fit-by-construction step.
Assumptions & free parameters
free parameters (3)
- High/Low thresholds for stress, attention, and V-A rule conditions
- Feature window sizes (EEG 4 s / 1000 samples, ECG 15 s, audio 5 s) =
EEG 1000 samples; ECG 15 s; audio 5 s
- V-A coordinates assigned to each FX channel
assumptions (5)
- domain assumption Increased beta power indicates heightened attention and increased alpha power indicates greater relaxation (§2, Biosignals).
- domain assumption The Baevsky Stress Index computed from 15-s RR-interval windows quantifies the performer's momentary stress (§2, Biosignals).
- domain assumption Valence-Arousal from Russell's circumplex adequately represents the emotion of dry solo audio, and PANNs CNN10 retrained on DEAM estimates it reliably in the deployment domain (§2, Audio Emotion Regressor).
- ad hoc to paper The intuitive mapping of Table 1 holds: high stress is 'somehow expressed via their performance' so boosting distant FX is appropriate; low stress calls for proximity; attention can be used as an indicator of intent (§2, Mixing Logic).
- standard math The piecewise-function partition notation correctly formalizes the mixing logic (§2, Mixing Logic).
Cite this review
Pith. "Pith review of Go witheFlow: Real-time Emotion Driven Audio Effects Modulation." pith.science (2026). https://pith.science/paper/E3UAUPRQ
@misc{pith2026251002171,
author = {Pith},
title = {Pith review of: Go witheFlow: Real-time Emotion Driven Audio Effects Modulation},
year = {2026},
howpublished = {\url{https://pith.science/paper/E3UAUPRQ}},
note = {Machine review of arXiv:2510.02171}
}
read the original abstract
Music performance is a distinctly human activity, intrinsically linked to the performer's ability to convey, evoke, or express emotion. Machines cannot perform music in the human sense; they can produce, reproduce, execute, or synthesize music, but they lack the capacity for affective or emotional experience. As such, music performance is an ideal candidate through which to explore aspects of collaboration between humans and machines. In this paper, we introduce the witheFlow system, designed to enhance real-time music performance by automatically modulating audio effects based on features extracted from both biosignals and the audio itself. The system, currently in a proof-of-concept phase, is designed to be lightweight, able to run locally on a laptop, and is open-source given the availability of a compatible Digital Audio Workstation and sensors.
Figures
Reference graph
Works this paper leans on
-
[1]
Developing a benchmark for emotional analysis of music.PLOS ONE, 12(3):e0173392, 2017
Anna Aljanaki, Yi-Hsuan Yang, and Mohammad Soleymani. Developing a benchmark for emotional analysis of music.PLOS ONE, 12(3):e0173392, 2017
2017
-
[2]
A systematic review of artificial intelligence-based music generation: Scope, applications, and future trends
Miguel Civit, Javier Civit-Masot, Francisco Cuadrado, and Maria J Escalona. A systematic review of artificial intelligence-based music generation: Scope, applications, and future trends. Expert Systems with Applications, 209:118190, 2022
2022
-
[3]
Using artificial intelligence for enhancing human creativity
Mayssa Elfar and Mina Dawood. Using artificial intelligence for enhancing human creativity. Journal of Art, Design and Music, 2:106–120, 07 2023
2023
-
[4]
Ddsp: Differentiable digital signal processing.arXiv preprint arXiv:2001.04643, 2020
Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, and Adam Roberts. Ddsp: Differentiable digital signal processing.arXiv preprint arXiv:2001.04643, 2020
arXiv 2001
-
[5]
Eeg-neurofeedback for optimising performance
John H Gruzelier. Eeg-neurofeedback for optimising performance. i: A review of cognitive and affective outcome in healthy participants.Neuroscience & Biobehavioral Reviews, 44:124–141, 2014
2014
-
[6]
Plumbley
Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880–2894, 2020
2020
-
[7]
Challenges and perspectives in interpretable music auto-tagging using perceptual features.IEEE Access, 2025
Vassilis Lyberatos, Spyridon Kantarelis, Edmund Dervakos, and Giorgos Stamou. Challenges and perspectives in interpretable music auto-tagging using perceptual features.IEEE Access, 2025
2025
-
[8]
Music interpretation and emotion perception: A computational and neurophysiological investigation, 2025
Vassilis Lyberatos, Spyridon Kantarelis, Ioanna Zioga, Christina Anagnostopoulou, Giorgos Stamou, and Anastasia Georgaki. Music interpretation and emotion perception: A computational and neurophysiological investigation, 2025
2025
Show all 22 references
-
[9]
Bahar Mahmud, Guan Hong, and Bernard Fong. A study of human–ai symbiosis for creative work: Recent developments and future directions in deep learning.ACM Transactions on Multimedia Computing, Communications and Applications, 20(2):1–21, 2023
2023
-
[10]
Autonomy, authenticity, authorship and intention in computer generated art
Jon McCormack, Tom Gifford, and Pat Hutchings. Autonomy, authenticity, authorship and intention in computer generated art. InProceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 15, pages 53–59, 2019
2019
-
[11]
Music genre classification: A review of deep-learning and traditional machine-learning approaches
Ndiatenda Ndou, Ritesh Ajoodha, and Ashwini Jadhav. Music genre classification: A review of deep-learning and traditional machine-learning approaches. In2021 IEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS), pages 1–6. IEEE, 2021
2021
-
[12]
Increase in beta power reflects attentional top-down modulation after psychosocial stress induction
Ismael Palacios-García, Jaime Silva, Mario Villena-González, Germán Campos-Arteaga, Clau- dio Artigas-Vergara, Nicolas Luarte, Eugenio Rodríguez, and Conrado A Bosman. Increase in beta power reflects attentional top-down modulation after psychosocial stress induction. Frontier...
2021
-
[13]
AR Editions, Inc., 1995
Joseph Rothstein.MIDI: A comprehensive introduction, volume 7. AR Editions, Inc., 1995
1995
-
[14]
James A. Russell. A circumplex model of affect.Journal of Personality and Social Psychology, 39(6):1161–1178, 1980
1980
-
[15]
Stress index calculation and analysis based on heart rate variability of ecg signal with arrhythmia
Tanmay Kumar Sahoo, Ashutos Mahapatra, and Nersisson Ruban. Stress index calculation and analysis based on heart rate variability of ecg signal with arrhythmia. In2019 Innovations in Power and Advanced Computing Technologies (i-PACT), volume 1, pages 1–7. IEEE, 2019
2019
-
[16]
Elec- troencephalographic biomarkers of relaxation: A systematic review and meta-analysis.bioRxiv, March 2024
Kairi Sugimoto, Hideaki Kurashiki, Yuting Xu, Mitsuaki Takemi, and Kaoru Amano. Elec- troencephalographic biomarkers of relaxation: A systematic review and meta-analysis.bioRxiv, March 2024
2024
-
[17]
Towards a human-centric design framework for ai assisted music production
Augoustinos Tsiros and Alessandro Palladini. Towards a human-centric design framework for ai assisted music production. InProceedings of the International Conference on New Interfaces for Musical Expression, pages 399–404, 2020. 7
2020
-
[18]
Ai see, you see: Human-ai musical collaboration in augmented reality
Yichen Wang and Charles Patrick Martin. Ai see, you see: Human-ai musical collaboration in augmented reality. InProceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA ’25, New York, NY , USA, 2025. Association for Computing Machinery
2025
-
[19]
Midi-ddsp: Detailed control of musical performance via hierarchical modeling.arXiv preprint arXiv:2112.09312, 2021
Yusong Wu, Ethan Manilow, Yi Deng, Rigel Swavely, Kyle Kastner, Tim Cooijmans, Aaron Courville, Cheng-Zhi Anna Huang, and Jesse Engel. Midi-ddsp: Detailed control of musical performance via hierarchical modeling.arXiv preprint arXiv:2112.09312, 2021
2021 arXiv
-
[20]
A clustering-based approach to automatic harmonic analysis: An exploratory study of harmony and form in mozart’s piano sonatas
Jason Yust, Jaeseong Lee, and Eugene Pinsky. A clustering-based approach to automatic harmonic analysis: An exploratory study of harmony and form in mozart’s piano sonatas. Transactions of the International Society for Music Information Retrieval, 2022
2022
-
[21]
Symbolic music representations for classification tasks: A systematic evalua- tion.arXiv preprint arXiv:2309.02567, 2023
Huan Zhang, Emmanouil Karystinaios, Simon Dixon, Gerhard Widmer, and Carlos Eduardo Cancino-Chacón. Symbolic music representations for classification tasks: A systematic evalua- tion.arXiv preprint arXiv:2309.02567, 2023
2023 arXiv
-
[22]
A survey of ai music generation tools and models.arXiv preprint arXiv:2308.12982, 2023
Yueyue Zhu, Jared Baca, Banafsheh Rekabdar, and Reza Rawassizadeh. A survey of ai music generation tools and models.arXiv preprint arXiv:2308.12982, 2023. 8
2023 arXiv
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.