Opening the Design Space: Two Years of Performance with Intelligent Musical Instruments
Pith reviewed 2026-05-08 05:20 UTC · model grok-4.3
The pith
Portable AI on cheap hardware opens the design space for intelligent musical instruments by letting artists use their own data and remap controls instead of retraining.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
An inexpensive generative AI instrument platform was built around a single-board computer that accepts MIDI input and runs models trained on artist-collected datasets. Five instruments were designed, tested, and performed with across two years of first-person artistic research. The resulting examples establish that remapping can replace retraining when discovering new AI interactions, that fast input interleaving functions as a co-creative strategy, that small-data models serve as transportable design resources, and that inexpensive hardware lowers barriers to inclusion in this kind of work.
What carries the argument
The inexpensive generative AI instrument platform on a single-board computer with MIDI connectivity that runs small artist-collected models for live performance and remapping.
If this is right
- Remapping inputs can replace retraining when artists want to discover new ways of interacting with generative models.
- Fast interleaving of performer inputs functions as a distinct co-creative strategy during live performance.
- Small-data models trained on artist-specific collections act as portable and reusable design resources.
- Inexpensive single-board computer hardware lowers barriers and increases inclusion for creating intelligent instruments.
- Artists can integrate generative AI directly into ongoing musical practices and performance schemes.
Where Pith is reading between the lines
- Other creative domains that rely on real-time control and small personal datasets might adopt similar low-cost AI platforms.
- Widespread use of artist-collected training data could produce more varied and context-specific generative outputs than generic models.
- The remapping approach may reduce the computational cost of iterating on AI instrument designs compared with repeated full retraining.
Load-bearing premise
Insights drawn from one person's two-year process and five specific instruments are representative enough to describe the broader design space and will generalize to other artists and instruments.
What would settle it
A group of musicians without the author's background builds and performs with similar cheap AI platforms and fails to discover effective remapping strategies or to treat fast input interleaving as a useful co-creative technique.
Figures
read the original abstract
Machine generation of symbolic music and digital audio are hot topics but there have been relatively few digital musical instruments that integrate generative AI. Present musical AI tools are not artist centred and do not support experimentation or integrating into musical instruments or practices. This work introduces an inexpensive generative AI instrument platform based on a single board computer that connects via MIDI to other musical devices. The platform uses artist-collected datasets with models trained on a regular computer. This paper asks what the design space of intelligent musical instruments might look like when accessible and portable AI systems are available for artistic exploration. I contribute five examples of instruments created and tested through a two-year first-person artistic research process. These show that (re)mapping can replace retraining for discovering AI interaction, that fast input interleaving is a new co-creative strategy, that small-data AI models can be a transportable design resource, and that cheap hardware can lower barriers to inclusion. This work could enable artists to explore new interaction and performance schemes with intelligent musical instruments.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces an inexpensive generative AI musical instrument platform using single-board computers and MIDI connectivity, with models trained on artist-collected small datasets. Through a two-year first-person artistic research process, the author develops and performs with five specific instruments, claiming that accessible portable AI opens the design space of intelligent musical instruments. The contributions include demonstrations that remapping can replace retraining for discovering interactions, fast input interleaving is a new co-creative strategy, small-data models are transportable, and cheap hardware lowers inclusion barriers.
Significance. If the insights from these examples generalize, the work could meaningfully expand access to AI-augmented instruments for a wider range of artists by providing a practical, low-cost platform and concrete interaction strategies. The first-person practice-based approach and emphasis on small-data, remappable models represent a strength in enabling rapid artistic experimentation without heavy computational resources.
major comments (1)
- [Abstract] Abstract and the section describing the five instruments: the central claim that these examples open the broader design space (via remapping replacing retraining, fast interleaving as co-creation, transportable small-data models, and lowered barriers) depends on the representativeness of one artist's two-year personal practice; no external validation, user studies with other practitioners, or comparative cases are provided to support generalization beyond the specific instruments.
Simulated Author's Rebuttal
We thank the referee for their constructive feedback and the opportunity to clarify the positioning and scope of our work. We address the major comment below.
read point-by-point responses
-
Referee: [Abstract] Abstract and the section describing the five instruments: the central claim that these examples open the broader design space (via remapping replacing retraining, fast interleaving as co-creation, transportable small-data models, and lowered barriers) depends on the representativeness of one artist's two-year personal practice; no external validation, user studies with other practitioners, or comparative cases are provided to support generalization beyond the specific instruments.
Authors: We acknowledge that the study is based exclusively on the first author's two-year personal artistic practice and provides no external validation, user studies, or comparative cases with other practitioners. This is a deliberate methodological choice aligned with first-person artistic research, which emphasizes longitudinal, embodied exploration of design possibilities rather than statistical generalization. The manuscript frames its contributions as concrete demonstrations of what becomes feasible with accessible, portable AI platforms, not as claims that the observed strategies are representative or optimal for all artists. To better reflect this scope, we will revise the abstract and the sections describing the instruments and discussion to explicitly state that the examples illustrate potential interaction strategies and design resources without asserting broader representativeness or generalizability. This clarification will strengthen the alignment between the claims and the practice-based methodology. revision: partial
Circularity Check
No circularity: insights drawn directly from described artistic practice
full rationale
The paper contains no equations, derivations, fitted parameters, or self-citations. Its central claims rest on five instruments created and performed by the author over two years; these examples are presented as direct evidence rather than as outputs of any model or mapping that reduces to the inputs by construction. Generalization from personal practice is a methodological limitation but does not constitute a circular reduction of the kind enumerated in the analysis criteria.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
Julia Barnett. 2023. The Ethical Implications of Generative Audio Models: A Systematic Literature Review. InProceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society(Montréal, QC, Canada)(AIES ’23). Association for Computing Machinery, New York, NY, USA, 146–161. https://doi.org/10.114 5/3600211.3604686
-
[2]
Edgar Berdahl and Wendy Ju. 2011. Satellite CCRMA: A Musical Interaction and Sound Synthesis Platform. InProceedings of the International Conference on New Interfaces for Musical Expression (NIME ’11), Alexander Refsum Jensenius, Anders Tveit, Rolf Inge Godøy, and Dan Overholt (Eds.). University of Oslo, Oslo, Norway, 173–178. http://www.nime.org/proceedi...
2011
-
[3]
Edgar Berdahl, Spencer Salazar, and Myles Borins. 2013. Embedded Network- ing and Hardware-Accelerated Graphics with Satellite CCRMA. InProceedings of the International Conference on New Interfaces for Musical Expression (NIME ’13), Woon Seung Yeo, Kyogu Lee, Alexander Sigman, Haru Ji, and Graham Wakefield (Eds.). KAIST, Daejeon, Republic of Korea, 325–330
2013
-
[4]
John A. Biles. 2007. Improvizing with Genetic Algorithms: GenJam. InEvo- lutionary Computer Music, Eduardo Reck Miranda and John Al Biles (Eds.). Opening the Design Space: Two Years of Performance with Intelligent Musical Instruments NIME ’26, June 23–26, 2026, London, UK Springer London, London, 137–169. https://doi.org/10.1007/978-1-84628-600- 1_7
-
[5]
John M Bowers. 2023. A Hapless But Entertaining Roar: Developing a Room Feedback System through Artistic Research and Aesthetic Reflection. InPro- ceedings of the International Conference on New Interfaces for Musical Expression, Miguel Ortiz and Adnan Marquez-Borbon (Eds.). Mexico City, Mexico, Article 71, 10 pages. https://doi.org/10.5281/zenodo.11189266
- [6]
-
[7]
Michelle Carney, Barron Webster, Irene Alvarado, Kyle Phillips, Noura How- ell, Jordan Griffith, Jonas Jongejan, Amit Pitaru, and Alexander Chen. 2020. Teachable Machine: Approachable Web-Based Tool for Exploring Machine Learning Classification. InExtended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, ...
-
[8]
Matthew Clauhs and Brian Dozoretz. 2023. The DAW Revolution. InThe Routledge Companion to Creativities in Music Education, Clint Randles and Pamela Burnard (Eds.). Taylor & Francis Group. https://doi.org/10.4324/9781 003248194
-
[9]
Dylan Davis. 2022. Compute and Resonate: An Ongoing Experiment in Creat- ing Acid Music Using Accessible Artificial Intelligence and Computer-Based Generative Tools. InDesigning Interactions for Music and Sound, Michael Filimowicz (Ed.). Routledge
2022
-
[10]
Emmanuel Deruty, Maarten Grachten, Stefan Lattner, Javier Nistal, and Cyran Aouameur. 2022. On the Development and Practice of AI Technology for Contemporary Popular Music Production.Transactions of the International Society for Music Information Retrieval5, 1 (Feb. 2022), 35. https://transactio ns.ismir.net/article/10.5334/tismir.100/
-
[11]
Rebecca Fiebrink and Laetitia Sonami. 2020. Reflections on Eight Years of Instrument Creation with Machine Learning. InProceedings of the International Conference on New Interfaces for Musical Expression. 237–242. https://doi.org/ 10.5281/zenodo.4813334
-
[12]
Rebecca Fiebrink, Dan Trueman, and Perry R Cook. 2009. A Meta-Instrument for Interactive, On-the-fly Machine Learning. InProc. NIME. 280–285
2009
-
[13]
2024.AI and Music: Market Development of AI in the Music Sector and Impact on Music Creators in Australia and New Zealand
Goldmedia. 2024.AI and Music: Market Development of AI in the Music Sector and Impact on Music Creators in Australia and New Zealand. Technical Report. https://www.apraamcos.com.au/about/supporting-the-industry/research- papers/aiandmusic
2024
-
[14]
Edmonds, and Damith Herath
Myounghoon Jeon, Rebecca Fiebrink, Ernest A. Edmonds, and Damith Herath
-
[15]
From rituals to magic: Interactive art and HCI of the past, present, and future.International Journal of Human-Computer Studies131 (2019), 108–119. https://doi.org/10.1016/j.ijhcs.2019.06.005 50 years of the International Journal of Human-Computer Studies. Reflections on the past, present and future of human-centred technologies
-
[16]
Shulei Ji, Xinyu Yang, and Jing Luo. 2024. A Survey on Deep Learning for Symbolic Music Generation: Representations, Algorithms, Evaluations, and Challenges.Comput. Surveys56, 1 (2024), 1–39. https://doi.org/10.1145/3597 493
-
[17]
Théo Jourdan and Baptiste Caramiaux. 2023. Machine Learning for Musical Expression: A Systematic Literature Review. InProceedings of the International Conference on New Interfaces for Musical Expression. 13 pages. http://nime.o rg/proceedings/2023/nime2023_46.pdf
2023
-
[18]
Annika Kaltenhauser, Evropi Stefanidi, and Johannes Schöning. 2024. Playing with Perspectives and Unveiling the Autoethnographic Kaleidoscope in HCI – A Literature Review of Autoethnographies. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA)(CHI ’24). Association for Computing Machinery, New York, NY, USA, ...
-
[19]
Purnima Kamath, Fabio Morreale, Priambudi Lintang Bagaskara, Yize Wei, and Suranga Nanayakkara. 2024. Sound Designer-Generative AI Interactions: Towards Designing Creative Support Tools for Professional Sound Designers. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA)(CHI ’24). Association for Computing Mac...
-
[20]
Laewoo (Leo) Kang, Steven J. Jackson, and Phoebe Sengers. 2018. Intermodu- lation: Improvisation and Collaborative Art Practice for HCI. InProceedings of the 2018 CHI Conference on Human Factors in Computing Systems(Montreal QC, Canada)(CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3173734
-
[21]
George E. Lewis. 2000. Too Many Notes: Computers, Complexity and Culture in “Voyager”.Leonardo Music Journal10 (2000), 33–39. https://doi.org/10.116 2/096112100570585
2000
-
[22]
Alex Lucas, Miguel Ortiz, and Franziska Schroeder. 2020. The Longevity of Bespoke, Accessible Music Technology: A Case for Community. InProceed- ings of the International Conference on New Interfaces for Musical Expression, Romain Michon and Franziska Schroeder (Eds.). Birmingham City University, Birmingham, UK, 243–248. https://doi.org/10.5281/zenodo.4813338
-
[23]
2019.cpmpercussion/keras-mdn-layer
Charles Martin. 2019.cpmpercussion/keras-mdn-layer. https://doi.org/10.528 1/zenodo.1482347
2019
-
[24]
2026.IMPSY: An Intelligent Musical Instrument Platform
Charles Patrick Martin. 2026.IMPSY: An Intelligent Musical Instrument Platform. https://doi.org/10.5281/zenodo.2580175
-
[25]
Charles Patrick Martin, Kyrre Glette, Tønnes Frostad Nygaard, and Jim Torre- sen. 2020. Understanding Musical Predictions With an Embodied Interface for Musical Machine Learning.Frontiers in Artificial Intelligence3 (2020). https://doi.org/10.3389/frai.2020.00006
-
[26]
Charles Patrick Martin and Jim Torresen. 2019. An Interactive Musical Predic- tion System with Mixture Density Recurrent Neural Networks. InProceedings of the International Conference on New Interfaces for Musical Expression. 260–
2019
-
[27]
https://doi.org/10.5281/zenodo.3672952
-
[28]
Raul Masu, Nicolò Merendino, Antonio Rodã, and Luca Turchet. 2024. Sus- tainable Internet of Musical Things: Strategies to Account for Environmental and Social Sustainability in Network-Based Interactive Music Systems.IEEE Access12 (2024), 62818–62833. https://doi.org/10.1109/ACCESS.2024.3393468
-
[29]
Raul Masu, Fabio Morreale, and Alexander Refsum Jensenius. 2023. The O in NIME: Reflecting on the Importance of Reusing and Repurposing Old Musical Instruments. InProceedings of the International Conference on New Interfaces for Musical Expression, Miguel Ortiz and Adnan Marquez-Borbon (Eds.). Mexico City, Mexico, Article 14, 10 pages. https://doi.org/10....
-
[31]
MIDI Manufacturers Association. 2014. MIDI 1.0 Electrical Specification Update. https://www.midi.org/specifications-old/item/midi-din-electrical- specification
2014
-
[32]
Blake Montgomery. 2023. Music publishers sue Amazon-backed AI company over song lyrics.The Guardian(oct 2023). https://www.theguardian.com/te chnology/2023/oct/19/music-lawsuit-ai-song-lyrics-anthropic
2023
-
[33]
Michael Muller, Lydia B Chilton, Anna Kantosalo, Charles Patrick Martin, and Greg Walsh. 2022. GenAICHI: Generative AI and HCI. InExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems(New Orleans, LA, USA)(CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 110, 7 pages. https://doi.org/10.1145/349110...
-
[34]
Torgrim Rudland Næss and Charles Patrick Martin. 2019. A Physical In- telligent Instrument using Recurrent Neural Networks. InProceedings of the International Conference on New Interfaces for Musical Expression. 79–82. https://doi.org/10.5281/zenodo.3672874
-
[35]
Carman Neustaedter and Phoebe Sengers. 2012. Autobiographical design in HCI research: designing and learning through use-it-yourself. InProceedings of the Designing Interactive Systems Conference(Newcastle Upon Tyne, United Kingdom)(DIS ’12). Association for Computing Machinery, New York, NY, USA, 514–523. https://doi.org/10.1145/2317956.2318034
-
[36]
François Pachet. 2003. The Continuator: Musical Interaction With Style. Journal of New Music Research32, 3 (2003), 333–341. https://doi.org/10.1076/ jnmr.32.3.333.16861
2003
-
[37]
Adam Parkinson and Graham Dunning. 2025. Improvising with Machines: A taxonomy of musical interactions.Organised Sound(2025), 1–12. https: //doi.org/10.1017/S1355771824000268
-
[38]
Benito Temprano, and Andrew McPher- son
Teresa Pelinski, Rodrigo Diaz, Adan L. Benito Temprano, and Andrew McPher- son. 2023. Pipeline for recording datasets and running neural networks on the Bela embedded hardware platform. InProceedings of the Interna- tional Conference on New Interfaces for Musical Expression. 160–166. http: //nime.org/proceedings/2023/nime2023_22.pdf
2023
-
[39]
Nicola Privato and Thor Magnusson. 2024. Querying the Ghost: AI Hauntog- raphy in NIME. InProceedings of the International Conference on New Interfaces for Musical Expression, S M Astrid Bin and Courtney N. Reed (Eds.). Utrecht, Netherlands, Article 63, 7 pages. https://doi.org/10.5281/zenodo.13904901
-
[40]
Anders Reuter. 2022. Who let the DAWs Out? The Digital in a New Generation of the Digital Audio Workstation.Popular Music and Society45, 2 (2022), 113–
2022
-
[41]
https://doi.org/10.1080/03007766.2021.1972701
-
[42]
Curtis Roads. 1985. Research in music and artificial intelligence.ACM Comput. Surv.17, 2 (June 1985), 163–190. https://doi.org/10.1145/4468.4469
-
[43]
Adam Roberts, Jesse Engel, Yotam Mann, Jon Gillick, Claire Kayacik, Signe Nørly, Monica Dinculescu, Carey Radebaugh, Curtis Hawthorne, and Douglas Eck. 2019. Magenta Studio: Augmenting Creativity with Deep Learning in Ableton Live. InProceedings of the International Workshop on Musical Metacre- ation (MUME). http://musicalmetacreation.org/buddydrive/file/...
2019
-
[44]
Victor Shepardson, Halla Steinunn Stefánsdóttir, and Thor Magnusson. 2025. Evolving the Living Looper: Artistic Research, Online Learning, and Tenta- cle Pendula. InProceedings of the International Conference on New Interfaces for Musical Expression, Doga Cavdir and Florent Berthaut (Eds.). Canberra, Australia, Article 36, 6 pages. https://doi.org/10.5281...
-
[45]
Hazel Smith and Roger T. Dean. 2009. Introduction: Practice-led Research, Research-led Practice — Towards the Iterative Cyclic Web. InPractice-led Research, Research-led Practice in the Creative Arts, Hazel Smith and Roger T. Dean (Eds.). Edinburgh University Press, 1–38
2009
-
[46]
Halla Steinunn Stefánsdóttir and Thor Magnusson. 2025. Of altered instrumen- tal relations: a practice-led inquiry into agency through musical performance with neural audio synthesis and violin.Frontiers in Computer ScienceVolume 7 - 2025 (2025). https://doi.org/10.3389/fcomp.2025.1578595
-
[47]
Miriam Sturdee, Makayla Lewis, Mafalda Gamboa, Thuong Hoang, John Miers, Ilja Šmorgun, Pranjal Jain, Angelika Strohmayer, Sarah Fdili Alaoui, and Christina R Wodtke. 2022. The State of the (CHI)Art. InExtended Abstracts NIME ’26, June 23–26, 2026, London, UK Charles Patrick Martin of the 2022 CHI Conference on Human Factors in Computing Systems(New Orlean...
-
[48]
Miriam Sturdee, Makayla Lewis, Angelika Strohmayer, Katta Spiel, Nantia Koulidou, Sarah Fdili Alaoui, and Josh Urban Davis. 2021. A Plurality of Practices: Artistic Narratives in HCI Research. InProceedings of the 13th Con- ference on Creativity and Cognition(Virtual Event, Italy)(C&C ’21). Associ- ation for Computing Machinery, New York, NY, USA, Article...
-
[49]
Koray Tahiroğlu. 2022. Evolving musical expectations: Mutual correlation between a human musician and an AI musical instrument. InEmbodied gestures. Technische Universität Wien, 30–36. https://doi.org/10.34727/2022/ isbn.978-3-85448-047-1_4
-
[50]
Koray Tahiroğlu, Miranda Kastemaa, and Oskar Koli. 2021. AI-terity 2.0: An Autonomous NIME Featuring GANSpaceSynth Deep Learning Model. InProceedings of the International Conference on New Interfaces for Musical Expression. Shanghai, China, Article 80. https://doi.org/10.21428/92fbeb44.3d 0e9e12
-
[51]
Gabriel Vigliensoni, Phoenix Perry, and Rebecca Fiebrink. 2022. A small- data mindset for generative AI creative work. InProceedings of the CHI 2022 Workshop on Generative AI and HCI. https://doi.org/10.5281/zenodo.7086327
-
[52]
Federico Visi. 2024. The Sophtar: a networkable feedback string instrument with embedded machine learning. InProceedings of the International Conference on New Interfaces for Musical Expression, S M Astrid Bin and Courtney N. Reed (Eds.). Utrecht, Netherlands, Article 22, 7 pages. https://doi.org/10.5281/zeno do.13904810
-
[53]
Federico Ghelli Visi and Atau Tanaka. 2021. Interactive Machine Learning of Musical Gesture. InHandbook of Artificial Intelligence for Music: Foundations, Advanced Approaches, and Developments for Creativity. Springer, 771–798. https://doi.org/10.1007/978-3-030-72116-9_27
-
[54]
Weisz, Jessica He, Michael Muller, Gabriela Hoefer, Rachel Miles, and Werner Geyer
Justin D. Weisz, Jessica He, Michael Muller, Gabriela Hoefer, Rachel Miles, and Werner Geyer. 2024. Design Principles for Generative AI Applications. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA)(CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 378, 22 pages. https://doi.org/10...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.