pith. sign in

arxiv: 1907.01154 · v1 · pith:FFDIQQJ7new · submitted 2019-07-02 · 💻 cs.MM · cs.AI· cs.SD· eess.AS

Adaptive Music Composition for Games

Patrick Hutchings , Jon McCormack This is my paper

Pith reviewed 2026-05-25 10:53 UTC · model grok-4.3

classification 💻 cs.MM cs.AIcs.SDeess.AS
keywords adaptive musicvideo gamesimmersionmulti-agent compositioncognitive modelsgame soundtracksemotional affect
0
0 comments X

The pith

An adaptive music system combining cognitive models and multi-agent composition increases reported immersion and music-concept correlation in games.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that adaptive game music requires simultaneous advances in real-time generation algorithms and in modeling player actions, world context, and emotion. It builds an Adaptive Music System that fuses cognitive models of knowledge organisation and emotional affect with multi-modal, multi-agent composition methods. The system is embedded in two stylistically different games. Players then rate the AMS versions higher than the original soundtracks on immersion and on how well the music matches game-world ideas. A sympathetic reader would care because the result points to a concrete route for making game audio more responsive and emotionally coherent without requiring entirely new hardware or manual scoring.

Core claim

The Adaptive Music System integrates cognitive models of knowledge organisation and emotional affect with multi-modal, multi-agent composition techniques; when placed inside two stylistically distinct games, it produces music that gamers rate as yielding higher overall immersion and stronger correlation with game-world concepts than the games' original soundtracks.

What carries the argument

The Adaptive Music System (AMS) that couples cognitive models of knowledge and emotion with multi-modal, multi-agent composition to drive real-time music adaptation.

If this is right

  • Real-time music generation can be made responsive to both player actions and inferred emotional state within a single system.
  • The same architecture works across games with different visual styles and mechanics.
  • Cognitive models of knowledge organisation can be used to guide musical structure so that themes align with in-game concepts.
  • Player self-reports can serve as an initial validation metric for adaptive music designs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approach could be tested with physiological signals instead of questionnaires to check whether reported immersion tracks objective arousal or attention measures.
  • Similar cognitive-plus-multi-agent methods might be applied to non-game interactive media such as museum exhibits or training simulations.
  • If the modeling of context and emotion scales, future games could generate entirely new musical material rather than rearranging pre-composed stems.

Load-bearing premise

Differences in player ratings arise from the AMS design itself rather than from novelty, demand effects, or uncontrolled differences in the music tracks.

What would settle it

A follow-up study that holds music content constant, uses blinded listening conditions, and finds no reliable difference in immersion or concept-correlation scores would falsify the central claim.

Figures

Figures reproduced from arXiv: 1907.01154 by Jon McCormack, Patrick Hutchings.

Figure 1
Figure 1. Figure 1: Architecture of the Adaptive Music System (AMS) for modelling game-world context and generating context-specific music. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Visualisation of spreading activation model during testing with a role [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: System Diagram for the AMS. new vertices, new edges, or update edge weights. An OSC [52] client was added to the game engines to communicate the game state as lists of content activations and relationships for this purpose. Every 30ms, the list of messages received by the OSC server is used to update the activation values of vertices in the graph. If a message showing activation of a concept is received an… view at source ↗
read the original abstract

The generation of music that adapts dynamically to content and actions has an important role in building more immersive, memorable and emotive game experiences. To date, the development of adaptive music systems for video games is limited by both the nature of algorithms used for real-time music generation and the limited modelling of player action, game world context and emotion in current games. We propose that these issues must be addressed in tandem for the quality and flexibility of adaptive game music to significantly improve. Cognitive models of knowledge organisation and emotional affect are integrated with multi-modal, multi-agent composition techniques to produce a novel Adaptive Music System (AMS). The system is integrated into two stylistically distinct games. Gamers reported an overall higher immersion and correlation of music with game-world concepts with the AMS than with the original game soundtracks in both games.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper proposes an Adaptive Music System (AMS) that integrates cognitive models of knowledge organisation and emotional affect with multi-modal, multi-agent composition techniques to enable dynamic, context-aware game music. The AMS is integrated into two stylistically distinct games; the central empirical claim is that a user study found gamers reported higher immersion and stronger correlation between music and game-world concepts with the AMS than with the original soundtracks.

Significance. If the user-study results can be shown to reflect genuine design improvements rather than confounds, the work would address a recognised limitation in game audio by jointly tackling algorithmic generation and richer modelling of player action, context and emotion; this could support more immersive adaptive music in commercial titles.

major comments (2)
  1. [Abstract / User Study] Abstract and user-study section: the manuscript states that 'gamers reported an overall higher immersion' but supplies no information on participant count, study design, blinding, counterbalancing, statistical tests, instructions, or controls for prior game familiarity; without these the central claim cannot be evaluated.
  2. [User Study] User-study reporting: no details are given on how the AMS condition was compared to the original soundtracks (e.g., same game levels, music volume matching, or implementation differences), leaving open the possibility that reported preferences reflect uncontrolled variables rather than the proposed cognitive/multi-agent architecture.
minor comments (1)
  1. [Abstract] The abstract introduces the acronym AMS without first spelling it out on first use.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their careful reading and constructive feedback on the user study reporting. We agree that additional methodological details are required to allow proper evaluation of the empirical claims and will revise the manuscript to address both major comments.

read point-by-point responses

  1. Referee: [Abstract / User Study] Abstract and user-study section: the manuscript states that 'gamers reported an overall higher immersion' but supplies no information on participant count, study design, blinding, counterbalancing, statistical tests, instructions, or controls for prior game familiarity; without these the central claim cannot be evaluated.

    Authors: We acknowledge that the current manuscript provides insufficient detail on the user study methodology. In the revised version we will expand the relevant section (and update the abstract if space permits) to report participant count, within- or between-subjects design, blinding procedures, counterbalancing, the statistical tests employed, the exact instructions given to participants, and any controls or screening for prior game familiarity. revision: yes

  2. Referee: [User Study] User-study reporting: no details are given on how the AMS condition was compared to the original soundtracks (e.g., same game levels, music volume matching, or implementation differences), leaving open the possibility that reported preferences reflect uncontrolled variables rather than the proposed cognitive/multi-agent architecture.

    Authors: We agree that the comparison protocol must be described more precisely. The revised manuscript will clarify that identical game levels and segments were used in both conditions, how audio levels were matched across conditions, and any other implementation differences between the AMS and the original soundtrack, thereby reducing the chance that results are attributable to uncontrolled variables. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical user study with no derivation chain

full rationale

The paper describes an Adaptive Music System (AMS) integrating cognitive models with multi-agent composition techniques, then reports empirical user study results on immersion and music correlation in two games. No equations, fitted parameters, predictions derived from inputs, or self-citation load-bearing steps are present in the abstract or described claims. The central claim rests on participant reports rather than any reduction of a result to its own inputs by construction. This is a standard empirical evaluation without the circular patterns enumerated.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

Abstract-only review; the ledger is therefore minimal and based solely on statements in the abstract.

axioms (1)
  • domain assumption Cognitive models of knowledge organisation and emotional affect can be usefully integrated with multi-modal multi-agent composition techniques
    Invoked in the abstract as the foundation for producing the AMS
invented entities (1)
  • Adaptive Music System (AMS) no independent evidence
    purpose: To generate music that adapts dynamically to game content, player actions, and emotion
    Presented as a novel constructed system in the abstract

pith-pipeline@v0.9.0 · 5662 in / 1348 out tokens · 26458 ms · 2026-05-25T10:53:30.965529+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

61 extracted references · 61 canonical work pages · 2 internal anchors

  1. [1]

    Emergent narra- tive, past, present and future of an interactive storytelling approach,

    S. J.-J. Louchart, J. Truesdale, N. Suttie, and R. Aylett, “Emergent narra- tive, past, present and future of an interactive storytelling approach,” in Interactive Digital Narrative: History, Theory and Practice. Routledge, 2015, pp. 185–200

    work page 2015

  2. [2]

    The cognitive processing of film and musical soundtracks,

    M. G. Boltz, “The cognitive processing of film and musical soundtracks,” Memory & Cognition , vol. 32, no. 7, pp. 1194–1205, 2004

    work page 2004

  3. [3]

    Time perception, immersion and music in videogames,

    T. Sanders and P. Cairns, “Time perception, immersion and music in videogames,” in Proceedings of the 24th BCS interaction specialist group conference. British Computer Society, 2010, pp. 160–167

    work page 2010

  4. [4]

    Music structure and emotional response: Some empirical findings,

    J. A. Sloboda, “Music structure and emotional response: Some empirical findings,”Psychology of music , vol. 19, no. 2, pp. 110–120, 1991

    work page 1991

  5. [5]

    A cognitive approach to the emotional function of game sound,

    I. Ekman, “A cognitive approach to the emotional function of game sound,” in The Oxford Handbook of Interactive Audio , 2014. IEEE TRANSACTIONS ON GAMES (PREPRINT) 10

    work page 2014

  6. [6]

    Experience-driven procedural content generation,

    G. N. Yannakakis and J. Togelius, “Experience-driven procedural content generation,” IEEE Transactions on Affective Computing , vol. 2, no. 3, pp. 147–161, 2011

    work page 2011

  7. [7]

    Shaker, J

    N. Shaker, J. Togelius, and M. J. Nelson, Procedural content generation in games. Springer, 2016

    work page 2016

  8. [8]

    Red dead redemption,

    “Red dead redemption,” Rockstar Games, 2010

    work page 2010

  9. [9]

    Red dead redemption - making of music vidoc

    XboxViewTV , “Red dead redemption - making of music vidoc.”

    work page

  10. [10]

    No man’s sky,

    “No man’s sky,” Hello Games, 2016

    work page 2016

  11. [11]

    How ‘no man’s sky’ composes completely original music for every player,

    M. Epstein, “How ‘no man’s sky’ composes completely original music for every player,” Digital Trends , August 2016. [Online]. Available: https://www.digitaltrends.com/gaming/no-mans-sky-music/

    work page 2016

  12. [12]

    A spreading-activation theory of semantic processing

    A. M. Collins and E. F. Loftus, “A spreading-activation theory of semantic processing.” Psychological review, vol. 82, no. 6, p. 407, 1975

    work page 1975

  13. [13]

    Grimshaw, Game Sound Technology and Player Interaction: Con- cepts and Developments: Concepts and Developments

    M. Grimshaw, Game Sound Technology and Player Interaction: Con- cepts and Developments: Concepts and Developments . IGI Global, 2010

    work page 2010

  14. [14]

    An introduction to procedural audio and its application in computer games,

    A. Farnell, “An introduction to procedural audio and its application in computer games,” in Audio mostly conference , vol. 23, 2007

    work page 2007

  15. [15]

    Sonancia: Sonification of procedurally generated game levels,

    P. Lopes, A. Liapis, and G. N. Yannakakis, “Sonancia: Sonification of procedurally generated game levels,” in Proceedings of the 1st computational creativity and games workshop , 2015

    work page 2015

  16. [16]

    The soundtrack of your mind: mind music-adaptive audio for game characters,

    M. Eladhari, R. Nieuwdorp, and M. Fridenfalk, “The soundtrack of your mind: mind music-adaptive audio for game characters,” in Proceedings of the 2006 ACM SIGCHI international conference on Advances in computer entertainment technology . ACM, 2006, p. 54

    work page 2006

  17. [17]

    Au- dioinspace: Exploring the creative fusion of generative audio, visuals and gameplay,

    A. K. Hoover, W. Cachia, A. Liapis, and G. N. Yannakakis, “Au- dioinspace: Exploring the creative fusion of generative audio, visuals and gameplay,” in International Conference on Evolutionary and Bio- logically Inspired Music and Art . Springer, 2015, pp. 101–112

    work page 2015

  18. [18]

    Audioverdrive: Exploring bidirectional communication between music and gameplay,

    N. I. Holtar, M. J. Nelson, and J. Togelius, “Audioverdrive: Exploring bidirectional communication between music and gameplay,” in ICMC. Citeseer, 2013

    work page 2013

  19. [19]

    Towards an emotion-driven adaptive system for video game music,

    M. L. Ib ´a˜nez, N. ´Alvarez, and F. Peinado, “Towards an emotion-driven adaptive system for video game music,” in International Conference on Advances in Computer Entertainment . Springer, 2017, pp. 360–367

    work page 2017

  20. [20]

    Affective evolutionary music composition with metacompose,

    M. Scirea, J. Togelius, P. Eklund, and S. Risi, “Affective evolutionary music composition with metacompose,” Genetic Programming and Evolvable Machines, vol. 18, no. 4, pp. 433–465, 2017

    work page 2017

  21. [21]

    Howard shore discusses the passion-play within the twilight saga: Eclipse score,

    M. Morton, “Howard shore discusses the passion-play within the twilight saga: Eclipse score,” https://bit.ly/2SpYWst, 2010

    work page 2010

  22. [22]

    Special issue on evolutionary music,

    F. F. de Vega, C. Cotta, and E. R. Miranda, “Special issue on evolutionary music,” Berlin, 2012

    work page 2012

  23. [23]

    E. R. Miranda and J. Al Biles, Evolutionary computer music. Springer, 2007

    work page 2007

  24. [24]

    Classifier fitness based on accuracy,

    S. W. Wilson, “Classifier fitness based on accuracy,” Evolutionary computation, vol. 3, no. 2, pp. 149–175, 1995

    work page 1995

  25. [25]

    Artificial ecosystems for creative discovery,

    J. McCormack, “Artificial ecosystems for creative discovery,” in Pro- ceedings of the 9th annual conference on Genetic and evolutionary computation. ACM, 2007, pp. 301–307

    work page 2007

  26. [26]

    Neural network music composition by prediction: Exploring the benefits of psychoacoustic constraints and multi-scale processing,

    M. C. Mozer, “Neural network music composition by prediction: Exploring the benefits of psychoacoustic constraints and multi-scale processing,” Connection Science, vol. 6, no. 2-3, pp. 247–280, 1994

    work page 1994

  27. [27]

    Ai methods for algorithmic composi- tion: A survey, a critical view and future prospects,

    G. Papadopoulos and G. Wiggins, “Ai methods for algorithmic composi- tion: A survey, a critical view and future prospects,” in AISB Symposium on Musical Creativity . Edinburgh, UK, 1999, pp. 110–117

    work page 1999

  28. [28]

    Generative algorithms for making music: Emergence, evolution, and ecosystems,

    J. McCormack, A. C. Eldridge, A. Dorin, and P. McIlwain, “Generative algorithms for making music: Emergence, evolution, and ecosystems,” in The Oxford Handbook of Computer Music , R. T. Dean, Ed. New York; Oxford: Oxford University Press, 2009, pp. 354–379

    work page 2009

  29. [29]

    A first look at music composition using LSTM recurrent neural networks,

    D. Eck and J. Schmidhuber, “A first look at music composition using LSTM recurrent neural networks,” Istituto Dalle Molle Di Studi Sull Intelligenza Artificiale, vol. 103, 2002

    work page 2002

  30. [30]

    Music transcription modelling and composition using deep learning

    B. L. Sturm, J. F. Santos, O. Ben-Tal, and I. Korshunova, “Music transcription modelling and composition using deep learning,” arXiv preprint arXiv:1604.08723, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  31. [31]

    Wavenet: A generative model for raw audio

    A. Van Den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. W. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio.” in SSW, 2016, p. 125

    work page 2016

  32. [32]

    Toward a theory of interactive media effects (time),

    S. S. Sundar, H. Jia, T. F. Waddell, and Y . Huang, “Toward a theory of interactive media effects (time),” The handbook of the psychology of communication technology, pp. 47–86, 2015

    work page 2015

  33. [33]

    Open design challenges for interactive emergent narrative,

    J. O. Ryan, M. Mateas, and N. Wardrip-Fruin, “Open design challenges for interactive emergent narrative,” in International Conference on Interactive Digital Storytelling . Springer, 2015, pp. 14–26

    work page 2015

  34. [34]

    A theoretical model of the effects and consequences of playing video games,

    K. E. Buckley and C. A. Anderson, “A theoretical model of the effects and consequences of playing video games,” Playing video games: Motives, responses, and consequences , pp. 363–378, 2006

    work page 2006

  35. [35]

    Priming and search processes in semantic memory: A test of three models of spreading activation,

    R. F. Lorch, “Priming and search processes in semantic memory: A test of three models of spreading activation,” Journal of verbal learning and verbal behavior, vol. 21, no. 4, pp. 468–492, 1982

    work page 1982

  36. [36]

    Category norms of verbal items in 56 categories a replication and extension of the connecticut category norms

    W. F. Battig and W. E. Montague, “Category norms of verbal items in 56 categories a replication and extension of the connecticut category norms.” Journal of experimental Psychology , vol. 80, p. 1, 1969

    work page 1969

  37. [37]

    Words, pictures, and priming: on semantic activation, conscious identification, and the automaticity of information processing,

    T. H. Carr, C. McCauley, R. D. Sperber, and C. Parmelee, “Words, pictures, and priming: on semantic activation, conscious identification, and the automaticity of information processing,” Journal of Experimen- tal Psychology: Human Perception and Performance , vol. 8, no. 6, pp. 757–777, 1982

    work page 1982

  38. [38]

    Priming in item recognition: Evidence for the propositional structure of sentences,

    R. Ratcliff and G. McKoon, “Priming in item recognition: Evidence for the propositional structure of sentences,” Journal of verbal learning and verbal behavior, vol. 17, no. 4, pp. 403–417, 1978

    work page 1978

  39. [39]

    Mood and memory

    G. H. Bower, “Mood and memory.” American psychologist , vol. 36, no. 2, p. 129, 1981

    work page 1981

  40. [40]

    Affective gaming: measuring emotion through the gamepad,

    J. Sykes and S. Brown, “Affective gaming: measuring emotion through the gamepad,” in CHI’03 extended abstracts on Human factors in computing systems. ACM, 2003, pp. 732–733

    work page 2003

  41. [41]

    Emotion, content, and context in sound and music,

    S. Cunningham, V . Grout, and R. Picking, “Emotion, content, and context in sound and music,” in Game sound technology and player interaction: Concepts and developments . IGI Global, 2011, pp. 235– 263

    work page 2011

  42. [42]

    Emotion perceived and emotion felt: Same or differ- ent?

    A. Gabrielsson, “Emotion perceived and emotion felt: Same or differ- ent?” Musicae Scientiae, vol. 5, no. 1 suppl, pp. 123–147, 2001

    work page 2001

  43. [43]

    Expression, perception, and induction of musical emotions: A review and a questionnaire study of everyday listening,

    P. N. Juslin and P. Laukka, “Expression, perception, and induction of musical emotions: A review and a questionnaire study of everyday listening,” Journal of New Music Research , vol. 33, no. 3, pp. 217–238, 2004

    work page 2004

  44. [44]

    Emotions evoked by the sound of music: characterization, classification, and measurement

    M. Zentner, D. Grandjean, and K. R. Scherer, “Emotions evoked by the sound of music: characterization, classification, and measurement.” Emotion, vol. 8, no. 4, p. 494, 2008

    work page 2008

  45. [45]

    The emotionality of sonic events: testing the geneva emotional music scale (GEMS) for popular and electroacoustic music,

    A. Lykartsis, A. Pysiewicz, H. von Coler, and S. Lepa, “The emotionality of sonic events: testing the geneva emotional music scale (GEMS) for popular and electroacoustic music,” in The 3rd International Conference on Music & Emotion, Jyv ¨askyl¨a, Finland, June 11-15, 2013. University of Jyv ¨askyl¨a, Department of Music, 2013

    work page 2013

  46. [46]

    What does music express? basic emotions and beyond,

    P. N. Juslin, “What does music express? basic emotions and beyond,” Frontiers in psychology, vol. 4, p. 596, 2013

    work page 2013

  47. [47]

    A circumplex model of affect

    J. A. Russell, “A circumplex model of affect.” Journal of personality and social psychology , vol. 39, no. 6, p. 1161, 1980

    work page 1980

  48. [48]

    Mixed affective responses to music with conflicting cues,

    P. G. Hunter, E. G. Schellenberg, and U. Schimmack, “Mixed affective responses to music with conflicting cues,” Cognition & Emotion, vol. 22, no. 2, pp. 327–352, 2008

    work page 2008

  49. [49]

    A comparison of the discrete and dimensional models of emotion in music,

    T. Eerola and J. K. Vuoskoski, “A comparison of the discrete and dimensional models of emotion in music,” Psychology of Music, vol. 39, no. 1, pp. 18–49, 2011

    work page 2011

  50. [50]

    Predictive modeling of ex- pressed emotions in music using pairwise comparisons,

    J. Madsen, B. S. Jensen, and J. Larsen, “Predictive modeling of ex- pressed emotions in music using pairwise comparisons,” in International Symposium on Computer Music Modeling and Retrieval . Springer, 2012, pp. 253–277

    work page 2012

  51. [51]

    Example of the glicko-2 system,

    M. E. Glickman, “Example of the glicko-2 system,” Boston University, 2012

    work page 2012

  52. [52]

    Open sound control: an enabling technology for musical networking,

    M. Wright, “Open sound control: an enabling technology for musical networking,” Organised Sound, vol. 10, no. 3, pp. 193–200, 2005

    work page 2005

  53. [53]

    A three cycle view of design science research,

    A. R. Hevner, “A three cycle view of design science research,” Scandi- navian journal of information systems , vol. 19, no. 2, p. 4, 2007

    work page 2007

  54. [54]

    Group creativity: Musical performance and collabora- tion,

    R. K. Sawyer, “Group creativity: Musical performance and collabora- tion,” Psychology of Music , vol. 34, no. 2, pp. 148–165, 2006

    work page 2006

  55. [55]

    Using autonomous agents to impro- vise music compositions in real-time,

    P. Hutchings and J. McCormack, “Using autonomous agents to impro- vise music compositions in real-time,” in International Conference on Evolutionary and Biologically Inspired Music and Art . Springer, 2017, pp. 114–127

    work page 2017

  56. [56]

    The real book - volume vi,

    H. L. Corp, “The real book - volume vi,” 2017

    work page 2017

  57. [57]

    A note on two problems in connexion with graphs,

    E. W. Dijkstra, “A note on two problems in connexion with graphs,” Numerische mathematik, vol. 1, no. 1, pp. 269–271, 1959

    work page 1959

  58. [58]

    Talking Drums: Generating drum grooves with neural networks

    P. Hutchings, “Talking drums: Generating drum grooves with neural networks,” arXiv preprint arXiv:1706.09558 , 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  59. [59]

    Zelda: Mystery of solarus,

    “Zelda: Mystery of solarus,” Solarus Team, 2011

    work page 2011

  60. [60]

    Starcraft ii: Wings of liberty,

    “Starcraft ii: Wings of liberty,” Blizzard Entertainment, 2010

    work page 2010

  61. [61]

    Evaluating musical foreshadowing of videogame narrative experiences,

    M. Scirea, Y .-G. Cheong, M. J. Nelson, and B.-C. Bae, “Evaluating musical foreshadowing of videogame narrative experiences,” inProceed- ings of the 9th Audio Mostly: A Conference on Interaction With Sound . ACM, 2014, p. 8

    work page 2014