The reviewed record of science sign in
Pith

arxiv: 2604.24321 · v1 · submitted 2026-04-27 · cs.HC · q-bio.NC

From Players to Participants: Citizen Science and Video Games to Understand Cognition

Syrine Salouhou , Edgar Dubourg , Maxwell Scott-Slade , Hugo Spiers , Antoine Coutrot This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-05-08 02:22 UTCgrok-4.3open to challenge →

classification cs.HC q-bio.NC
keywords citizen sciencevideo gamescognitive researchbehavioral data collectionecological validityscalabilitypublic engagementgame-based experiments
0
0 comments X

The pith

Citizen science video games turn players into participants by embedding cognitive experiments in engaging gameplay.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper reviews how video games can be used for citizen science in cognitive research. By embedding experimental tasks into games, scientists gain access to large and diverse groups of participants while collecting detailed data in everyday settings. This method offers advantages in scale and realism over traditional lab studies. It also encourages public involvement in science. The review discusses both the opportunities and the design challenges involved in making such games effective for research.

Core claim

The central claim is that citizen science video games bridge the gap between entertainment and scientific inquiry, allowing researchers to study cognition by transforming voluntary players into data contributors through carefully designed game experiences that maintain scientific rigor while delivering engaging play.

What carries the argument

Embedding controlled experimental tasks within the mechanics and narratives of video games to elicit and record cognitive behaviors from a broad audience.

Load-bearing premise

The data gathered from game-based tasks faithfully represents the underlying cognitive processes without significant influence from the game's entertainment features or user interface.

What would settle it

Observing substantial differences in task performance or response times when the identical cognitive experiment is run both as a standalone task and embedded within a game environment.

Figures

Figures reproduced from arXiv: 2604.24321 by Antoine Coutrot, Edgar Dubourg, Hugo Spiers, Maxwell Scott-Slade, Syrine Salouhou.

Figure 2
Figure 2. Figure 2: Cognitive Citizen Science Games : Sample Size as a function of starting year. For each game we show its logo, research area, sample size and whether it is a bespoke game or embedded in an existing game. These effects appear consistent across age groups and levels of gaming experience. Results from cognitive studies can also be implemented back in video games to design better experiences that make repetitiv… view at source ↗
read the original abstract

Citizen science is transforming how cognitive scientists study the human mind, and video games are at the heart of this shift. By embedding experimental tasks into engaging, game-like experiences, researchers can reach large, diverse populations while collecting rich behavioral data outside the lab. In this review, we explore how citizen science video games bridge the gap between players and participants, turning entertainment into large-scale cognitive research. Drawing on recent projects such as Sea Hero Quest and The Music Lab, we outline the key benefits of this approach: scalability, ecological validity, and public engagement. We also examine the challenges of designing games that are scientifically rigorous, ethically sound, and meaningful for both researchers and players. Through professional game developer insights, we highlight what it takes to develop a successful citizen science video game for cognitive science, and why this approach is still rare in the literature.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 3 minor

Summary. This review paper examines the use of citizen science video games as a tool for cognitive science research. It claims that embedding experimental tasks within engaging game mechanics allows researchers to recruit large, diverse participant pools and gather rich behavioral data in naturalistic settings outside the laboratory. Drawing on case studies such as Sea Hero Quest and The Music Lab, the manuscript outlines benefits including scalability, improved ecological validity, and enhanced public engagement, while also addressing design challenges related to scientific rigor, ethical considerations, and mutual value for players and researchers. It incorporates perspectives from professional game developers on development requirements and notes the relative rarity of such projects in the literature.

Significance. If the review's synthesis is accurate and balanced, the work could help cognitive scientists and game designers collaborate more effectively on large-scale studies. The emphasis on developer insights provides practical value that is often missing from purely academic discussions. However, the overall significance is limited by the descriptive nature of the analysis and the absence of new empirical validation for the core assumption that game-derived data maps reliably onto established cognitive constructs.

major comments (2)
  1. [Benefits discussion (Sea Hero Quest / The Music Lab examples)] The benefits section (discussing scalability and rich data collection via Sea Hero Quest and The Music Lab): the central claim that embedding tasks yields behavioral data that faithfully reflects underlying cognitive processes is load-bearing for the entire argument, yet the manuscript provides no quantitative validation (e.g., direct correlations between game metrics and matched laboratory tasks) or discussion of how game mechanics, motivation, or interface effects are isolated or corrected. This leaves the asserted advantages conditional on an untested equivalence assumption.
  2. [Challenges of designing games that are scientifically rigorous] The challenges section: while design challenges for scientific rigor are acknowledged, the manuscript does not evaluate whether existing projects have successfully mitigated threats to validity (such as self-selection bias or task distortion) with concrete evidence or metrics; without this, the review cannot fully substantiate that the approach is both scalable and rigorous.
minor comments (3)
  1. [Abstract and introduction] The abstract and introduction use the phrase 'rich behavioral data' without defining what richness means operationally (e.g., number of trials, granularity of timestamps, or diversity of measures).
  2. [Throughout (case study paragraphs)] References to external projects would benefit from more precise citations to the original papers or datasets rather than general mentions, to allow readers to locate the validation studies.
  3. [Developer insights subsection] The manuscript could clarify the target audience (cognitive scientists vs. game developers) to better frame the developer-insights subsection.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for their thoughtful and constructive review of our manuscript on citizen science video games for cognitive research. We address each major comment point by point below, clarifying the review's scope while agreeing to strengthen the discussion of existing evidence where possible.

read point-by-point responses

  1. Referee: [Benefits discussion (Sea Hero Quest / The Music Lab examples)] The benefits section (discussing scalability and rich data collection via Sea Hero Quest and The Music Lab): the central claim that embedding tasks yields behavioral data that faithfully reflects underlying cognitive processes is load-bearing for the entire argument, yet the manuscript provides no quantitative validation (e.g., direct correlations between game metrics and matched laboratory tasks) or discussion of how game mechanics, motivation, or interface effects are isolated or corrected. This leaves the asserted advantages conditional on an untested equivalence assumption.

    Authors: As a review paper, our manuscript synthesizes findings from the primary literature rather than generating new empirical data. The original publications on Sea Hero Quest report quantitative validations, including correlations between game-based spatial navigation metrics and established laboratory tasks (e.g., virtual water maze performance). Similar validation approaches appear in The Music Lab studies. We will revise the benefits section to include a new subsection explicitly summarizing these published correlations and addressing potential confounds such as motivation, interface design, and game mechanics, drawing directly on the cited works to make the equivalence assumptions more transparent and evidence-based. revision: yes

  2. Referee: [Challenges of designing games that are scientifically rigorous] The challenges section: while design challenges for scientific rigor are acknowledged, the manuscript does not evaluate whether existing projects have successfully mitigated threats to validity (such as self-selection bias or task distortion) with concrete evidence or metrics; without this, the review cannot fully substantiate that the approach is both scalable and rigorous.

    Authors: We agree that the challenges section would benefit from greater evaluation of mitigation success. The current draft discusses threats such as self-selection bias and task distortion and references strategies employed by projects like Sea Hero Quest (e.g., broad recruitment campaigns and post-hoc data quality filters). We will expand this section to incorporate concrete examples and available metrics from the literature on how these projects have assessed and reduced such biases. We note, however, that standardized quantitative benchmarks for mitigation effectiveness remain limited in this emerging field, which constrains the depth of evaluation a review can provide without overstepping the available evidence. revision: partial

standing simulated objections not resolved
  • The absence of new empirical validation for the core assumption that game-derived data maps reliably onto established cognitive constructs, as this would require original data collection and analysis outside the scope of a review paper.

Circularity Check

0 steps flagged

No circularity in this descriptive review paper

full rationale

The paper is a review article with no mathematical derivations, equations, fitted parameters, or predictive models. It draws on external projects such as Sea Hero Quest and The Music Lab as independent examples, without self-definitional reductions, fitted inputs renamed as predictions, or load-bearing self-citations that collapse the central claims to their own inputs. All load-bearing steps are supported by external references and general discussion rather than internal construction, making the text self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

As a review, the paper draws on existing literature and projects without introducing new free parameters, axioms, or invented entities.

pith-pipeline@v0.9.0 · 5454 in / 908 out tokens · 60093 ms · 2026-05-08T02:22:40.248374+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 27 canonical work pages

  1. [1]

    Abstract Citizen science is transforming how cognitive scientists study the human mind, and video games are at the heart of this shift

    Glitchers Ltd., Edinburgh, Scotland, United Kingdom. Abstract Citizen science is transforming how cognitive scientists study the human mind, and video games are at the heart of this shift. By embedding experimental tasks into engaging, game-like experiences, researchers can reach large, diverse populations while collecting rich behavioral data outside the...

    work page 1900

  2. [2]

    More recently, cognitive scientists have begun to adopt similar approaches (Van den Bussche et al., 2024, 2026)

    or astronomy (Lintott et al., 2008). More recently, cognitive scientists have begun to adopt similar approaches (Van den Bussche et al., 2024, 2026). A distinctive feature of cognitive science is that the participants’ behavior is precisely the subject of the research. This specificity raises methodological and ethical considerations, particularly regardi...

    work page 2008

  3. [3]

    This last population has been derived from 311 samples of humans used in studies published between 2015 and 2019 in evolutionary psychology journals (Pollet & Saxton 2019)

    and participants to psychology experiments. This last population has been derived from 311 samples of humans used in studies published between 2015 and 2019 in evolutionary psychology journals (Pollet & Saxton 2019). several causes, including publication bias, researcher bias, and misuse of statistics. Several structural changes have been implemented by s...

    work page 2015

  4. [4]

    Skill Lab, for instance, a mobile video game assessing diverse cognitive skills (e.g., working memory, reasoning) through engaging tasks, collected over 10,000 users’ data

    exemplify how gamified cognitive research can leverage mobile platforms to gather rich data sets. Skill Lab, for instance, a mobile video game assessing diverse cognitive skills (e.g., working memory, reasoning) through engaging tasks, collected over 10,000 users’ data. Similarly, Sea Hero Quest, a spatial orientation video game available on smartphones c...

    work page 2025

  5. [5]

    Although video gamers are not a perfect representation of the population, with some underrepresented demographics (see below), they are a significant improvement from our small in-lab samples, mainly made of college students. The Moral Machine, a game where players make moral decision, is a great example of diversity sampling, the platform gathered 40 mil...

    work page 2018

  6. [6]

    the use of game-design elements in non-game contexts

    is a good example. This project combines cognitive games and environmental data to study the impact of air pollution on brain functions, and identified a correlation between chronic NO2 exposure and reduced cognitive performance. Improving data quality Beyond participation, gamification can improve data quality. Gamified tasks are associated with greater ...

    work page 2018

  7. [7]

    game for good

    Spatial Cognition 2016 Spatial trajectories Wayfinding + Path Integration Global (all countries) 4.3 M Bespoke mobile app Game developers Industrial sponsorship Large-scale “game for good” campaign Which English? (Hartshorne et al.,

    work page 2016

  8. [8]

    English grammar variation 2009 Linguistic judgments Rate grammaticality of English sentences Native and non-native English speakers 670,000 Bespoke online platform Academics Gov + academic grants Social media engagement The Music Lab (Mehr et al.,

    work page 2009

  9. [9]

    Cross-cultural music cognition 2019 Beat, tuning, melody perception Online musical tasks Global 2.7 M Online platform Academics Gov + academic grants Broad dissemination via website and media CitieS-Health (Gignac et al.,

    work page 2019

  10. [10]

    Air pollution & cognition 2020 Cognitive & mental health tests, GPS Stroop-like tasks Barcelona residents 288 Bespoke mobile app Academics EU Horizon 2020 Local campaigns, health orgs, social media Little Alchemy 2 (Brändle et al.,

    work page 2020

  11. [11]

    Exploration behavior & empowerment 2019 Gameplay logs Combine elements to form complex objects Global 29,493 Existing commercial game Game developers Academic + industry Publication-driven outreach Skill Lab (Pedersen et al.,

    work page 2019

  12. [12]

    Cognitive ability profiling 2018 Cognitive tests Six mini-games + 14 validated tasks Danish participants 10,725 Bespoke app Game developers + Academics ERC H2020 + industry TV collaboration, school outreach Neureka (Rosická et al.,

    work page 2018

  13. [13]

    Brain health & mental risk screening 2020 Cognitive tasks + demographics Mini games on working memory & planning Global (54 countries) 9,918 Bespoke app Academics Gov + academic grants TV/radio ads, SciStarter, paid sub-studies The Great Brain Experiment (Brown et al.,

    work page 2020

  14. [14]

    Smartphone-based cognitive phenotyping 2013 Cognitive + demographic data Inhibition, visual & working memory Global 100,000 Bespoke app Game developers + Academics Academic grants Twitter, media blogs (e.g., WSJ Speakeasy) Moral Machine (Awad.,

    work page 2013

  15. [15]

    Ethics of autonomous vehicles 2016 Moral decisions Choose outcomes in moral dilemmas Global (all countries) 2.3 M Bespoke online platform Academics Academic grants Media coverage, TEDx talks, publications Hexxed.io (Quendera et al.,

    work page 2016

  16. [16]

    human learning process 2022 Interaction logs (tap+swipe) no instructions, find the patterns that maximize the number of points Global 10,000 Bespoke app Academics Academic grant Local campaigns, publications, social media follow a game-first model (focused on gameplay, e.g., Sea Hero Quest, Spiers et al.,

    work page 2022

  17. [17]

    gamification doesn’t work for science

    or a problem-first model (centered on scientific challenges with narrative elements, e.g., Foldit, Cooper et al., 2010). Once the model is chosen, development can proceed through partnerships with existing platforms, professional game studios, or internal teams. Interdisciplinary collaborations with fields like educational science or behavioral economics ...

    work page 2010

  18. [18]

    These results suggest that less experienced gamers, who likely represent a large proportion of the general public, may generate noisier or less reliable data in game-based paradigms, potentially introducing unnoticed biases into large-scale cognitive datasets. To address this, validation strategies comparing citizen-generated data to established experimen...

    work page 2021

  19. [19]

    gamification

    Brändle, F., Stocks, L. J., Tenenbaum, J. B., Gershman, S. J., & Schulz, E. (2023). Empowerment contributes to exploration behaviour in a creative video game. Nature Human Behaviour, 7(9), 1481–1489. Brouwer, S., & Hessels, L. K. (2019). Increasing research impact with citizen science: The influence of recruitment methods on sample composition and data qu...

    work page doi:10.1016/j.cub.2018.06.009 2023

  20. [20]

    M., Di Iorio, A., & Vitali, F

    Genovese, F., Bolognesi, M. M., Di Iorio, A., & Vitali, F. (2024). The advantages of gamification for collecting linguistic data: A case study using Word Ladders. Online Journal of Communication and Media Technologies, 14(2), 1–17. Gignac, F., Righi, V., Toran, R., Errandonea, L. P., Ortiz, R., Mijling, B., ... & Basagaña, X. (2022). Short-term NO2 exposu...

    work page 2024

  21. [21]

    K., Tenenbaum, J

    Hartshorne, J. K., Tenenbaum, J. B., & Pinker, S. (2018). A critical period for second language acquisition: Evidence from 2/3 million English speakers. Cognition, 177, 263-277. Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83. Heyen, N. B., Gardecki, J., Eidt-Koch, D., Sch...

    work page 2018

  22. [22]

    https://newzoo.com/resources/trend-reports/newzoo-global-games-market-report-2025 (retrieved March

    work page 2025

  23. [23]

    Nugent, J. (2021). Citizen science: Accelerating Alzheimer's research with Stall Catchers. The Science Teacher, 88(4), 16–17. Pandya, R. E. (2012). A framework for engaging diverse communities in citizen science in the US. Frontiers in Ecology and the Environment, 10(6), 314–317. Pedersen, M. K., Díaz, C. M. C., Wang, Q. J., Alba-Marrugo, M. A., Amidi, A....

    work page arXiv 2021

  24. [24]

    Rodd, J. M. (2024). Moving experimental psychology online: How to obtain high quality data when we can't see our participants. Journal of Memory and Language, 134, 104472. Rosická, A. M., Teckentrup, V., Fittipaldi, S., Ibanez, A., Pringle, A., Gallagher, E., ... & Gillan, C. M. (2024). Modifiable dementia risk factors associated with objective and subjec...

    work page 2024

  25. [25]

    A., Hughes, G., & Reynvoet, B

    Van den Bussche, E., Verhaegen, K. A., Hughes, G., & Reynvoet, B. (2024). Towards a cognitive citizen science. Nature Reviews Psychology, 3, 1–2. Van den Bussche, E., Verhaegen, K. A., Reynvoet, B., & Hughes, G. (2026). Citizen science in psychology: Challenges, opportunities, and recommendations. Behavior Research Methods, 58(5),

    work page 2024

  26. [26]

    F., Coutrot, A., & Spiers, H

    Velasco, P. F., Coutrot, A., & Spiers, H. J. (2026). Human Navigation Behaviour and Brain Dynamics in Real-world Contexts. arXiv preprint arXiv:2603.11347. Vergara-Borge, F., López-de-Ipiña, D., Emaldi, M., Olivares-Rodríguez, C., Khan, Z., & Soomro, K. (2025). Gamifying engagement in spatial crowdsourcing: An exploratory mixed-methods study on gamificati...

    work page arXiv 2026

  27. [27]

    Vigliocco, G., Convertino, L., De Felice, S., Gregorians, L., Kewenig, V., Mueller, M. A., ... & Spiers, H. J. (2024). Ecological brain: Reframing the study of human behaviour and cognition. Royal Society Open Science, 11(11), 240762. Vohland, K., Land-Zandstra, A., Ceccaroni, L., Lemmens, R., Perelló, J., Pontier, M., ... & Wehn, U. (Eds.). (2021). The s...

    work page 2024