Pre-Release Experimentation in Indie Game Development: An Interview Survey

Elizabeth Bjarnason; Fabian Fagerholm; Johan Lin{\aa}ker

REVIEW 3 major objections 5 minor 1 cited by

Pre-Release Experimentation in Indie Game Development: An Interview Survey

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read From ten indie-studio interviews, this paper proposes an emerging continuous-experimentation framework with five parts, and claims that pre-release experimentation is centered on qualitative data from observation and playtesting.

desk verdict Useful inductive framework for indie pre-release experimentation, honestly reported, but the single-informant sample and hidden interview guide mean the five parts are provisional. read the letter →

arxiv 2411.17183 v1 pith:FGLQSYN3 submitted 2024-11-26 cs.SE

Johan Lin{aa}ker , Elizabeth Bjarnason , Fabian Fagerholm This is my paper · ORCID

classification cs.SE

keywords continuousexperimentationindiegamedevelopmentuserresearchplaytestingqualitativedatapre-releaseinterviewsurveyframework

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that pre-release experimentation in indie game development can be captured in a single emerging framework with five parts: goal definition, design strategy, experiment object, sampling strategy, and execution strategy. Based on interviews with ten indie developers, it argues that before a game is released, experimentation is centered on qualitative data, with observation of play and playtesting doing most of the work, rather than on large-scale quantitative telemetry. The paper also finds that time and resource limits constrain how many participants indie studios can reach, and that these studios manage bias and representativeness with small, carefully chosen samples. A sympathetic reader would care because most software experimentation frameworks assume abundant user data, whereas indie studios must validate game ideas with very little; the proposed framework describes how experimentation can still function under that constraint and may extend to larger game companies and other software-intensive organizations.

What carries the argument

The load-bearing object is the five-part CE framework for game development itself. It functions as a classification and planning vocabulary: each part names a decision an indie studio must make when running an experiment, and the interview data supplies the options observed for each part, such as split/sequential/exploratory testing for design strategy and internal team/friends and family/publishers/external unknowns/communities/streamers for sampling. The framework's role is to turn scattered qualitative interview findings into a reusable structure that can be checked against future cases.

What would settle it

A replication with a larger, geographically varied sample of indie studios, coded without prior knowledge of the five categories, would falsify the framework if the categories did not emerge or if most studios reported quantitative telemetry rather than qualitative playtesting as their primary pre-release data source.

Watch

Extended reading notes

Core claim

The central discovery is an emerging continuous-experimentation framework for game development, derived from ten interviews. The framework says that any pre-release experiment in an indie studio can be planned and understood through five key parts: goal definition combines the experiment's purpose (scoping a game idea or a feature) with the game aspect being evaluated (aesthetics, mechanics, fun factor, or understandability); design strategy chooses among split, sequential, and exploratory testing; experiment object selects the medium (sketches, video, playable minimum viable game) and refinement level (conceptual, functional, content); sampling strategy picks participant demographics and numbers while weighing bias and representativeness; and execution strategy collects feedback mainly through observation, screen recordings, surveys, and occasional interviews. Across these parts, the study finds that pre-release experimentation runs primarily on qualitative data from a small number of participants, with split testing concentrated in early ideation and prototyping, and sequential and exploratory testing taking over as development matures.

Load-bearing premise

The framework rests on the assumption that one roughly 30-minute online interview per company, with developers recruited at a single indie-game conference and mostly based in Sweden, gives an accurate picture of how that studio experiments before release.

Editorial extensions

If this is right

Pre-release indie experimentation is primarily a qualitative activity: observing people play and recording sessions gives useful feedback even with 8-10 participants, so studios need not wait for large user bases to validate ideas.
Using the five parts as a checklist can help a studio see which decision is weak; for example, a playtest with unclear goals or an unrepresentative sample can be diagnosed as a sampling or goal-definition problem.
Split (A/B) testing is most affordable in early ideation and prototyping, while sequential and exploratory testing fit later stages, which reverses the common assumption that A/B testing belongs mainly to mature products.
Since pre-release experimentation is centered on qualitative data, quantitative telemetry remains a later-stage or mobile-game practice whose tooling costs currently block small studios.
If the framework holds, larger game companies and other software-intensive organizations with limited early user data can adopt the same five-part structure for pre-release experimentation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The five parts amount to a domain-specific restatement of a general experimental-design checklist; an implicit next step would be to test whether the same five parts organize pre-release experimentation in non-game software startups with scarce user data.
The interviews suggest an implicit maturity path in sampling: studios move from internal teams and friends/family toward dedicated communities and external unknowns; a testable extension would be whether studios that reach community and streamer samples make better feature decisions.
Because observation and screen recordings carry most of the evidence, cheap remote-playtest tooling that records sessions and annotates events could lower the cost barrier and make sequential experimentation more systematic in small studios.
A hypothesis-driven comparison is left untested: the data describe exploratory open-minded playtesting as common, but do not establish whether explicitly writing hypotheses before playtests improves the quality of decisions; that would be a natural next experiment.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, this is a careful, modest exploratory study that gives indie game pre-release experimentation a workable vocabulary. The new bit is the five-part framework — goal definition, design strategy, experiment object, sampling strategy, execution strategy — built inductively from ten interviews, plus the observation that pre-release experimentation in this setting is essentially qualitative. The emphasis on understandability as an evaluation aspect alongside mechanics, aesthetics, and fun factor is a useful addition. The method is transparent: conference sampling, semi-structured interviews, pairwise coding with a merged codebook, member checking, and an explicit note about saturation. The authors are honest that this is an emerging framework, not a validated one.

Soft spots are the usual ones for interview surveys at this scale. One representative per company, mostly founder-level, interviewed for about 30 minutes, gives a picture of the founder's mental model more than distributed team practice. The sample skews Swedish and was recruited at a single conference. I7, with 50+ employees and experience at larger mobile studios, sits awkwardly in an 'indie' sample. The interview guide is not in the paper, so the degree to which the five parts were pre-structured by the instrument is not fully checkable from the text; the codebook is supplementary, which helps. None of this is fatal for an exploratory qualitative study, and the authors already frame the result as something to refine with cases and larger companies.

In proportion: this is a useful, well-reported qualitative contribution to the continuous experimentation and game software engineering literature. It doesn't overclaim. It deserves a serious referee. I'd ask the authors to make the interview guide available in the supplementary material and to be more explicit about the single-informant limitation, but I wouldn't hold the paper hostage to those.

Recommendation: send to peer review. It's the kind of paper that gets cited for its framework in future CE-in-games work.

Referee Report

3 major / 5 minor

Summary. The paper presents an exploratory qualitative interview survey of ten indie game developers, one per company, about pre-release experimentation practices. Through open and axial coding, the authors synthesize an emerging continuous experimentation framework with five key parts: goal definition, design strategy, experiment object, sampling strategy, and execution strategy. They report that pre-release experimentation is centered on qualitative data, with playtesting and observation as dominant methods, and that resource constraints and limited access to participants shape sampling and execution. The paper positions the framework as a first step to be validated in future case studies.

Significance. If the framework holds, it fills a gap in continuous experimentation research by characterizing experimentation in indie game development, a context with limited user data and resources. The study's methodological reporting is a strength: pairwise coding with merged codebooks, peer review of coding, audit trail, saturation noted by the tenth interview, member checking of an early article version, and a supplementary codebook. The framework is explicitly exploratory and the authors include a candid limitations paragraph. The significance is nevertheless bounded by the sample (10 companies, mostly Swedish, mostly desktop), the single-informant design, and the lack of triangulation; the result is best read as a hypothesis-generating synthesis rather than a validated company-level account.

major comments (3)

[Section 3, Table 1] The paper asserts that one representative per indie company was deemed sufficient to understand practice at these 'very small companies,' but the sample includes I7 with 50+ employees, and all informants are founders, CEOs, or lead developers. Because the framework describes company-level experimentation practice, a single retrospective self-report from one role cannot validate distributed practices, and the five parts may capture founder mental models rather than what the team actually does. Section 4.2 further attributes I7's mature A/B testing descriptions to experience from larger mobile game companies, so that interview may not describe current practice at the sampled indie company. Please either narrow the claims to individual-level perceptions, add triangulation for larger companies, or justify the single-informant assumption for each company.
[Section 3 and supplementary material [15]] The semi-structured interview questionnaire is not quoted or included in the manuscript, and the final codebook appears only in the supplementary material. Since the framework's five parts are derived from coded interview responses, the absence of the instrument makes it impossible to assess whether categories such as 'design strategy' or 'execution strategy' were prompted by the question wording rather than emergent from the data. Please include the interview guide (or at least the core questions) in an appendix and indicate how each framework part maps to the questions that elicited the supporting quotes.
[Section 5] The limitations paragraph appropriately cautions that the sample is small and mostly Swedish and that generalization should be anecdotal. However, several Discussion statements generalize beyond this evidence base, for example the claim that split testing is often performed in the initial idea stage and the assertion that the results 'may be of value also for larger game companies, and for software intensive organisations in other industries.' Given the explicitly emerging status of the framework, these inferences should be presented as hypotheses for future validation rather than as conclusions.

minor comments (5)

[Table 1] The number of employees is written with both a comma and a period ('2,5' and '1.5'), and 'Rougelite' should be 'Roguelite.'
[Section 3] The sentence 'Representatives from ten indie company were identified' should read 'ten indie companies.'
[Section 4.2] The word 'perofmred' should be 'performed' in the sentence about the cadence with which experiments are performed.
[Section 4.3] The phrase 'present their game ideas to publishers early one' should be 'early on.'
[Figure 1] Please ensure that Figure 1 is legible in print and in grayscale, since it is central to understanding the proposed framework.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the framework is an inductive synthesis of interviews, not a prediction derived from its own inputs.

full rationale

The paper's central claim is an emerging framework for continuous experimentation in pre-release indie game development, synthesised from ten semi-structured interviews through open and axial coding (Section 3 and Section 4). There are no equations, no fitted parameters, and no quantity is predicted from a model that was calibrated on the same data. The five key parts are presented as categories that emerged from the interview material, and the paper explicitly frames them as an initial framework to be validated in future work. Consequently, none of the circularity patterns apply: no concept is defined in terms of another concept it is supposed to derive, no fitted input is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work. The related-work citations, including some authored by the present authors (e.g., references 4, 6, and 8), serve as background and motivation rather than as load-bearing support for the framework's categories. The paper also reports mitigations for researcher bias through pair coding, peer review, and member checking, and it explicitly limits generalisation to anecdotal inference given the small, Sweden-skewed sample. The methodological limitations noted in the paper, such as one representative per company and recruitment at a single conference, are threats to empirical validity rather than instances of circular reasoning. Overall, the derivation chain is self-contained as a qualitative synthesis, and the paper makes no prediction that reduces by construction to its inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The empirical claims rest on standard qualitative survey assumptions: that interviewees represent their companies, that self-reports reflect practice, and that saturation was reached. These are reasonable for an exploratory study but are not independently verified. There are no free parameters or invented entities.

assumptions (3)

domain assumption One representative per indie company is sufficient to understand that company's experimentation practice.
Stated in Section 3 Research Design. If a single 30-minute interview does not capture the company's practice, the framework categories may be incomplete.
domain assumption Interviewees' self-reports of their experimentation practices are accurate.
Implied throughout the analysis. No independent observation of actual practices was made; the study relies on participants' descriptions during semi-structured interviews.
domain assumption Saturation of findings was reached by the tenth interview.
Section 3 states that by I10 the data was mainly confirming. This supports the stopping point, but saturation is inferred rather than formally demonstrated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Pre-Release Experimentation in Indie Game Development: An Interview Survey." pith.science (2026). https://pith.science/paper/FGLQSYN3

@misc{pith2026241117183,
  author       = {Pith},
  title        = {Pith review of: Pre-Release Experimentation in Indie Game Development: An Interview Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FGLQSYN3}},
  note         = {Machine review of arXiv:2411.17183}
}

read the original abstract

[Background] The game industry faces fierce competition and games are developed on short deadlines and tight budgets. Continuously testing and experimenting with new ideas and features is essential in validating and guiding development toward market viability and success. Such continuous experimentation (CE) requires user data, which is often limited in early development stages. This challenge is further exacerbated for independent (indie) game companies with limited resources. [Aim] We wanted to gain insights into CE practices in pre-release indie game development. [Method] We performed an exploratory interview survey with 10 indie game developers from different companies and synthesised findings through an iterative coding process. [Results] We present a CE framework for game development that highlights key parts to consider when planning and implementing an experiment and note that pre-release experimentation is centred on qualitative data. Time and resource constraints impose limits on the type and extent of experimentation and playtesting that indie companies can perform, e.g. due to limited access to participants, biases and representativeness of the target audience. [Conclusions] Our results outline challenges and practices for conducting experiments with limited user data in early stages of indie game development, and may be of value also for larger game companies, and for software intensive organisations in other industries.

Figures

Figures reproduced from arXiv: 2411.17183 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Experimentation in Gaming: an Adoption Guide
cs.HC 2025-01 unverdicted novelty 2.0 of 10

A practitioner guide that explains how to apply experimentation and A/B testing throughout game development and live operations, with a matrix-based framework and ownership recommendations.

Reference graph

Works this paper leans on

24 extracted references · 23 canonical work pages · cited by 1 Pith paper

[1]

J of Softw Engin Res and Dev 4, 1–30 (2016)

Aleem, S., Capretz, L.F., Ahmed, F.: Game development software engineering pro- cess life cycle: a systematic review. J of Softw Engin Res and Dev 4, 1–30 (2016)

work page 2016
[2]

In: Proc

Andersen, E., Liu, Y.E., Snider, R., Szeto, R., Cooper, S., Popovi´ c, Z.: On the harmfulness of secondary game objectives. In: Proc. of the 6th Int. Conf. on Foun- dations of Digital Games. p. 30–37. FDG ’11, ACM (2011)

work page 2011
[3]

Information and Software Technology 134, 106551 (2021) Pre-Release Experimentation in Indie Game Development 15

Auer, F., Ros, R., Kaltenbrunner, L., Runeson, P., Felderer, M.: Controlled exper- imentation in continuous experimentation: Knowledge and challenges. Information and Software Technology 134, 106551 (2021) Pre-Release Experimentation in Indie Game Development 15

work page 2021
[4]

Emp Softw Eng28(5) (2023)

Bjarnason, E., Lang, F., Mj¨ oberg, A.: An empirically based model of software prototyping: a mapping study and a multi-case study. Emp Softw Eng28(5) (2023)

work page 2023
[5]

Chueca, J., Ver´ on, J., Font, J., P´ erez, F., Cetina, C.: The consolidation of game software engineering: A systematic literature review of software engineering for industry-scale computer games. Inf. and Softw. Technology p. 107330 (2023)

work page 2023
[6]

Edison, H., Melegati, J., Bjarnason, E.: Experimentation in early-stage video game startups: Practices and challenges. In: Int. Conf. on Softw. Business. pp. 360–366. Springer (2023)

work page 2023
[7]

Engstr¨ om, H.: Game development research (2020)

work page 2020
[8]

Journal of Systems and Software 123, 292–305 (2017)

Fagerholm, F., Guinea, A.S., M¨ aenp¨ a¨ a, H., M¨ unch, J.: The right model for contin- uous experimentation. Journal of Systems and Software 123, 292–305 (2017)

work page 2017

Show all 24 references

[9]

Game Studies 16(1) (2016)

Grabarczyk, P.: Is every indie game independent? towards the concept of indepen- dent game. Game Studies 16(1) (2016)

2016
[10]

In: Continuous software engineering, pp

Holmstr¨ om Olsson, H., Bosch, J.: The hypex model: from opinions to data-driven software development. In: Continuous software engineering, pp. 155–164. Springer International Publishing (2014). https://doi.org/10.1007/978-3-319-11283-1 13

2014 doi
[11]

In: 17th IFIP WG 6.11 Conf

Hyrynsalmi, S., Klotins, E., Unterkalmsteiner, M., Gorschek, T., Tripathi, N., Pom- permaier, L.B., Prikladnicki, R.: What is a minimum viable (video) game? towards a research agenda. In: 17th IFIP WG 6.11 Conf. on e-Business, e-Services, and e- Society. pp. 217–231. Springer (2018)

2018
[12]

Swedish Games Industry (2023)

Industry, S.G.: Swedish Games Industry 2023 Game Developer Index. Swedish Games Industry (2023)

2023
[13]

In: Software Business: 6th Int

J¨ arvi, A., Taajamaa, V., Hyrynsalmi, S.: Lean software startup–an experience re- port from an entrepreneurial software business course. In: Software Business: 6th Int. Conf., 2015, Proc. 6. pp. 230–244. Springer (2015)

2015
[14]

on e-Business, e- Services, and e-Society

Koskenvoima, A., M¨ antym¨ aki, M.: Why do small and medium-size freemium game developers use game analytics? In: 14th IFIP WG 6.11 Conf. on e-Business, e- Services, and e-Society. pp. 326–337. Springer (2015)

2015
[15]

Lin ˚ aker, J., Bjarnason, E., Fagerholm, F.: Online supplementary material (2024), https://doi.org/10.6084/m9.figshare.26934910

2024 doi
[16]

The Wiley Handbook of Human Computer Interaction 1, 299–346 (2018)

Pagulayan, R.J., Gunn, D.V., Hagen, J.R., Hendersen, D.J., Kelley, T.A., Phillips, B.C., Guajardo, J., Nichols, T.A.: Applied user research in games. The Wiley Handbook of Human Computer Interaction 1, 299–346 (2018)

2018
[17]

qualitative surveys

Ralph, P.: Acm sigsoft empirical standards for software engineering research. qualitative surveys. arxiv:2010.03525 [cs.se] (2021), https://www2.sigsoft.org/ EmpiricalStandards/docs/standards?standard=QualitativeSurveys#

2021
[18]

In: Proc

Ros, R., Runeson, P.: Continuous experimentation and a/b testing: A mapping study. In: Proc. 4th RCoSE. pp. 35–41 (2018)

2018
[19]

33–48 (2017)

Rosenfield Boeira, J.N., Rosenfield Boeira, J.N.: Mvps: Do we really need them? Lean Game Dev: Apply Lean Framew to the Proc of Game Dev pp. 33–48 (2017)

2017
[20]

Sage (2021)

Salda˜ na, J.: The coding manual for qualitative researchers. Sage (2021)

2021
[21]

In: 47th HICSS

Schmalz, M., Finn, A., Taylor, H.: Risk management in video game development projects. In: 47th HICSS. pp. 4325–4334. IEEE (2014)

2014
[22]

Comp Ent16(2) (2018)

Stahlke, S.N., Mirza-Babaei, P.: Usertesting without the user: Opportunities and challenges of an ai-driven approach in games user research. Comp Ent16(2) (2018)

2018
[23]

Service Oriented Comp and Appl 15(2), 141–156 (2021)

Su, Y., Backlund, P., Engstr¨ om, H.: Comprehensive review and classification of game analytics. Service Oriented Comp and Appl 15(2), 141–156 (2021)

2021
[24]

In: Proc

Yaman, S., Mikkonen, T., Suomela, R.: Continuous experimentation in mobile game development. In: Proc. of 44th SEAA. pp. 345–352 (2018)

2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

J of Softw Engin Res and Dev 4, 1–30 (2016)

Aleem, S., Capretz, L.F., Ahmed, F.: Game development software engineering pro- cess life cycle: a systematic review. J of Softw Engin Res and Dev 4, 1–30 (2016)

work page 2016

[2] [2]

In: Proc

Andersen, E., Liu, Y.E., Snider, R., Szeto, R., Cooper, S., Popovi´ c, Z.: On the harmfulness of secondary game objectives. In: Proc. of the 6th Int. Conf. on Foun- dations of Digital Games. p. 30–37. FDG ’11, ACM (2011)

work page 2011

[3] [3]

Information and Software Technology 134, 106551 (2021) Pre-Release Experimentation in Indie Game Development 15

Auer, F., Ros, R., Kaltenbrunner, L., Runeson, P., Felderer, M.: Controlled exper- imentation in continuous experimentation: Knowledge and challenges. Information and Software Technology 134, 106551 (2021) Pre-Release Experimentation in Indie Game Development 15

work page 2021

[4] [4]

Emp Softw Eng28(5) (2023)

Bjarnason, E., Lang, F., Mj¨ oberg, A.: An empirically based model of software prototyping: a mapping study and a multi-case study. Emp Softw Eng28(5) (2023)

work page 2023

[5] [5]

Chueca, J., Ver´ on, J., Font, J., P´ erez, F., Cetina, C.: The consolidation of game software engineering: A systematic literature review of software engineering for industry-scale computer games. Inf. and Softw. Technology p. 107330 (2023)

work page 2023

[6] [6]

Edison, H., Melegati, J., Bjarnason, E.: Experimentation in early-stage video game startups: Practices and challenges. In: Int. Conf. on Softw. Business. pp. 360–366. Springer (2023)

work page 2023

[7] [7]

Engstr¨ om, H.: Game development research (2020)

work page 2020

[8] [8]

Journal of Systems and Software 123, 292–305 (2017)

Fagerholm, F., Guinea, A.S., M¨ aenp¨ a¨ a, H., M¨ unch, J.: The right model for contin- uous experimentation. Journal of Systems and Software 123, 292–305 (2017)

work page 2017