Pith. sign in

REVIEW 3 major objections 5 minor 42 references

Gamifying Testing in IntelliJ: A Replicability Study

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 174-participant replication reports that IntelliGame, a gamification plugin for IntelliJ, leads student developers to write and run more tests and to use coverage and debugging tools more often.

desk verdict A careful replication with credible behavioral effects, but the abstract overstates a non-significant functional result and the missing-TestReport exclusions need a group-balance check. read the letter →

arxiv 2504.19294 v1 pith:7QI2R77Z submitted 2025-04-27 cs.SE

Philipp Straubinger , Tommaso Fulcini , Giacomo Garaccione , Luca Ardito , Gordon Fraser This is my paper · ORCID
classification cs.SE
keywords gamificationsoftwaretestingIntelliJIDEAGamereplicabilitystudyunitTypeScriptempiricalengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a gamification plugin embedded in an IDE can change how developers test, and that the effect survives a move from Java to TypeScript and from 49 to 174 participants. In a controlled two-group experiment, students using IntelliGame wrote more Jest tests (15.95 vs 11.38 on average), ran tests more than twice as often, and used coverage and debug tools more than a control group that ran the plugin's tracking without visible achievements. The paper also reports that gamified participants reached passing implementations earlier and had fewer failures on a golden test suite, while code coverage and mutation scores were not significantly different between groups. If correct, this would mean that a relatively lightweight achievement system can raise testing engagement in educational settings, with the caveat that it also produced more test smells and brittle date-sensitive tests.

What carries the argument

The carrying mechanism is IntelliGame itself, an IntelliJ IDEA plugin that awards multi-level achievements for testing-related actions: running tests, adding assertions, fixing failing tests, measuring coverage, debugging, and refactoring test code. Progress is displayed as trophies and progress bars, and notifications provide immediate feedback when a level is reached. In this study the plugin's achievement system is the independent variable; the central analytic move is that the control group runs the identical plugin with tracking enabled but no visible achievements, isolating the motivational effect of the game elements. Achievement levels are then used as a quantitative engagement signal, correlated with test-suite metrics.

What would settle it

Recompute the analysis with the 44 excluded projects restored where possible, and test whether missing TestReport rates differ between treatment and control; if exclusion is unbalanced, the central comparisons (test count, executions, coverage-tool use) may reflect differential attrition rather than gamification. Alternatively, a preregistered replication in which neither the task nor the incentive depends on plugin installation success could falsify the claim if the test-count gap disappears.

Watch

Extended reading notes

Core claim

The paper's central claim is that achievements, not just monitoring, drive the effect: participants in the treatment group, who saw trophies, progress bars, and notifications, behaved differently from the control group, who ran the same data-collecting plugin without the gamified display. Specifically, the authors report significant differences for test executions ($p<0.001$), coverage-report executions ($p<0.001$), debug-mode uses ($p=0.015$), and final test count ($p=0.003$), and a near-significant difference in golden-suite failures ($p=0.054$). They further claim achievement levels correlate with test-suite quality (number of tests $r=0.65$, mutation score $r=0.48$). The replication confirms the original study's behavioral findings but not its coverage or mutation-score improvements, and it reveals that the gamified group produced more test smells and more failing tests in their own suites.

Load-bearing premise

The result rests on the assumption that the 44 projects without a TestReport (omitted from the analysis) were lost for reasons unrelated to whether a participant was in the gamified group; if plugin setup failures were more common in one arm, the measured differences could be an artifact of who got counted.

Editorial extensions

If this is right

  • If the replication is right, adding an achievement layer to an IDE can shift how much testing novices do within a single session, without changing the testing tools or task.
  • Test count and execution frequency will rise, but coverage and mutation scores may stay statistically flat, so gamification should be paired with quality-based achievements to avoid quantity-over-quality.
  • Achievement levels can be used as a real-time engagement signal: higher levels in this study track with more tests and better suite metrics.
  • Because treatment participants reported more time pressure and lower confidence, designers should calibrate achievement thresholds to avoid inducing pressure.
  • The golden-suite result (14.32 vs 19.25 failures, $p=0.054$) suggests gamified users implement required functionality earlier, though the effect is borderline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment would compare IntelliGame against a version that only credits passing or newly-fixed tests; this paper's finding that failing tests count equally suggests the quantity effect could be reduced by incentive alignment.
  • If the mechanism is simply more test executions, then professional onboarding might show similar effects, but with a risk of superficial tests (console prints, redundant cases) unless test-smell or robustness achievements are added.
  • The absence of significant coverage and mutation effects may be an artifact of the chosen date-library task, whose functions have few branches; a replication on branch-heavy code would tell whether the quality effects from the original study generalize.
  • Re-running tests at the participant's commit time resolved 61 of 214 initial failures; other experiments on time-sensitive code should adopt time-travel test execution to avoid counting environment drift as test failure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper reports a replication study of IntelliGame, a gamification plugin for IntelliJ IDEA, in a new context: TypeScript/Jest and a larger pool of 174 master's students from Politecnico di Torino. The experiment randomly assigns participants to a treatment group using the full plugin or a control group using a data-collection-only version, and compares testing behavior (tests written, executions, coverage runs, debug runs), test-suite quality (coverage, mutation score), achievement levels, code functionality against a golden suite, developer experience, and test quality via test smells. The authors report significant effects for several behavioral outcomes, non-significant differences for coverage and mutation score, and a near-significant difference in golden-suite failures, and they position the study as confirming and nuancing the original IntelliGame validation.

Significance. If the findings hold, the study is a useful independent replication: it extends gamification evidence to TypeScript, uses a larger sample than the original study, preregisters no analysis, ships data at a public repository, reports exact Wilcoxon-Mann-Whitney and Fisher tests, uses random assignment with separate sessions to hide treatment, and includes a transparent time-travel re-analysis for date-sensitive test failures. The clearest strength is the direct randomized comparison for the main behavioral outcomes, which supports causal language for those outcomes. However, the paper overstates the functionality result (p = 0.054 is not significant), relies on an untested missing-data assumption for the 44 excluded participants, and treats partly self-referential achievement correlations as evidence of a causal path to better test suites. These issues are fixable but must be addressed before the claims as written can be accepted.

major comments (3)
  1. [§3.5 and §4.1] The analysis restricts the sample to the 174 projects with an IntelliGame TestReport, but the paper does not report the number of missing TestReports in the treatment and control groups separately. Since the RQ1 outcomes (tests written, test executions, coverage executions, debug executions) are extracted from this report, group-correlated missingness could bias the headline effects: for example, if control-group participants who failed to install the plugin or produced no tests were preferentially excluded, the apparent treatment effect would be inflated. Please report missingness by group, compare included and excluded participants on available covariates (e.g., survey demographics and course section), and provide a sensitivity analysis treating missing outcomes as zeros or using multiple imputation. Until this is done, the central RQ1 claims rest on an untested missing-at-random assumption.
  2. [§4.4 and Abstract] The abstract and the introduction claim that IntelliGame led to 'better code implementations' and 'fewer failing tests of the reference test suite,' but the supporting comparison in §4.4 is only nearly significant (p = 0.054, 14.32 vs. 19.25 failing tests). At the stated alpha = 0.05, this difference is not statistically significant, and the text itself correctly hedges it as 'nearly statistically significant.' The abstract, the RQ4 summary, and the concluding sentence that 'the treatment group also achieved ... fewer failing tests' should be revised to state that the difference was in the expected direction but did not reach significance, or they should report the result with a confidence interval and a clearly stated interpretation.
  3. [§4.3, Fig. 4b] RQ3 correlates achievement levels with test-suite metrics, but the achievement levels are computed from the very activities being measured: running tests, achieving coverage, and debugging. The strong correlation between number of levels and number of tests (r = 0.65) is therefore in part a construction artifact rather than independent evidence that gamification 'leads to better-quality test suites.' The text should either reframe RQ3 as a descriptive account of how the plugin tracks behavior, or validate levels against an external outcome such as golden-suite failures; as written, the causal language in the RQ3 summary is not supported by the correlation design.
minor comments (5)
  1. [§4.1, Fig. 2] The text reports that differences are significant when confidence intervals stop overlapping, and Fig. 2b mentions '84.6% confidence intervals' without explaining why that level was chosen; please clarify the rationale for the 84.6% level or report the exact-test p-values at the relevant time points instead.
  2. [§4.3, Fig. 4b] The figure header labels the column 'r2' but the text reports 'Pearson correlations' with values such as 0.42, which look like r, not r^2; also, the text says 'Pearson rank correlations,' which conflates Pearson and Spearman correlation. Please correct the label and use consistent terminology.
  3. [Throughout] There are repeated typos, including 'thetreatment' in §4.1 and §4.4, 'asumption' in Listing 3, and inconsistent spacing in 'thetreatment group' and 'thecontrol group'; a careful proofread is needed.
  4. [§4.6] The paper first reports '214 initially failing tests' and later says 'Out of 214 initially failing tests, 61 passed,' but the number of failing tests in RQ2 is described differently; please reconcile these counts and clarify whether they include only participant-written tests or also golden-suite failures.
  5. [§4.7, Table 1] In Table 1, 'Take some breaks' and 'Break the Line' have identical mean values (2.31 treatment, 8.29 control) and identical p-values (0.43), which looks like a copy-paste error; please verify these entries.

Circularity Check

1 steps flagged · score 6.0 of 10

The central treatment-effect claim is an independent randomized comparison, but RQ3's achievement-level correlations are partly mechanical because achievement levels are computed from the very testing activities with which they are correlated.

  1. self definitional [Section 4.3 (RQ3), in conjunction with Section 2.2 and Section 3.6.3]
    "To assess whether developers with higher achievement levels produce better test suites overall, Fig. 4b presents the Pearson rank correlations between test suite metrics (RQ2) and achievement levels. A strong significant correlation exists between the number of tests and achievement levels, along with a moderate positive correlation for both line coverage and mutation score. Additionally, there is a weak significant correlation between achievement levels and branch coverage."

    Achievement levels are not independent of the metrics they are correlated with. Section 2.2 says that 'progress [is] earned through actions such as running tests, using assertions, debugging, and improving test coverage' and that 'Testing achievements reward developers for writing and running tests.' Therefore the number of tests, line coverage, and branch coverage are inputs to the achievement-level computation, so positive correlations between these metrics and achievement levels are expected by construction. The conclusion that 'higher achievement levels... lead to better-quality test suites' presents this built-in relationship as evidence of effectiveness, when it partly only verifies that the plugin counts the activities it was designed to count.

full rationale

The paper's headline result is an independent controlled experiment: treatment and control groups were randomized, the outcome metrics come from automatically collected plugin and Jest data, and the functionality comparison uses an external golden test suite. That central causal chain does not reduce to a fitted parameter or a self-citation. The self-citations to the original IntelliGame study are expected in a replicability study and are not load-bearing in a circular way; they identify the artifact and prior thresholds, while the new data are independently gathered. The exclusion of 44 projects without a TestReport is a genuine missingness and internal-validity concern, but it is a data-availability filter, not a by-construction equivalence, so it does not by itself constitute circularity. The one concrete circular element is RQ3: achievement levels are earned from test writing, executions, coverage, and debugging, so correlating those levels with the same classes of metrics partly measures the plugin against its own scoring definition. Because this mechanical correlation supports a secondary claim rather than the main treatment effect, the overall circularity is partial.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The study's central claims rest primarily on experimental-design assumptions (ground truth validity, group comparability, tool-use compliance) rather than mathematical axioms. The only hand-fitted numerical inputs are the achievement level thresholds inherited from the original IntelliGame pilot study, which shape the treatment but are not used to compute the outcome metrics.

free parameters (1)
  • Achievement level thresholds = Set in pilot study before original IntelliGame study
    The boundaries for each achievement level are predefined thresholds that determine when rewards are granted. The paper states these were set based on an analysis of typical developer workflows in a pilot study before the original study (Section 2.2). Different thresholds could alter which behaviors are rewarded and hence the measured treatment effect.
assumptions (5)
  • domain assumption The reference test suite (60 tests, 95.83% line coverage, 100% mutation score for the selected functions) is a valid ground truth for code functionality.
    Used for RQ4 to count passing/failing tests for each participant's implementation. Location: Section 3.7.3.
  • domain assumption Random assignment produced comparable groups and no systematic differences in skill or motivation.
    Threats to conclusion validity section argues this is unlikely, though not entirely dismissible. Location: Section 3.7.1.
  • domain assumption Participants did not use external tools (GitHub Copilot, ChatGPT) despite monitoring.
    The authors instructed participants not to use such tools and monitored them, but sporadic use cannot be entirely ruled out. Location: Section 3.7.2.
  • domain assumption The conversion of the original Vitest test suite to Jest preserved equivalence.
    The authors verified both suites produced no failures and the same mutation scores. Location: Section 3.2.
  • standard math The exact Wilcoxon-Mann-Whitney test is valid for the observed metric distributions.
    Used to compute p-values; assumes independent observations and provides exact inference. Location: Section 3.6.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gamifying Testing in IntelliJ: A Replicability Study." pith.science (2026). https://pith.science/paper/7QI2R77Z

@misc{pith2026250419294,
  author       = {Pith},
  title        = {Pith review of: Gamifying Testing in IntelliJ: A Replicability Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7QI2R77Z}},
  note         = {Machine review of arXiv:2504.19294}
}
read the original abstract

Gamification is an emerging technique to enhance motivation and performance in traditionally unengaging tasks like software testing. Previous studies have indicated that gamified systems have the potential to improve software testing processes by providing testers with achievements and feedback. However, further evidence of these benefits across different environments, programming languages, and participant groups is required. This paper aims to replicate and validate the effects of IntelliGame, a gamification plugin for IntelliJ IDEA to engage developers in writing and executing tests. The objective is to generalize the benefits observed in earlier studies to new contexts, i.e., the TypeScript programming language and a larger participant pool. The replicability study consists of a controlled experiment with 174 participants, divided into two groups: one using IntelliGame and one with no gamification plugin. The study employed a two-group experimental design to compare testing behavior, coverage, mutation scores, and participant feedback between the groups. Data was collected through test metrics and participant surveys, and statistical analysis was performed to determine the statistical significance. Participants using IntelliGame showed higher engagement and productivity in testing practices than the control group, evidenced by the creation of more tests, increased frequency of executions, and enhanced utilization of testing tools. This ultimately led to better code implementations, highlighting the effectiveness of gamification in improving functional outcomes and motivating users in their testing endeavors. The replication study confirms that gamification, through IntelliGame, positively impacts software testing behavior and developer engagement in coding tasks.

Figures

Figures reproduced from arXiv: 2504.19294 by the authors.

Figure 1
Figure 1. IntelliGame window showing part of the achievements and their progress [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Differences between control and treatment groups over time Control Treatment 0 50 100 200 300 Test executions in absolute numbers (a) Number of test executions during the experiment Control Treatment 0 5 10 15 20 Test executions with coverage (b) Number of test executions with coverage in IntelliJ Control Treatment 0 5 10 15 Number of times debug mode was used (c) Number of times the IntelliJ debug mode was used Con… view at source ↗
Figure 3
Figure 3. Differences between control and treatment groups Proc. ACM Softw. Eng., Vol. 2, No. ISSTA, Article ISSTA106. Publication date: July 2025 [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The number of achievement levels, their Pearson correlations with different test suite metrics as well [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Survey responses of the participants as Likert plots [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 29 canonical work pages

  1. [1]

    Carlos Futino Barreto and César França. 2021. Gamification in Software Engineering: A literature Review. In 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) . 105–108. https://doi.org/10.1109/CHASE52884.2021.00020

    arXiv 2021

  2. [2]

    Jonathan Bell, Swapneel Sheth, and Gail Kaiser. 2011. Secret ninja testing with HALO software engineering. In Proceedings of the 4th International Workshop on Social Software Engineering (Szeged, Hungary) (SSE ’11). Association for Computing Machinery, New York, NY, USA, 43–47. https://doi.org/10.1145/2024645.2024657

    work page arXiv 2011

  3. [3]

    Raquel Blanco, Manuel Trinidad, María José Suárez-Cabal, Alejandro Calderón, Mercedes Ruiz, and Javier Tuya. 2023. Can gamification help in software testing education? Findings from an empirical study.Journal of Systems and Software 200 (2023), 111647. https://doi.org/10.1016/j.jss.2023.111647

    arXiv 2023

  4. [4]

    Keith M Bower. 2003. When to use Fisher’s exact test. In American Society for Quality, Six Sigma Forum Magazine , Vol. 2. American Society for Quality Milwaukee, WI, USA, 35–37

    work page 2003

  5. [5]

    Filippo Cacciotto, Tommaso Fulcini, Riccardo Coppola, and Luca Ardito. 2021. A Metric Framework for the Gamification of Web and Mobile GUI Testing. In 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 126–129. https://doi.org/10.1109/ICSTW52544.2021.00032

    arXiv 2021

  6. [6]

    Clarke, Debra L

    Peter J. Clarke, Debra L. Davis, Raymond Chang-Lau, and Tariq M. King. 2017. Impact of Using Tools in an Undergrad- uate Software Testing Course Supported by WReSTT. ACM Trans. Comput. Educ. 17, 4, Article 18 (Aug. 2017), 28 pages. https://doi.org/10.1145/3068324

    work page doi:10.1145/3068324 2017

  7. [7]

    Riccardo Coppola, Tommaso Fulcini, Luca Ardito, Marco Torchiano, and Emil Alègroth. 2024. On Effectiveness and Efficiency of Gamified Exploratory GUI Testing. IEEE Transactions on Software Engineering 50, 2 (2024), 322–337. https://doi.org/10.1109/TSE.2023.3348036

    arXiv 2024

  8. [8]

    and Sandro Ronaldo Bezerra Oliveira

    Igor Ernesto Ferreira Costa. and Sandro Ronaldo Bezerra Oliveira. 2019. A Systematic Strategy to Teaching of Exploratory Testing using Gamification. In Proceedings of the 14th International Conference on Evaluation of Novel Approaches to Software Engineering - ENASE. INSTICC, SciTePress, 307–314. https://doi.org/10.5220/0007711603070314

    work page doi:10.5220/0007711603070314 2019

Show all 42 references
  1. [9]

    Gabriela Martins de Jesus, Fabiano Cutigi Ferrari, Daniel de Paula Porto, and Sandra Camargo Pinto Ferraz Fabbri

  2. [10]

    Gabriela Martins de Jesus, Leo Natan Paschoal, Fabiano Cutigi Ferrari, and Simone R. S. Souza. 2019. Is It Worth Using Gamification on Software Testing Education? An Experience Report. In Proceedings of the XVIII Brazilian Symposium on Software Quality (Fortaleza, Brazil) (SBQ...

    2019

  3. [11]

    Daniel de Paula Porto, Gabriela Martins de Jesus, Fabiano Cutigi Ferrari, and Sandra Camargo Pinto Ferraz Fabbri

  4. [12]

    Gamification

    Sebastian Deterding, Dan Dixon, Rilla Khaled, and Lennart Nacke. 2011. From Game Design Elements to Gamefulness: Defining "Gamification". In Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments (Tampere, Finland) (MindTrek ’...

    2011

  5. [13]

    24-08-2020

    Association for Computing Machinery. 24-08-2020. Artifact Review and Badging Version 1.1. https://www.acm.org/ publications/policies/artifact-review-and-badging-current. [Accessed 28-10-2024]

    2020

  6. [14]

    Tommaso Fulcini and Luca Ardito. 2022. Gamified Exploratory GUI Testing of Web Applications: a Preliminary Evaluation. In 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) . 215–222. https://doi.org/10.1109/ICSTW55395.2022.00045

    2022

  7. [15]

    Tommaso Fulcini, Riccardo Coppola, Luca Ardito, and Marco Torchiano. 2023. A Review on Tools, Mechanics, Benefits, and Challenges of Gamified Software Testing. ACM Comput. Surv. 55, 14s, Article 310 (July 2023), 37 pages. https://doi.org/10.1145/3582273

    2023 doi

  8. [16]

    Giacomo Garaccione, Tommaso Fulcini, Paolo Stefanut Bodnarescul, Riccardo Coppola, and Luca Ardito. 2024. Gamified GUI testing with Selenium in the IntelliJ IDE: A Prototype Plugin. In Proceedings of the 1st ACM/IEEE Workshop on Integrated Development Environments (Lisbon, Por...

    2024

  9. [17]

    Giacomo Garaccione, Tommaso Fulcini, and Marco Torchiano. 2022. GERRY: a gamified browser tool for GUI testing. In Proceedings of the 1st International Workshop on Gamification of Software Development, Verification, and Validation (Singapore, Singapore) (Gamify 2022). Associat...

    2022

  10. [18]

    Jedlitschka and D

    A. Jedlitschka and D. Pfahl. 2005. Reporting guidelines for controlled experiments in software engineering. In 2005 International Symposium on Empirical Software Engineering, 2005. 10 pp.–. https://doi.org/10.1109/ISESE.2005.1541818 Proc. ACM Softw. Eng., Vol. 2, No. ISSTA, Ar...

    2005

  11. [19]

    Natalia Juristo and Omar S. Gómez. 2012. Replication of Software Engineering Experiments . Springer Berlin Heidelberg, Berlin, Heidelberg, 60–88. https://doi.org/10.1007/978-3-642-25231-0_2

    2012 doi

  12. [20]

    Beáta Lőrincz, Bogdan Iudean, and Andreea Vescan. 2021. Experience report on teaching testing through gamification. In Proceedings of the 3rd International Workshop on Education through Advanced Software Engineering and Artificial Intelligence (Athens, Greece) (EASEAI 2021) . ...

    2021

  13. [21]

    Tobias Lorey, Stefan Mohacsi, Armin Beer, and Michael Felderer. 2022. Storm: A model for sustainably onboarding software testers. arXiv preprint arXiv:2206.01020 (2022)

    2022 arXiv

  14. [22]

    H. B. Mann and D. R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics 18, 1 (1947), 50 – 60. https://doi.org/10.1214/aoms/1177730491

    1947

  15. [23]

    Antonio Materazzo, Tommaso Fulcini, Riccardo Coppola, and Marco Torchiano. 2023. Survival of the Tested: Gamified Unit Testing Inspired by Battle Royale. In2023 IEEE/ACM 7th International Workshop on Games and Software Engineering (GAS). 1–7. https://doi.org/10.1109/GAS59301.2...

    2023

  16. [24]

    Edgy Paiva, Danielly Barbosa, Roberto Lima, and Adriano Albuquerque. 2010. Factors that Influence the Productivity of Software Developers in a Developer View. In Innovations in Computing Sciences and Software Engineering , Tarek Sobh and Khaled Elleithy (Eds.). Springer Nether...

    2010

  17. [25]

    and Peter J

    Yujian Fu P.E. and Peter J. Clarke. 2016. Gamification-Based Cyber-Enabled Learning Environment of Software Testing. In 2016 ASEE Annual Conference & Exposition . ASEE Conferences, New Orleans, Louisiana. https://peer.asee.org/27000

    2016

  18. [26]

    Karl Pearson. 1895. VII. Note on regression and inheritance in the case of two parents. proceedings of the royal society of London 58, 347-352 (1895), 240–242

  19. [27]

    Kitchenham, Romain Robbes, Daniel Méndez, Jefferson Molleri, Diomidis Spinellis, Miroslaw Staron, Klaas-Jan Stol, Damian A

    Paul Ralph, Sebastian Baltes, Domenico Bianculli, Yvonne Dittrich, Michael Felderer, Robert Feldt, Antonio Filieri, Carlo Alberto Furia, Daniel Graziotin, Pinjia He, Rashina Hoda, Natalia Juristo, Barbara A. Kitchenham, Romain Robbes, Daniel Méndez, Jefferson Molleri, Diomidis...

    2020

  20. [28]

    Muthu Ramachandran. 2003. Testing Software Components Using Boundary Value Analysis. In 29th EUROMICRO Conference 2003, New Waves in System Architecture, 3-5 September 2003, Belek-Antalya, Turkey . IEEE Computer Society, 94–98. https://doi.org/10.1109/EURMIC.2003.1231572

    2003 arXiv

  21. [29]

    José Miguel Rojas and Gordon Fraser. 2016. Code Defenders: A Mutation Testing Game. In2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW) . 162–167. https://doi.org/10.1109/ ICSTW.2016.43

    2016

  22. [30]

    Martin Shepperd, Nemitari Ajienka, and Steve Counsell. 2018. The role and value of replication in empirical software engineering results. Information and Software Technology99 (2018), 120–132. https://doi.org/10.1016/j.infsof.2018.01.006

    2018 doi

  23. [31]

    Swapneel Sheth, Jonathan Bell, and Gail Kaiser. 2015. A gameful approach to teaching software design and software testing. Computer Games and Software Engineering 9 (2015), 91

    2015

  24. [32]

    Klaas-Jan Stol, Mario Schaarschmidt, and Shelly Goldblit. 2021. Gamification in software engineering: the mediating role of developer engagement and job satisfaction. Empirical Software Engineering 27, 2 (30 Dec 2021), 35. https: //doi.org/10.1007/s10664-021-10062-w

    2021 doi

  25. [33]

    Philipp Straubinger and Gordon Fraser. 2023. A Survey on What Developers Think About Testing. In 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) . 80–90. https://doi.org/10.1109/ISSRE59848.2023. 00075

    2023

  26. [34]

    Philipp Straubinger and Gordon Fraser. 2024. Engaging Developers in Exploratory Unit Testing through Gamification. In Proceedings of the 3rd ACM International Workshop on Gamification in Software Development, Verification, and Validation, Gamify 2024, Vienna, Austria, 17 Septe...

    2024

  27. [35]

    Philipp Straubinger and Gordon Fraser. 2024. Gamifying a Software Testing Course with Continuous Integration. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training . 34–45

    2024

  28. [36]

    Philipp Straubinger and Gordon Fraser. 2024. Improving Testing Behavior by Gamifying IntelliJ. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article 49...

    2024

  29. [37]

    Maneela Tuteja, Gaurav Dubey, et al. 2012. A research study on importance of testing and quality assurance in software development life cycle (SDLC) models. International Journal of Soft Computing and Engineering (IJSCE) 2, 3 (2012), 251–257

    2012

  30. [38]

    Arie Van Deursen, Leon Moonen, Alex Van Den Bergh, and Gerard Kok. 2001. Refactoring test code. In Proceedings of the 2nd international conference on extreme programming and flexible processes in software engineering (XP2001) . Citeseer, 92–95. Proc. ACM Softw. Eng., Vol. 2, N...

    2001

  31. [39]

    Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, Anders Wesslén, et al. 2012. Experimentation in software engineering. Vol. 236. Springer

    2012

  32. [40]

    Zornitsa Yordanova. 2019. Educational Innovations and Gamification for Fostering Training and Testing in Software Implementation Projects. In Software Business, Sami Hyrynsalmi, Mari Suoranta, Anh Nguyen-Duc, Pasi Tyrväinen, and Pekka Abrahamsson (Eds.). Springer International...

    2019

  33. [2018]

    In Proceedings of the III Brazilian Symposium on Systematic and Automated Software Testing (SAO CARLOS, Brazil) (SAST ’18)

    Gamification in Software Testing: A Characterization Study. In Proceedings of the III Brazilian Symposium on Systematic and Automated Software Testing (SAO CARLOS, Brazil) (SAST ’18). Association for Computing Machinery, New York, NY, USA, 39–48. https://doi.org/10.1145/326600...

  34. [2021]

    Journal of Systems and Software 173 (2021), 110870

    Initiatives and challenges of using gamification in software engineering: A Systematic Mapping. Journal of Systems and Software 173 (2021), 110870. https://doi.org/10.1016/j.jss.2020.110870

    2021

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.