{"id":"40afbfd4-39f8-4fb3-a1bd-fd3a3bcc5c58","arxiv_id":"2504.19294","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"In a 174-participant replication, the IntelliGame plugin increased test counts and executions but did not significantly improve coverage, mutation scores, or final code functionality.","lead":"This paper repeated a controlled experiment on a gamified IDE plugin, this time with 174 students writing TypeScript tests. It found that achievements increased testing activity, but the payoff in code quality was smaller and mostly not statistically significant.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The analysis excludes 44 participants without a TestReport without reporting whether missingness is balanced across groups; if setup failures or inactivity were group-correlated, the headline behavioral effects could be biased.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the 174-participant analysis set is defined by the presence of a TestReport, and the paper does not test whether absence of a TestReport is balanced between treatment and control. I agree this is the most important threat to the central behavioral claim. If the missingness is differential, every RQ1 metric derived from the TestReport could be biased, and the headline result that IntelliGame increases testing activity would not be established. The paper does provide a plausible mechanism for missingness (setup issues, plugin installation failure, lack of participation), but no data on how that mechanism interacts with group assignment. The concrete test described above would settle the concern using the provided artifact. I also note the secondary issue flagged by the reader: the abstract and RQ4 summary describe better code implementations, while the golden-suite failing-test comparison is only nearly significant (p = 0.054). That is a real reporting mismatch, but it is secondary to the behavioral claim and can be fixed by aligning the language with the p-value. The independent-support credit goes to the detailed reporting of effect sizes, p-values, time-course plots, and the artifact link; the study is valuable as a replication attempt. Because the missingness balance check is a required analysis rather than a reason to reject, the existing CONDITIONAL verdict remains appropriate, so I do not move the verdict.","tokens_in":20388,"tokens_out":4063,"duration_ms":39463,"concrete_test":"Extract from the replication artifact the group assignment and recorded reason (setup issue, plugin installation failure, lack of participation) for the 44 participants without a TestReport. Run a Fisher exact test of missing TestReport by group to check balance. Then re-run the RQ1 comparisons (tests written, test executions, coverage-assisted executions, debug-mode executions) twice: once with the 44 excluded participants assigned zero values on all TestReport-derived metrics, and once with them excluded as in the paper. If missingness is balanced across groups and the significant differences persist with the zero-included analysis, the concern is resolved; if either condition fails, the reported effect sizes and p-values are not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.5 reports that, of 218 usable projects, only 174 included the IntelliGame TestReport and that incomplete data was largely due to setup issues, plugin installation failure, or lack of actual participation. Because the TestReport is the source for the outcome metrics in RQ1 (tests written, test executions, coverage-assisted executions, debug-mode executions) and for the achievement-level data in RQ3, the restriction to 174 participants is not an arbitrary analysis choice: it selects the exact subpopulation that engaged with the instrument at all. If missing TestReports were more frequent in the control group, excluding those participants removes low-activity zeros from the control group and inflates the apparent treatment effect; if they were more frequent in the treatment group, the effect would be underestimated. The paper does not report the number of excluded participants by group, nor any comparison of excluded and included participants on available covariates. The central claim that IntelliGame increases testing activity therefore rests on an untested missingness assumption. This is load-bearing because the most important RQ1 results (e.g., p < 0.001 for test executions, p = 0.015 for debug use) could change if the 44 exclusions are unbalanced and should be treated as zeros rather than missing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a replication study of IntelliGame, a gamification plugin for IntelliJ IDEA, in a new context: TypeScript/Jest and a larger pool of 174 master's students from Politecnico di Torino. The experiment randomly assigns participants to a treatment group using the full plugin or a control group using a data-collection-only version, and compares testing behavior (tests written, executions, coverage runs, debug runs), test-suite quality (coverage, mutation score), achievement levels, code functionality against a golden suite, developer experience, and test quality via test smells. The authors report significant effects for several behavioral outcomes, non-significant differences for coverage and mutation score, and a near-significant difference in golden-suite failures, and they position the study as confirming and nuancing the original IntelliGame validation.","tokens_in":20598,"tokens_out":3095,"duration_ms":35839,"significance":"If the findings hold, the study is a useful independent replication: it extends gamification evidence to TypeScript, uses a larger sample than the original study, preregisters no analysis, ships data at a public repository, reports exact Wilcoxon-Mann-Whitney and Fisher tests, uses random assignment with separate sessions to hide treatment, and includes a transparent time-travel re-analysis for date-sensitive test failures. The clearest strength is the direct randomized comparison for the main behavioral outcomes, which supports causal language for those outcomes. However, the paper overstates the functionality result (p = 0.054 is not significant), relies on an untested missing-data assumption for the 44 excluded participants, and treats partly self-referential achievement correlations as evidence of a causal path to better test suites. These issues are fixable but must be addressed before the claims as written can be accepted.","major_comments":[{"comment":"The analysis restricts the sample to the 174 projects with an IntelliGame TestReport, but the paper does not report the number of missing TestReports in the treatment and control groups separately. Since the RQ1 outcomes (tests written, test executions, coverage executions, debug executions) are extracted from this report, group-correlated missingness could bias the headline effects: for example, if control-group participants who failed to install the plugin or produced no tests were preferentially excluded, the apparent treatment effect would be inflated. Please report missingness by group, compare included and excluded participants on available covariates (e.g., survey demographics and course section), and provide a sensitivity analysis treating missing outcomes as zeros or using multiple imputation. Until this is done, the central RQ1 claims rest on an untested missing-at-random assumption.","section":"§3.5 and §4.1"},{"comment":"The abstract and the introduction claim that IntelliGame led to 'better code implementations' and 'fewer failing tests of the reference test suite,' but the supporting comparison in §4.4 is only nearly significant (p = 0.054, 14.32 vs. 19.25 failing tests). At the stated alpha = 0.05, this difference is not statistically significant, and the text itself correctly hedges it as 'nearly statistically significant.' The abstract, the RQ4 summary, and the concluding sentence that 'the treatment group also achieved ... fewer failing tests' should be revised to state that the difference was in the expected direction but did not reach significance, or they should report the result with a confidence interval and a clearly stated interpretation.","section":"§4.4 and Abstract"},{"comment":"RQ3 correlates achievement levels with test-suite metrics, but the achievement levels are computed from the very activities being measured: running tests, achieving coverage, and debugging. The strong correlation between number of levels and number of tests (r = 0.65) is therefore in part a construction artifact rather than independent evidence that gamification 'leads to better-quality test suites.' The text should either reframe RQ3 as a descriptive account of how the plugin tracks behavior, or validate levels against an external outcome such as golden-suite failures; as written, the causal language in the RQ3 summary is not supported by the correlation design.","section":"§4.3, Fig. 4b"}],"minor_comments":[{"comment":"The text reports that differences are significant when confidence intervals stop overlapping, and Fig. 2b mentions '84.6% confidence intervals' without explaining why that level was chosen; please clarify the rationale for the 84.6% level or report the exact-test p-values at the relevant time points instead.","section":"§4.1, Fig. 2"},{"comment":"The figure header labels the column 'r2' but the text reports 'Pearson correlations' with values such as 0.42, which look like r, not r^2; also, the text says 'Pearson rank correlations,' which conflates Pearson and Spearman correlation. Please correct the label and use consistent terminology.","section":"§4.3, Fig. 4b"},{"comment":"There are repeated typos, including 'thetreatment' in §4.1 and §4.4, 'asumption' in Listing 3, and inconsistent spacing in 'thetreatment group' and 'thecontrol group'; a careful proofread is needed.","section":"Throughout"},{"comment":"The paper first reports '214 initially failing tests' and later says 'Out of 214 initially failing tests, 61 passed,' but the number of failing tests in RQ2 is described differently; please reconcile these counts and clarify whether they include only participant-written tests or also golden-suite failures.","section":"§4.6"},{"comment":"In Table 1, 'Take some breaks' and 'Break the Line' have identical mean values (2.31 treatment, 8.29 control) and identical p-values (0.43), which looks like a copy-paste error; please verify these entries.","section":"§4.7, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a genuine replication attempt with a solid two-group design and public artifacts, and the main behavioral results (test executions, coverage use, debug use, test count) are significant and likely robust. The major issues are localized: the missingness analysis for the 44 excluded participants must be reported, the p = 0.054 functionality result must not be presented as significant in the abstract, and the RQ3 correlations need reframing to avoid self-referential causal claims. None of these appears fatal to the study's core contribution, but they are load-bearing for the paper's stated conclusions, so major revision is appropriate. I would not require a new experiment; a careful re-analysis and rewriting should suffice."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. First, this is a genuinely careful replication of the IntelliGame effect on a new stack (TypeScript/Jest) with 174 students, and the headline behavioral effects—more tests, more executions, more coverage and debug usage—look credible. Second, the abstract's claim that the plugin led to 'better code implementations' rests on a non-significant p=0.054, and the paper never checks whether the 44 excluded participants without a TestReport are balanced across groups. Both need fixing before this is citable as stated.\n\nWhat's actually new: the adaptation of the plugin to a second language, a roughly tripled sample, and two new research questions on test quality and achievement importance. The experiment is well organized: random assignment, separate sessions to hide treatment, exact tests, and a time-travel re-analysis of date-sensitive failures. The null results for coverage and mutation score are useful, as is the honest reporting that treatment participants wrote more tests but also more test smells and slightly more failing tests. The artifact is available. These are real contributions to the empirical literature.\n\nThe soft spots are proportionate but real. The abstract and RQ4 summary overstate the functionality result: 14.32 vs 19.25 failing tests with p=0.054 is not a significant difference, and the text itself calls it 'nearly statistically significant' but the abstract converts it to 'better code implementations.' That has to be aligned. The missing-TestReport analysis is the more serious issue. Section 3.5 says 174 of 218 usable projects had the report, with the rest lost to setup issues, plugin failure, or inactivity, but the paper doesn't report the split by group. Since the TestReport is the source of the RQ1 outcome metrics, excluding 44 participants without testing them for balance could bias the very effects the paper is about. The paper should report per-group exclusions and a sensitivity analysis treating missing as zeros. Finally, RQ3's correlation between achievement levels and test metrics is partly circular—levels are earned by running tests, achieving coverage, and debugging. Frame it descriptively, not as independent evidence.\n\nWho's this for? Researchers and educators working on gamification or testing tools, and anyone doing replications in SE. It deserves a serious referee. I'd conditionally accept after the abstract is corrected, the missingness is analyzed, and RQ3 is reframed. Recommend sending to peer review.","headline":"A careful replication with credible behavioral effects, but the abstract overstates a non-significant functional result and the missing-TestReport exclusions need a group-balance check.","tokens_in":21122,"tokens_out":3036,"would_cite":true,"duration_ms":31371,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 174-participant replication reports that IntelliGame, a gamification plugin for IntelliJ, leads student developers to write and run more tests and to use coverage and debugging tools more often.","keywords":["gamification","software testing","IntelliJ IDEA","IntelliGame","replicability study","unit testing","TypeScript","empirical software engineering"],"falsifier":"Recompute the analysis with the 44 excluded projects restored where possible, and test whether missing TestReport rates differ between treatment and control; if exclusion is unbalanced, the central comparisons (test count, executions, coverage-tool use) may reflect differential attrition rather than gamification. Alternatively, a preregistered replication in which neither the task nor the incentive depends on plugin installation success could falsify the claim if the test-count gap disappears.","tokens_in":20191,"feed_emoji":"🎮","tokens_out":6892,"duration_ms":65350,"temperature":0.7,"pith_summary":"This paper tries to establish that a gamification plugin embedded in an IDE can change how developers test, and that the effect survives a move from Java to TypeScript and from 49 to 174 participants. In a controlled two-group experiment, students using IntelliGame wrote more Jest tests (15.95 vs 11.38 on average), ran tests more than twice as often, and used coverage and debug tools more than a control group that ran the plugin's tracking without visible achievements. The paper also reports that gamified participants reached passing implementations earlier and had fewer failures on a golden test suite, while code coverage and mutation scores were not significantly different between groups. If correct, this would mean that a relatively lightweight achievement system can raise testing engagement in educational settings, with the caveat that it also produced more test smells and brittle date-sensitive tests.","feed_headline":"Gamified IDE plugin boosts test output in 174-person study","feed_subtitle":"TypeScript replication: gamified students wrote more tests, ran them more often, and used coverage/debug tools.","key_machinery":"The carrying mechanism is IntelliGame itself, an IntelliJ IDEA plugin that awards multi-level achievements for testing-related actions: running tests, adding assertions, fixing failing tests, measuring coverage, debugging, and refactoring test code. Progress is displayed as trophies and progress bars, and notifications provide immediate feedback when a level is reached. In this study the plugin's achievement system is the independent variable; the central analytic move is that the control group runs the identical plugin with tracking enabled but no visible achievements, isolating the motivational effect of the game elements. Achievement levels are then used as a quantitative engagement signal, correlated with test-suite metrics.","core_discovery":"The paper's central claim is that achievements, not just monitoring, drive the effect: participants in the treatment group, who saw trophies, progress bars, and notifications, behaved differently from the control group, who ran the same data-collecting plugin without the gamified display. Specifically, the authors report significant differences for test executions ($p<0.001$), coverage-report executions ($p<0.001$), debug-mode uses ($p=0.015$), and final test count ($p=0.003$), and a near-significant difference in golden-suite failures ($p=0.054$). They further claim achievement levels correlate with test-suite quality (number of tests $r=0.65$, mutation score $r=0.48$). The replication confirms the original study's behavioral findings but not its coverage or mutation-score improvements, and it reveals that the gamified group produced more test smells and more failing tests in their own suites.","pith_inferences":["A natural next experiment would compare IntelliGame against a version that only credits passing or newly-fixed tests; this paper's finding that failing tests count equally suggests the quantity effect could be reduced by incentive alignment.","If the mechanism is simply more test executions, then professional onboarding might show similar effects, but with a risk of superficial tests (console prints, redundant cases) unless test-smell or robustness achievements are added.","The absence of significant coverage and mutation effects may be an artifact of the chosen date-library task, whose functions have few branches; a replication on branch-heavy code would tell whether the quality effects from the original study generalize.","Re-running tests at the participant's commit time resolved 61 of 214 initial failures; other experiments on time-sensitive code should adopt time-travel test execution to avoid counting environment drift as test failure."],"forward_implications":["If the replication is right, adding an achievement layer to an IDE can shift how much testing novices do within a single session, without changing the testing tools or task.","Test count and execution frequency will rise, but coverage and mutation scores may stay statistically flat, so gamification should be paired with quality-based achievements to avoid quantity-over-quality.","Achievement levels can be used as a real-time engagement signal: higher levels in this study track with more tests and better suite metrics.","Because treatment participants reported more time pressure and lower confidence, designers should calibrate achievement thresholds to avoid inducing pressure.","The golden-suite result (14.32 vs 19.25 failures, $p=0.054$) suggests gamified users implement required functionality earlier, though the effect is borderline."],"supporting_citations":[{"why":"Defines the original IntelliGame tool, its Java experiment, and the research questions and achievement thresholds this study ports to TypeScript.","marker":"[36]"},{"why":"Reports the pilot study that validated the TypeScript port, informed the reduction from 23 to 11 functions, and led to separate treatment/control sessions.","marker":"[16]"},{"why":"Provides the non-parametric rank-sum significance test used for all group comparisons.","marker":"[22]"},{"why":"Defines the test-smell taxonomy and supplies the test-smell detector used to compare test quality across groups.","marker":"[38]"},{"why":"Supplies the replicability taxonomy used to frame the study's design as a replicability study.","marker":"[13]"},{"why":"Supplies the empirical standards checklist used for reporting and validity structuring.","marker":"[27]"}],"fun_headline_variants":["Gamified IDE boosts test behavior in TypeScript replication","Achievements, not monitoring, drive test engagement in gamified IDE","Gamification ups test executions and coverage use in 174-dev study","More tests, more smells: gamified testing replication shows trade-offs","Replication: gamified IDE increases testing activity, but also smells"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result rests on the assumption that the 44 projects without a TestReport (omitted from the analysis) were lost for reasons unrelated to whether a participant was in the gamified group; if plugin setup failures were more common in one arm, the measured differences could be an artifact of who got counted.","fun_headline_variants_meta":{"raw":{"variants":["Gamified IDE boosts test behavior in TypeScript replication","Achievements, not monitoring, drive test engagement in gamified IDE","Gamification ups test executions and coverage use in 174-dev study","More tests, more smells: gamified testing replication shows trade-offs","Replication: gamified IDE increases testing activity, but also smells"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1463,"prompt_tokens":1006,"completion_tokens":457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":367}},"tokens_in":622,"tokens_out":457,"duration_ms":5062,"temperature":1.0,"reasoning_tokens":367,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:55:56.294404+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the analysis with the 44 excluded projects restored where possible, and test whether missing TestReport rates differ between treatment and control; if exclusion is unbalanced, the central comparisons (test count, executions, coverage-tool use) may reflect differential attrition rather than gamification. Alternatively, a preregistered replication in which neither the task nor the incentive depends on plugin installation success could falsify the claim if the test-count gap disappears.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the test-smell taxonomy and supplies the test-smell detector used to compare test quality across groups."},{"cited_title":"24-08-2020","cited_arxiv_id":null,"evidence_quote":"Supplies the replicability taxonomy used to frame the study's design as a replicability study."}],"review_version":1}