{"id":"657fac11-3711-49d1-857f-8b8377cd03e6","arxiv_id":"1908.06637","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Two teaching concepts, a voluntary game-building lecture and a planned MOOC, aim to engage non-CS students and women in computer science through mobile app development.","lead":"This paper describes a university course where students from any major build simple games on smartphones using the Pocket Code app, plus a planned free online course based on it. A smart generalist might read it to see one concrete attempt to attract non-programmers and women to computer science.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central evidence for the lecture's success — 202 diverse participants — is undermined by an internal contradiction: the abstract calls the course voluntary, while Section 3 calls it obligatory; the claim's force depends on which is true.","rationale":"I read the paper as an experience report with a modest forward-looking claim: the 2018 lecture and the planned MOOC are 'promising concepts' for broadening participation in CS. The reader's conditional verdict is appropriate because the paper is transparent about its work-in-progress status and does not overclaim to have measured learning outcomes. However, I found a more specific load-bearing problem than the missing evaluations: the abstract says the lecture was voluntary, while Section 3 twice calls it 'obligatory' (and Section 4 repeats 'obligatory lecture'). The 202-participant diversity statistic is the only observational evidence for the lecture's appeal, and its interpretation changes radically depending on which description is correct. If enrollment was required, the statistic cannot support the 'successful/engaging' conclusion, and the MOOC's design rationale loses its empirical anchor. This is an internal inconsistency, not a disagreement with the field's consensus, and it is easily fixed by checking the course catalog. I therefore do not move the verdict away from CONDITIONAL, but the revision must clarify this point and, if the lecture was voluntary, provide the evaluation data and a gender breakdown. Partial agreement with the reader: we both see insufficient outcome evidence, but the reader's weakest assumption ('popularity and unshown evaluations') presumes the popularity is real; I am questioning whether the popularity evidence even counts as voluntary.","tokens_in":9318,"tokens_out":5777,"duration_ms":53861,"concrete_test":"Check the official University of Graz course catalog and registration records for summer term 2018 to determine the classification of 'Design your own App' (voluntary free elective, elective with required enrollment for certain programs, or obligatory Pflichtlehrveranstaltung). If it was voluntary, retrieve and present the 2018 evaluation questionnaire results and the gender distribution of the 202 participants; if it was obligatory, revise the abstract and the success claim to reflect that enrollment was not voluntary.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that the voluntary lecture 'Design your own App' successfully engaged a diverse set of non-CS students rests on the observation that 202 students from 120+ degree programs participated (Section 3). But the abstract explicitly frames this as a 'voluntary lecture,' while Section 3 states that 'we introduced for the first time the obligatory lecture ... for all bachelor's, master's, and doctoral students at the University of Graz,' and Section 4 repeats the phrase 'obligatory lecture.' These are incompatible descriptions of the same intervention. If the lecture was obligatory, or even a required component of some degree programs, then the enrollment numbers primarily measure administrative compulsion rather than voluntary interest, and the claimed demonstration of appeal and diversity is not supported. The MOOC concept is then justified as 'based on the experiences and feedback gained from the obligatory lecture' (Section 4), so the weak link propagates to the MOOC promise. Even if the lecture was voluntary, the paper reports no evaluation instrument or results — the abstract's 'positive evaluations' do not appear in the body — no gender distribution of the 202 participants, and no learning or attitude outcome, so the gender-disparity component of the claim is especially underdetermined. The voluntary/obligatory contradiction must be resolved before the evidence can bear the conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports on two educational interventions aimed at broadening participation in computer science: a 2018 university lecture called 'Design your own App' that used the Pocket Code visual programming tool, and a planned MOOC, 'Get FIT in Computer Science', to be launched on the Austrian iMooX platform. The authors report that 202 students from over 120 degree programs attended the lecture, with 135 completing an initial survey, and they argue that the lecture's success motivated the MOOC design. The MOOC is described as a seven-module course with videos, exercises, and a final Pocket Code game-design project, targeting young women, teenagers, teachers, and the general public. The central claim is that these two concepts are 'promising' ways to address low CS knowledge and gender disparity in technology fields.","tokens_in":9465,"tokens_out":2351,"duration_ms":25881,"significance":"If the empirical claims were properly supported, the paper would make a useful contribution to CS-education practice: it documents a concrete, transferable course design (Game Development-Based Learning with Pocket Code), integrates a voluntary university course with a national MOOC platform (iMooX), and explicitly targets under-represented groups including women and non-CS students. The planned MOOC is a concrete artifact with clearly specified modules and open educational resources. However, the paper's current evidentiary basis is thin: the key evaluation results are not shown, the course status is described inconsistently, and no learning, attitude, or gender-outcome data are reported. The design and rationale are plausible, but the central claims of success and promise are not yet established.","major_comments":[{"comment":"The intervention is described with contradictory terms: the abstract calls the lecture 'voluntary', Section 1 calls it an 'elective lecture', but Section 3 states 'we introduced for the first time the obligatory lecture ... for all bachelor's, master's, and doctoral students at the University of Graz', and Section 4 refers to the 'obligatory lecture'. This is not a terminological nuance: if the lecture was obligatory, the observed enrollment of 202 students measures administrative compulsion rather than voluntary interest, and the claimed demonstration of appeal to non-CS students is not supported. If it was voluntary, then Section 3 must be corrected. The distinction is load-bearing for the paper's central claim and must be resolved, with a precise statement of which students were required or encouraged to attend.","section":"Abstract, Section 1, Section 3, Section 4"},{"comment":"The abstract states that the course 'received positive evaluations', but the body of the paper does not present any evaluation instrument, scale, results, or statistical summary. The only quantitative evidence is Figure 3, labeled 'Expectations of the course', but the figure has no axis labels, no legend, and no accompanying explanation of what the plotted values represent. The reader cannot verify the 'positive evaluations' claim or the claim that the lecture was 'very successful'. The authors should include the actual evaluation questionnaire and its results, with a breakdown by gender and major, as well as outcome measures such as learning gains, attitude change, or subsequent enrollment in CS courses.","section":"Section 3, Figure 3"},{"comment":"The paper's stated motivation includes the gender disparity in CS, yet no gender distribution is reported for the 202 participants or for the 135 survey respondents. Without this data, the claim that the lecture engaged 'a diversity of students' with respect to gender, and the implicit claim that the approach helps address gender disparity, is unsupported. The MOOC section describes design choices meant to attract women (all explanations by women, a video on female scientists), but these are not evaluated. The authors should at minimum report the gender breakdown of participants and, ideally, pre/post measures related to interest, self-efficacy, or sense of belonging in CS.","section":"Section 3"},{"comment":"The MOOC is described entirely prospectively: at the time of writing it had not been launched, and no enrollment, retention, or learning data are reported. This is acceptable for a concept paper, but the paper's conclusion calls the MOOC and the lecture 'two promising concepts' and asserts they are 'based on the experiences and feedback gained from the obligatory lecture' (Section 4). Since the lecture feedback is never presented, the claimed evidentiary link from the lecture to the MOOC design is not verifiable. The authors should either present the lecture feedback that informed the MOOC, or explicitly frame the MOOC section as design rationale rather than as an empirically validated result.","section":"Section 4"}],"minor_comments":[{"comment":"The caption contains a typo: 'Percentage if of Computer Science degree programs' should read 'Percentage of Computer Science degree programs'. The figure's y-axis label and source should be clarified, and the data range (2003/04 to 2017/18) should be reconciled with the text that says 'winter term 2017/18'.","section":"Figure 1"},{"comment":"The sentence '20 hours per student was allotted' should be rephrased for agreement, e.g., '20 hours per student were allotted' or 'Students were allotted 20 hours'.","section":"Section 3"},{"comment":"The paper uses both 'Bebras' and 'Brebas'; the correct spelling is 'Bebras'. Also, the reference to the 'Brebas Challenge Austria' URL should be checked, as the contest is typically titled 'Biber' in Austria and Germany.","section":"Section 3 and 2.2"},{"comment":"The reference 'Pappana, L. (2012)' should be 'Pappano, L.' The New York Times article is by Laura Pappano. Also, the reference 'Khaleel et. al.' uses an inconsistent abbreviation style; this should be harmonized with the other references.","section":"References"},{"comment":"The paragraph describing the one-week September lecture says 'the goal to do programming lessons together, ask questions and to work on their final submission'; this should be rephrased for parallelism, e.g., 'to do programming lessons together, to ask questions, and to work on their final submission'.","section":"Section 4"},{"comment":"Figure 3 lacks a description of the survey items, response scale, and sample size in the caption. Please add these details and, ideally, the exact question wording, so readers can interpret the plotted values.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the ECGBL community, but the evidentiary gap and the voluntary/obligatory contradiction need to be addressed before the claims can be accepted. If the lecture was genuinely voluntary, adding the evaluation data would substantially strengthen the paper; if it was obligatory, the authors should reframe the claims to avoid implying that enrollment reflects voluntary interest. I would not recommend rejection, as the design and MOOC concept are potentially useful, but the current version does not provide the evidence needed to support its central assertions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a short experience report from ECGBL about a university lecture and a planned MOOC, not a controlled study. The genuinely useful part is the observation that a game-development-based elective using Pocket Code attracted 202 students from over 120 degree programs, with a large share from non-CS fields. That kind of cross-disciplinary pull is worth knowing about, and the paper describes the course structure and final project requirements in enough detail to be replicable. The MOOC design, with female explainers and OER materials, is a reasonable extension.\n\nThe paper's weaknesses are real but mostly addressable. The abstract says the lecture was voluntary; Section 3 calls it 'obligatory' and Section 4 repeats that. That is not a minor wording issue. The entire claim that the course successfully engaged non-CS students depends on students choosing to attend. If the course was required, the enrollment numbers are less meaningful. The authors need to clarify this. Also, while the paper mentions 'positive evaluations' and shows a figure of student expectations, it never presents the evaluation instrument or results, and there is no gender breakdown of the 202 participants. So the gender-disparity part of the argument is largely assertion. The MOOC had not launched when the paper was written, so its described impact is prospective.\n\nThe literature review is adequate, and the authors are transparent that pre/post testing and MOOC evaluation are planned. I don't see a hidden agenda or overclaiming beyond what experience reports usually do. The main fix is to resolve the voluntary/obligatory contradiction and include whatever 2018 evaluation data they actually have. If they cannot show that attendance was voluntary, they should soften the conclusion accordingly.\n\nThis paper deserves a serious referee because the intervention is concrete and the diversity data, if confirmed, would be a useful data point for broadening participation. I would not cite it yet, but I might bring it to a reading group to discuss reporting standards in CS education research.\n\nRecommendation: send it out, but the reviewers should ask for a clear statement on the course's formal status and for the underlying evaluation numbers.","headline":"Useful experience report with an intriguing diversity data point, but the voluntary/obligatory mismatch and missing evaluation data undercut the central claim.","tokens_in":154,"tokens_out":2657,"would_cite":false,"duration_ms":40079,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A voluntary smartphone game-building course can draw a broad mix of non-CS students into programming, and the same approach is being scaled into a MOOC for teenagers, women, and teachers.","keywords":["computer science education","digital literacy","game development based learning","Pocket Code","MOOC","computational thinking","gender disparity in CS","non-CS students"],"falsifier":"Run the lecture or MOOC with a pre-test and post-test of computational thinking, track MOOC completion rates and gender balance, and follow participants to see whether they enroll in further CS courses; if the game-development group shows no greater learning or later CS enrollment than a conventional introductory session, or if the MOOC reproduces the usual dropout and gender skew, the central claim fails.","tokens_in":9029,"feed_emoji":"🎮","tokens_out":7950,"duration_ms":82596,"temperature":0.7,"pith_summary":"The paper tries to establish that a playful, low-threshold introduction to programming can reach people who normally stay away from computer science, and that the same design can be scaled from a university lecture to a massive open online course. In 2018, 202 students from over 120 degree programs took a voluntary lecture in which they built games directly on smartphones with the visual coding tool Pocket Code; the authors report positive evaluations and call the mix of majors impressive. That experience led them to design the MOOC titled Get FIT in Computer Science, a seven-module course that teaches CS fundamentals through game-design exercises in Pocket Code, aimed at young women, teenagers, teachers, and the general public. If the paper is right, a short, game-centered course, and its MOOC equivalent, can act as an alternative entry point into computing for people with no prior CS background.","feed_headline":"Voluntary app-design lecture drew 202 non-CS students","feed_subtitle":"A game-based coding course could widen the path into computer science for women, teens, and teachers.","key_machinery":"The load-bearing mechanism is Game Development-Based Learning (GDBL) carried by Pocket Code, a visual block-based programming environment that runs directly on smartphones and lets learners assemble games without typing syntax. The lecture structure turns this into a repeatable format: 30 minutes of theory, 15 minutes of guided hands-on challenges, small homework tasks, and a final project in which pairs from different majors design a game with specified genre, theme, goal, and extra elements such as sensors or quizzes. The planned MOOC transfers this format into seven modules combining videos, interactive exercises, self-assessment questions, and a final Pocket Code game, with presentation choices such as female narrators and a module on famous women in computing meant to counter stereotypes.","core_discovery":"The paper's central claim is that a playful, hands-on introduction to programming built around making games on a smartphone can bring computer science within reach of people who would not otherwise study it. In 2018, 202 students from 120 different degree programs enrolled in a voluntary lecture that used the visual coding tool Pocket Code, after which students built their own games in mixed-major pairs for a final project; the authors report positive evaluations and use this experience to argue that the same design can be transferred to a seven-module MOOC aimed at young women, teenagers, teachers, and anyone who wants a more realistic picture of what CS is. The paper presents both the lecture and the planned MOOC as promising concepts for overcoming low CS knowledge in schools and the gender disparity in technology fields.","pith_inferences":["If the effect is real rather than novelty, this lecture format could become a reusable CS-literacy elective at other universities; the experiment that would show this is the pre/post computational-thinking test the authors say they have planned.","The MOOC's stereotype-countering choices could be tested against an otherwise identical version without female narrators and without the women-in-CS video, isolating whether representation drives engagement.","Because the MOOC materials are open educational resources, their largest long-term effect may come through teachers who reuse them, a channel the paper mentions but does not measure."],"forward_implications":["A single elective course can draw hundreds of students from more than 120 majors, including teacher-training, business, law, and psychology students.","Game-development-based exercises can be delivered entirely on smartphones, making the approach feasible on hardware students already own.","The same materials, released as open educational resources, give high-school teachers a ready-made way to bring CS concepts into class.","Putting women on camera and discussing women in computing are concrete, low-cost design choices aimed at countering stereotype-based disinterest."],"supporting_citations":[{"why":"Supplies the guideline for game-development-based learning that motivates the lecture's design.","marker":"(Wu and Wang, 2012)"},{"why":"Supplies the prior GDBL experience and the final-project requirements used in the course.","marker":"(Spieler and Slany, 2018)"},{"why":"Defines computational thinking as a universal skill, the justification for teaching CS beyond majors.","marker":"(Wing, 2006)"},{"why":"Documents stereotype barriers that discourage women, motivating the MOOC's counter-stereotypical design choices.","marker":"(Cheryan et al., 2013)"},{"why":"Establishes MOOC dropout rates as the problem the blended on-campus follow-up is meant to mitigate.","marker":"(Khalil and Ebner, 2014)"},{"why":"Describes inverse blended learning, the didactic model pairing the MOOC with a face-to-face week.","marker":"(Ebner and Schön, 2019)"},{"why":"Provides Austrian STEM gender data showing the disparity the interventions target.","marker":"(Binder et al., 2017)"}],"fun_headline_variants":["Pocket Code app draws 202 non-CS students to build games","Game-based coding lecture lures 202 students from 120 majors","Smartphone game dev course sparks CS interest in 202 students","New MOOC aims to make computer science fit for everyone"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that high enrollment and positive evaluations show the approach helps overcome low CS knowledge and gender disparity, even though the paper reports no measurement of learning, attitude change, or later CS choices.","fun_headline_variants_meta":{"raw":{"variants":["Pocket Code app draws 202 non-CS students to build games","Game-based coding lecture lures 202 students from 120 majors","Smartphone game dev course sparks CS interest in 202 students","New MOOC aims to make computer science fit for everyone"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1352,"prompt_tokens":1030,"completion_tokens":322,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":250}},"tokens_in":646,"tokens_out":322,"duration_ms":3915,"temperature":1.0,"reasoning_tokens":250,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:37:17.764115+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the lecture or MOOC with a pre-test and post-test of computational thinking, track MOOC completion rates and gender balance, and follow participants to see whether they enroll in further CS courses; if the game-development group shows no greater learning or later CS enrollment than a conventional introductory session, or if the MOOC reproduces the usual dropout and gender skew, the central claim fails.","supporting_citations":[],"review_version":1}