Pith. sign in

REVIEW 3 major objections 4 minor 33 references

Exploring the Impact of Quizzes Interleaved with Write-Code Tasks in Elementary-Level Visual Programming

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Interleaving diverse quizzes—code debugging, equivalence, and task design—with write-code tasks improves transfer of programming skill in grades 6–7, even though learning-phase performance looks identical.

desk verdict A genuinely new classroom comparison with careful materials, but the headline p<0.01 ignores class-level clustering and therefore does not yet establish the causal claim. read the letter →

arxiv 2411.14275 v1 pith:RVGLBY6J submitted 2024-11-21 cs.CY

Ahana Ghosh , Liina Malva , Alkis Gotovos , Danial Hooshyar , Adish Singla This is my paper · ORCID
classification cs.CY
keywords quizzesblock-basedvisualprogrammingK-8studentscomputationalthinkingBloom'srevisedtaxonomytransferoflearningHourCodewrite-codetasks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the type of quiz interleaved into an elementary block-based programming curriculum matters for what students can do afterward. It compares two curricula built on the same 20 Hour of Code: Maze Challenge write-code tasks: one (HoC-ACE) interleaved with 21 diverse multiple-choice quizzes covering solution tracing, code debugging, code equivalence, and task design, and one (HoC-Fill) with 21 fill-in-the-gap solution-finding quizzes matched to the same grids. In a study with 405 students in grades 6–7, the two groups performed about equally on the learning-phase tasks, but on a post-test of 15 transfer tasks, the diverse-quiz group scored 0.799 versus 0.702 for the fill-in-the-gap group, a difference significant at p<0.01. The paper interprets this as evidence that quizzes built on higher-order skills (analyzing, evaluating, creating in Bloom's revised taxonomy) transfer better than basic solution-finding quizzes, which may even encourage an over-reliance that hurts later performance.

What carries the argument

The mechanism is the interleaving of write-code tasks (T01–T20, from Hour of Code: Maze Challenge) with 21 quizzes occupying identical sequence positions in both groups, so the only difference is the quiz type. The experimental quizzes Q01–Q21 come from the ACE computational thinking assessment and target three Bloom's revised taxonomy levels—analyzing, evaluating, creating—through solution tracing, code debugging, code equivalence, and task design; the control quizzes F01–F21 are fill-in-the-gap solution-finding items at the analyzing level only. Transfer is measured with PostHoC, 15 write-code tasks of which P01–P07 are familiar and P08–P15 are harder, with P10–P12 combining concepts never seen together in HoC.

What would settle it

Give both groups a genuine programming pre-test before the curriculum; if controlling for pre-test scores eliminates the 0.097 PostHoC advantage, the claim that richer quizzes caused the transfer gain is false.

Watch

Extended reading notes

Core claim

The central claim is that quiz diversity, not quiz presence, drives durable learning in elementary visual programming. The authors design HoC-ACE and HoC-Fill so that the only systematic difference is the cognitive level of the interleaved quizzes: ACE quizzes require students to trace solutions, debug erroneous code, judge code equivalence, and design grids or tasks, whereas FILL quizzes only ask which single block completes a given partial solution. On the post-learning phase, HoC-ACE outperformed HoC-Fill in aggregate (0.799 vs 0.702, p<0.01) and on every item category, including the novel combination items P10–P12 (0.742 vs 0.593). The paper also reports that relative to a no-quiz baseline on PostHoC, the ACE group was above baseline (+0.036) while the FILL group was below (−0.061), suggesting basic quizzes can transfer negatively. The authors conclude that richer quizzes aligned with higher-order Bloom's levels yield higher post-learning utility across grades and experience levels, with the largest gap in grade 6.

Load-bearing premise

The study assumes that the two groups were equally skilled at programming before the intervention, since no pre-test of programming ability was given; the only equivalence checks were self-reported years of experience and grade level.

Editorial extensions

If this is right

  • Changing only the quiz type, while keeping write-code tasks and quiz positions fixed, changes post-learning transfer by a significant margin.
  • Quizzes built on analyzing, evaluating, and creating skills transfer to novel task combinations (P10–P12) more than solution-finding quizzes do.
  • Fill-in-the-gap solution-finding quizzes can be worse than no quizzes at all for later transfer, since HoC-Fill scored below the no-quiz baseline.
  • The advantage appears across grades 6 and 7 and across all self-reported experience levels, and is largest for grade 6 students.
  • Learning-phase performance alone would have missed the difference: both groups looked similar during the curriculum itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The negative baseline gap for HoC-Fill is the most actionable result: it suggests that easy, repetitive fill-in-the-gap quizzes may train the immediate answer rather than the underlying planning, a hypothesis the paper raises but does not test.
  • A direct extension would be to split the ACE quizzes into separate arms (tracing only, debugging only, task design only) to see which skill drives the transfer gain.
  • The result likely generalizes beyond block-based mazes to any scaffolded programming curriculum where quizzes are interleaved, but that generalization is the authors' implicit hope, not a claim established here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports a classroom study comparing two versions of code.org's Hour of Code: Maze Challenge for grades 6-7, in which write-code tasks are interleaved with quizzes. The experimental group, HoC-ACE, received quizzes targeting analyzing, evaluating, and creating skills (solution tracing, code debugging, code equivalence, task design), while the control group, HoC-Fill, received fill-in-the-gap solution-finding quizzes. With 405 students in 37 classes assigned at the class level, the two groups performed similarly during the learning phase on the HoC tasks, but HoC-ACE outperformed HoC-Fill on the 15-item PostHoC post-learning transfer tasks (normalized aggregate performance 0.799 vs. 0.702, reported p < 0.01). The paper interprets this as evidence that richer quiz types improve transfer of problem-solving skills in elementary block-based programming.

Significance. If the reported effect is valid, the paper makes a useful contribution to K-8 computing education by going beyond the general question of whether quizzes help and addressing which quiz types promote post-learning transfer. The study's scale (405 students, 17 schools), the use of a standard curriculum, and the design of a separate post-test item set are strengths. The outcome measure PostHoC consists of code.org write-code tasks that are not taken from the ACE item bank used to source the experimental quizzes, so the central comparison is not forced by construction. The authors are also candid in their limitations section about the lack of a true pre-test and about class-level randomization. However, the statistical analysis as reported does not yet align with the experimental design, and one non-random baseline comparison is over-interpreted.

major comments (3)
  1. [Section 5.2, Figure 6b] The headline result, HoC-ACE 0.799 vs. HoC-Fill 0.702 with chi-square = 75.1 and p < 0.01, is reported without accounting for the fact that randomization was at the class level (37 classes, 17 schools; Section 4.1). If the chi-square test pooled item-level responses across students and tasks, it treats observations within a class as independent even though students share teachers, schools, and classroom contexts. With 37 randomized units, the effective sample size is much smaller than the 405 students or the thousands of item responses implied by the test. The paper should report a cluster-robust analysis, a mixed-effects model with random intercepts for class or school, or a permutation test at the class level, together with the intraclass correlation. Because this p-value is the main quantitative support for the causal claim in RQ2, this is a load-bearing gap.
  2. [Section 4.3 and Section 5.2] The baseline comparison is not a proper control for the HoC-ACE vs. HoC-Fill comparison. The baseline data come from 14 classes that were requested separately from the 37 randomized classes, received no quizzes, and completed the standard HoC curriculum in one 45-minute lesson, whereas the two intervention groups received two 45-minute lessons with interleaved quizzes. The differences of Δ = +0.036 for HoC-ACE and Δ = -0.061 for HoC-Fill are therefore subject to selection effects, dosage effects, and time-on-task confounds. The statement that HoC-Fill's lower score 'suggests an overreliance on basic quizzes' (Section 5.2) is not supported by this comparison. The baseline should be presented only as descriptive context, or dropped, unless a comparable randomized control condition can be reported.
  3. [Section 4.2 and Section 5.4] The equivalence of the two groups before the intervention is checked only by grade and self-reported years of programming experience, not by an actual measure of programming skill. The authors acknowledge this limitation, but it remains a direct threat to the causal interpretation of RQ2. With 37 classes randomized, unmeasured differences in prior ability could easily be correlated with the class-level assignment, and the reported post-test gap could partly reflect initial differences. A conservative analysis that includes class-level prior achievement, or at least a cluster-level summary of the pretest measures, would materially strengthen the claim.
minor comments (4)
  1. [Sections 4.3 and 5.2] Please define 'normalized aggregated performance' explicitly: is it the mean of per-student proportions, or the proportion of correct item-level responses pooled over students, and what is the unit of analysis for the chi-square tests? This matters for interpreting the reported standard errors and p-values.
  2. [Figures 5b and 6b] The column structure is confusing because the 'Time (s)' column is not a performance column. Consider separating performance metrics and time metrics, or labeling the table more clearly.
  3. [Section 5.1] The conclusion that HoC-ACE spent 'double the time' on quizzes is based on total time per item, but the groups also differ in number of attempts per quiz (1.88 vs. 1.32). Reporting time per attempt or time per successful attempt would make the comparison more informative.
  4. [Section 5.4] The limitation paragraph correctly notes the lack of an explicit pre-test, but the reader is left without a sense of how large a bias this could introduce. A sentence quantifying the imbalance in self-reported experience (which is small) and the likely sensitivity of the main result to a plausible pretest imbalance would help.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the ACE quizzes are a self-cited treatment component, but the post-learning outcome comes from external code.org tasks and the central comparison is not forced by construction.

full rationale

The paper's central claim is that HoC-ACE outperformed HoC-Fill on PostHoC in the post-learning phase (Section 5.2). The only notable self-citation is the use of quiz items from the authors' own ACE assessment [12]: 'For the quizzes, we selected multi-choice questions from the Computational Thinking assessment, ACE, specifically designed for K-8 learners [12]' (Section 3.2). However, this supplies the intervention material, not the outcome measure. The post-learning outcome PostHoC is described as 'write-code tasks from the Hour of Code: Maze Challenge domain' and is composed of items from code.org's HoC curriculum plus newly designed write-code tasks (Section 3.3), not from ACE. The reported chi-square comparisons are direct group comparisons on PostHoC performance, not fitted parameters renamed as predictions. The Bloom's taxonomy classification is grounded in the external framework of Anderson and Krathwohl [1], not in a self-authored uniqueness theorem. Section 5.4 acknowledges limitations around the absence of a pre-test and class-level randomization; these are validity concerns, not circularity. No step in the paper's argument reduces by definition or by self-citation to its own inputs, so the circularity burden is low.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities. The analysis relies on standard statistical assumptions and on the construct validity of ACE quizzes and PostHoC transfer tasks.

assumptions (3)
  • domain assumption ACE quiz items measure the higher-order skills (analyzing, evaluating, creating) they are grouped into.
    The experimental condition is defined by these quiz types; if the quizzes do not in fact exercise those skills, the interpretation of the treatment changes. The ACE taxonomy comes from the authors' prior work [12].
  • domain assumption PostHoC tasks measure transfer of learned skills.
    The post-learning outcome is used as evidence of utility; if performance on these new maze tasks does not reflect transferable problem-solving skill, the main conclusion is weakened.
  • domain assumption Class-level randomization balances unmeasured confounders such as motivation and prior knowledge.
    No pre-test was given; equivalence rests on this assumption plus grade and experience checks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring the Impact of Quizzes Interleaved with Write-Code Tasks in Elementary-Level Visual Programming." pith.science (2026). https://pith.science/paper/RVGLBY6J

@misc{pith2026241114275,
  author       = {Pith},
  title        = {Pith review of: Exploring the Impact of Quizzes Interleaved with Write-Code Tasks in Elementary-Level Visual Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RVGLBY6J}},
  note         = {Machine review of arXiv:2411.14275}
}
read the original abstract

We explore the role of quizzes in elementary visual programming domains popularly used for K-8 computing education. Prior work has studied various quiz types, such as fill-in-the-gap write-code questions. However, the overall impact of these quizzes is unclear: studies often show utility in the learning phase when enhanced with quizzes, though limited transfer of utility in the post-learning phase. In this paper, we aim to better understand the impact of different quiz types and whether quizzes focusing on diverse skills (e.g., code debugging and task design) would have higher utility. We design a study with Hour of Code: Maze Challenge by code.org as the base curriculum, interleaved with different quiz types. Specifically, we examine two learning groups: (i) HoC-ACE with diverse quizzes including solution tracing, code debugging, code equivalence, and task design; (ii) HoC-Fill with simple quizzes on solution finding. We conducted a large-scale study with 405 students in grades 6--7. Our results highlight that the curriculum enhanced with richer quizzes led to higher utility during the post-learning phase.

Figures

Figures reproduced from arXiv: 2411.14275 by the authors.

Figure 1
Figure 1. Sequence of items in the groups HoC-ACE and HoC-Fill during the learning and post-learning phases. The items in the left gray box are used during the learning phase. In this set, the light blue circles containing “T” represent write-code tasks taken from HoC and the green and orange squares represent quizzes. The items in the right gray box are used during the post-learning phase. In this set, the dark blue circles … view at source ↗
Figure 2
Figure 2. Distribution of items used in our study based on programming concepts of their solution codes. The quizzes are also [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) and (b) show examples of write-code tasks from [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Distribution of students’ grade (year of study) on [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Performance in the learning phase. (a) shows the fraction of students who scored above a certain threshold on [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Performance in the post-learning phase. (a) shows the fraction of students who scored above a certain threshold on [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Performance on PostHoC w.r.t experience. Aggregated mean (std err) performance scaled between 0 and 1 on PostHoC based on students’ grade (year of study) and years of prior programming experience. ∗∗ indicates significance with 𝑝 < 0.01. when accounting for students’ p…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 32 canonical work pages

  1. [1]

    Lorin W Anderson and David R Krathwohl. 2001. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Addison Wesley Longman, Inc

    work page 2001

  2. [2]

    Benjamin S Bloom, Max D Engelhart, Edward J Furst, Walker H Hill, and David R Krathwohl. 1956. Taxonomy of Educational Objectives: The Classification of Edu- cational Goals. Handbook 1: Cognitive Domain . Longman New York

    work page 1956

  3. [3]

    Matt Bower. 2008. A Taxonomy of Task Types in Computing. In Proceedings of the Conference on Innovation and Technology in Computer Science Education (ITiCSE)

    work page 2008

  4. [4]

    William G Cochran. 1952. The 𝜒2 Test of Goodness of Fit. The Annals of Mathematical Statistics (1952), 315–345

    work page 1952

  5. [5]

    Code.org. 2022. Code.org. https://code.org/

    work page 2022

  6. [6]

    Code.org. 2022. Hour of Code: Classic Maze Challenge. https://studio.code.org/s/ hourofcode

    work page 2022

  7. [7]

    Code.org. 2024. Express Course (2024). https://studio.code.org/s/express-2024

    work page 2024

  8. [8]

    Will Crichton and Shriram Krishnamurthi. 2024. Profiling Programming Lan- guage Learning. In Proceedings of the Conference on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)

    work page 2024

Show all 33 references
  1. [9]

    Louis Deslauriers, Logan S McCarty, Kelly Miller, Kristina Callaghan, and Greg Kestin. 2019. Measuring Actual Learning Versus Feeling of Learning in Response to Being Actively Engaged in the Classroom. Proceediings of National Academy of Sciences (PNAS) 116 (2019), 19251–19257

    2019

  2. [10]

    Alexandru Ene and Cosmin Stirbu. 2019. Automatic Generation of Quizzes for Java Programming Language. In Proceedings of the European Conference on Artificial Intelligence (ECAI)

    2019

  3. [11]

    Ericson, Hisamitsu Maeda, and Paramveer S

    Barbara J. Ericson, Hisamitsu Maeda, and Paramveer S. Dhillon. 2022. Detecting Struggling Students from Interactive Ebook Data: A Case Study Using CSAwe- some. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE)

    2022

  4. [12]

    Ahana Ghosh, Liina Malva, and Adish Singla. 2024. Analyzing–Evaluating– Creating: Assessing Computational Thinking and Problem Solving in Visual Programming Domains. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE)

    2024

  5. [13]

    Ahana Ghosh, Sebastian Tschiatschek, Sam Devlin, and Adish Singla. 2022. Adap- tive Scaffolding in Block-Based Programming via Synthesizing New Tasks as Pop Quizzes. In Proceedings of the International Conference on Artificial Intelligence in Education (AIED)

    2022

  6. [14]

    Mohammed Hassan and Craig B. Zilles. 2021. Exploring ‘reverse-tracing’ Ques- tions as a Means of Assessing the Tracing Skill on Computer-based CS 1 Exams. In Proceedings of the Conference on International Computing Education Research (ICER)

    2021

  7. [15]

    Hilton, Genevieve M

    Andrew D. Hilton, Genevieve M. Lipp, and Susan H. Rodger. 2019. Translation from Problem to Code in Seven Steps. In Proceedings of the Conference on Global Computing Education (CompEd)

    2019

  8. [16]

    Xinying Hou, Barbara Jane Ericson, and Xu Wang. 2022. Using Adaptive Parsons Problems to Scaffold Write-Code Problems. In Proceedings of the Conference on International Computing Education Research (ICER)

    2022

  9. [17]

    Jeffrey D Karpicke and Janell R Blunt. 2011. Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping. Science 331 (2011), 772–775

    2011

  10. [18]

    Williams, and Kelli M

    Lucas Layman, Laurie A. Williams, and Kelli M. Slaten. 2007. Note to Self: Make Assignments Meaningful. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE)

    2007

  11. [19]

    Ko, Will Jernigan, Alannah Oleson, Christopher J

    Dastyni Loksa, Amy J. Ko, Will Jernigan, Alannah Oleson, Christopher J. Mendez, and Margaret M. Burnett. 2016. Programming, Problem Solving, and Self- Awareness: Effects of Explicit Guidance. In Proceedings of the Conference on Human Factors in Computing Systems (CHI)

    2016

  12. [20]

    John Maloney, Mitchel Resnick, Natalie Rusk, Brian Silverman, and Evelyn East- mond. 2010. The Scratch Programming Language and Environment. Transactions on Computing Education (TOCE) 10 (2010), 1–15

    2010

  13. [21]

    Almstrum, Danny Diaz, Mark Guzdial, Dianne Hagan, Yifat Ben-David Kolikant, Cary Laxer, Lynda Thomas, Ian Utting, and Tadeusz Wilusz

    Michael McCracken, Vicki L. Almstrum, Danny Diaz, Mark Guzdial, Dianne Hagan, Yifat Ben-David Kolikant, Cary Laxer, Lynda Thomas, Ian Utting, and Tadeusz Wilusz. 2001. A Multi-national, Multi-institutional Study of Assessment of Programming Skills of First-Year CS Students. In...

    2001

  14. [22]

    Victor-Alexandru Pădurean and Adish Singla. 2024. Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programming. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks

    2024

  15. [23]

    Yulia Pechorina, Keith Anderson, and Paul Denny. 2023. Metacodenition: Scaf- folding the Problem-Solving Process for Novice Programmers. In Proceedings of the Australasian Computing Education Conference (ACE)

    2023

  16. [24]

    David N Perkins and Gavriel Salomon. 1992. Transfer of Learning. International Encyclopedia of Education (1992)

    1992

  17. [25]

    Ron Porter and Paul R. Calder. 2004. Patterns in Learning to Program - An Experiment?. In Proceedings of the Australasian Computing Education Conference (ACE)

    2004

  18. [26]

    Price, Rui Zhi, and Tiffany Barnes

    Thomas W. Price, Rui Zhi, and Tiffany Barnes. 2017. Evaluation of a Data- driven Feedback Algorithm for Open-ended Programming. In Proceedings of the International Conference on Educational Data Mining (EDM)

    2017

  19. [27]

    Alexander Ruf, Marc Berges, and Peter Hubwieser. 2013. Types of Assignments for Novice Programmers. InProceedings of the Workshop in Primary and Secondary Computing Education (WiPSCE)

    2013

  20. [28]

    Alexander Ruf, Marc Berges, and Peter Hubwieser. 2015. Classification of Pro- gramming Tasks According to Required Skills and Knowledge Representation. In Proceedings of the Conference on Informatics in Schools: Situation, Evolution, and Perspectives (ISSEP)

    2015

  21. [29]

    Reza Soltanpoor, Charles Thevathayan, and Daryl J. D’Souza. 2018. Adaptive Remediation for Novice Programmers through Personalized Prescriptive Quizzes. In Proceedings of the Conference on Innovation and Technology in Computer Science Education (ITiCSE)

    2018

  22. [30]

    J Ángel Velázquez-Iturbide. 2022. Designing Exercises for Block-based Languages: The Case of ScratchJr. InProceedings of the Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM)

    2022

  23. [31]

    Bernard L Welch. 1947. The Generalization of Students’ Problem When Several Different Population Variances Are Involved. Biometrika (1947)

    1947

  24. [32]

    Lishan Zhang, Baoping Li, Qiujie Zhang, and I-Han Hsiao. 2020. Does a Dis- tributed Practice Strategy for Multiple Choice Questions Help Novices Learn Programming? International Journal of Emergent Technology and Learning (2020)

    2020

  25. [33]

    Price, Samiha Marwan, Alexandra Milliken, Tiffany Barnes, and Min Chi

    Rui Zhi, Thomas W. Price, Samiha Marwan, Alexandra Milliken, Tiffany Barnes, and Min Chi. 2019. Exploring the Impact of Worked Examples in a Novice Pro- gramming Environment. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE)

    2019

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.