Pith. sign in

REVIEW 4 major objections 3 minor 38 references

Investigating the Use of Productive Failure as a Design Paradigm for Learning Introductory Python Programming

T0 review · 4 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read For introductory Python lists, having students struggle through an unfamiliar sensor-data task before any lesson yields the same immediate performance as direct instruction, better retention two weeks later, and a larger drop in measured…

desk verdict A well-designed PF pilot whose headline retention and cognitive-load claims outrun the evidence; the cognitive-load comparison is confounded by task timing. read the letter →

arxiv 2411.11227 v1 pith:T7LAXWQY submitted 2024-11-18 cs.CY cs.HC

classification cs.CYcs.HC
keywords productivefailuredirectinstructionPythonlistsCS1educationcognitiveloadheartratevariabilitywearablesensorsknowledgeretention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish that productive failure—letting beginners wrestle with a novel programming problem before teaching the underlying concept—can be imported from mathematics and physics into introductory programming education. In a small controlled study with 20 undergraduates in an introductory Python course learning lists, students who first attempted an open-ended weather-data task, and only then received instruction, did just as well on an immediate programming task as students who were taught lists first. Two weeks later, however, all seven productive-failure students still solved a similar heart-rate-tracking task, while only six of nine directly instructed students did. The paper further claims that the productive-failure group showed a larger decrease in cognitive load, inferred from heart-rate variability, between their pre-instruction task and their post-instruction task. If these findings hold, the practical payoff is a design pattern for first programming courses that costs nothing in immediate performance and appears to improve retention.

What carries the argument

The mechanism that carries the argument is the productive-failure sequence itself: a problem-solving phase before instruction, followed by a consolidation phase in which canonical solutions are compared with the students' own attempts. The concrete task is a sliding-window problem on a stream of sensor readings—students must keep the most recent seven weather readings or ten heart-rate readings—which deliberately admits many partial solutions built from concepts students already know. On the measurement side, a consumer wristband records photoplethysmography data, and a published heart-rate analysis pipeline converts it into RMSSD, the root mean square of successive differences between normal heartbeats, which falls as cognitive load rises. RMSSD changes relative to a five-minute baseline at the start of the session serve as the cognitive-load signal, and the two-week-later re-attempt of the heart-rate task provides the retention measure. Together these parts let the study attach a physiological story to a learning-outcome story.

What would settle it

Randomize which of the two sliding-window tasks comes before and which after instruction in each condition, so that the pre-instruction versus post-instruction contrast is not fixed to the productive-failure versus direct-instruction comparison. If the larger RMSSD drop follows the task-order pattern rather than the productive-failure condition, the cognitive-load evidence collapses; a larger sample with balanced groups would also check the stability of the 7/7 versus 6/9 retention gap.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that productive failure produces more durable knowledge than direct instruction when teaching Python lists to novices. Students in the productive-failure condition attempted an isomorphic sliding-window problem on weather-station data before any lesson, produced mostly non-canonical solutions using strings, tuples, and temporary variables, and then consolidated around the canonical list-based solution; students in the direct-instruction condition received the lesson first and then solved the same practice task. Immediate performance on a heart-rate version of the task was nearly identical across conditions, but the delayed post-test separated them: 7 of 7 productive-failure students succeeded two weeks later versus 6 of 9 direct-instruction students. The paper interprets this as evidence that the direct-instruction group's early success was in part unproductive success, while productive-failure students internalised the list concept more deeply. It also claims that the productive-failure group's larger drop in RMSSD-derived cognitive load from the pre-instruction to the post-instruction task supports the mechanism proposed by productive-failure theory: an initially heavy load, followed by instruction, yields easier subsequent performance.

Load-bearing premise

The physiological comparison assumes that the larger drop in cognitive load seen for productive-failure students—from their pre-instruction weather task to their post-instruction heart-rate task—comes from the teaching sequence itself, not from the fact that the first task was attempted before any instruction and the second after it.

Editorial extensions

If this is right

  • Adopting a problem-first sequence for introductory Python lists should not reduce what students can do immediately after the lesson; the retention benefit comes without an immediate performance cost.
  • Immediate post-tests can mask differences between instructional designs; a delayed, similar task is where productive failure's advantage shows up.
  • Embedding a wearable sensor in the programming activity itself is a workable way to collect physiological cognitive-load data without pulling students out of the learning task.
  • Initial failure is to be expected and is not a bad sign: only two of seven productive-failure students solved the weather task, yet all seven solved the equivalent heart-rate task after consolidation and two weeks later.
  • The same sliding-window task structure, built around live data, can serve as a reusable template for productive-failure activities on other introductory programming concepts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper leaves untested is whether the pattern generalises to other CS1 topics; the same weather-then-heart-rate task structure could be rebuilt around dictionaries, loops, or functions.
  • The physiological result is compatible with a simpler explanation the study cannot fully rule out: the larger RMSSD drop for productive-failure students may reflect task order rather than the teaching sequence, since their first task was pre-instruction and their second post-instruction. A crossover design would separate these explanations.
  • Because the sensor data simultaneously drives the programming exercise and the measurement, the paper suggests a broader classroom-research method: real-time physiological data can be an unobtrusive instrument for studying learning whenever the lesson itself can be built around live sensor streams.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper reports a controlled experiment comparing Productive Failure (PF) and Direct Instruction (DI) for teaching Python lists to 20 undergraduate students. In the PF condition, students attempted a practice task before instruction; in the DI condition, they received instruction first and then attempted the same task. Both groups then completed a heart-rate programming task, and returned two weeks later for a delayed post-test. The authors report that initial learning outcomes were similar, that PF students showed better retention on the delayed task (7/7 vs. 6/9), and that physiological measures indicated a larger decrease in cognitive load for PF students from the practice task to the programming task. The paper also presents qualitative data on student perceptions, with mixed preferences between PF and DI.

Significance. If the empirical claims were supported, the paper would be a useful contribution to computing education research on Productive Failure, which is currently under-explored for programming. The design has clear strengths: a concrete PF activity for a CS1 topic, an open-source Python library for wearable sensor data, random assignment, a delayed post-test, and qualitative analysis of student perceptions. However, the central quantitative claims are not supported by the evidence as presented. The cognitive-load comparison is confounded by task order, the retention result rests on a tiny sample with no inferential statistics, and four of twenty participants are missing from the analysis without explanation. These issues are load-bearing for the abstract's claims and cannot be repaired by re-analysis of the existing data.

major comments (4)
  1. [§4.2.1, Figure 2, §5.2] The physiological comparison is confounded by task order. In the initial session, the weather practice task is completed before the lists lesson in the PF condition and after the lesson in the DI condition, while the heart-rate task is after instruction for both groups. Section 5.2 compares the change in -RMSSD from the weather task to the heart-rate task and attributes the larger PF decrease to instruction; however, for PF students this contrast conflates 'before vs. after instruction' with 'weather vs. heart-rate task', whereas for DI students both tasks are post-instruction. The paper's own observation that 'on the heart-rate programming task, students in the PF and DI groups exhibited a similar change in their cognitive load overall' (Section 5.2) shows that the between-group difference is driven by the pre-instruction baseline in the PF group. This confound is load-bearing because the abstract's second main claim relies on this comparison, and the existing data cannot identify an instruction-induced cognitive-load advantage for PF.
  2. [Table 1, §5.1.3] The retention claim rests on 7/7 correct for PF versus 6/9 for DI on the delayed post-test, but no inferential test, effect size, or confidence interval is reported anywhere in Sections 5.1–5.2. With group sizes of 7 and 9, the difference is consistent with chance, and the paper's conclusion that 'students who followed the PF approach showed better knowledge retention' is not supported by the evidence presented. The same claim is repeated in the abstract and conclusion without statistical support.
  3. [§4.3, Table 1] The analysis shifts from N=20 participants to n=9 and n=7 (16 total) without any explanation for the four missing participants. Section 4.3 states that data from 16 students who returned completed questionnaires were analyzed, but it does not report the condition assignment or performance of the four non-included participants. If attrition is related to condition or performance, the comparison in Table 1 is potentially biased. The manuscript should report the missing participants' group assignments and any available data, and justify the exclusion.
  4. [§5.2, Figure 3] The physiological analysis reports only descriptive patterns of average -RMSSD changes and provides no statistical comparison between groups or across tasks. The central statement that PF students had a 'much larger reduction in load' is not accompanied by a test, and the distributions in Figure 3 appear overlapping and are not summarized numerically. Without inferential statistics, the claim of a between-group difference in cognitive-load change is unsupported.
minor comments (3)
  1. [Listings 1 and 2] The code listings are difficult to read because identifiers are broken with inserted spaces and underscores (e.g., 'N EW _DA Y_ AV AI LA BLE'); the code should be typeset cleanly.
  2. [Abstract and RQ1] The abstract claims 'better knowledge retention and performance on delayed but similar tasks,' but the study includes only one delayed task; the plural 'tasks' overstates the evidence.
  3. [§4.3.2] The cognitive-load pipeline is described at a high level, but the paper does not report how PPG artifacts were handled, how many segments were excluded, or whether any participants' physiological data were discarded; this information is needed to trust the RMSSD measurements.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the study is an empirical between-groups comparison, and the flagged cognitive-load issue is a design confound, not a self-referential or fitted-input claim.

full rationale

The paper's claims are empirical rather than derived: PF and DI are manipulated conditions, learning outcomes are measured on an initial task and a two-week delayed post-test, and cognitive load is inferred from RMSSD physiology. There is no fitted parameter later renamed as a prediction, no equation in which an output is identical to an input by construction, and no load-bearing uniqueness theorem or ansatz smuggled in via citation. The self-citations present, such as [31] on prior sensor-based work, are background and not used to justify the central result. The strongest concern identified by the reader is that the larger PF decrease in -RMSSD from the weather task to the heart-rate task is confounded with instruction timing: the PF weather task occurs before instruction while the DI weather task occurs after instruction (Section 4.2.1 and Figure 2). That is a genuine threat to the validity of the physiological comparison, but it is not circularity. The outcome is not defined in terms of the manipulation, and the manipulation is not chosen because of the outcome. Therefore, under the hard rules requiring a specific reduction or fitted-input equivalence, no circular step can be identified, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted parameters or new theoretical entities. Its load-bearing assumptions concern the validity of HRV as a cognitive load measure, the retention measure, and group equivalence after missing data.

assumptions (3)
  • domain assumption RMSSD derived from PPG data is a valid indicator of cognitive load.
    Section 4.3.2 uses this assumption to interpret heart-rate data as cognitive load; it is cited from prior literature but not validated in this specific wearable sensor setup.
  • domain assumption The delayed heart-rate task measures knowledge retention comparably across groups.
    Section 4.2.2 treats re-attempting the same task after two weeks as a retention measure without checking for differential practice, motivation, or task familiarity.
  • domain assumption Random assignment produced equivalent groups despite unequal analyzed sample sizes.
    Sections 4.1 and 4.2 describe random allocation, but the analysis uses 9 DI and 7 PF students with no baseline ability check or equivalence test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigating the Use of Productive Failure as a Design Paradigm for Learning Introductory Python Programming." pith.science (2026). https://pith.science/paper/T7LAXWQY

@misc{pith2026241111227,
  author       = {Pith},
  title        = {Pith review of: Investigating the Use of Productive Failure as a Design Paradigm for Learning Introductory Python Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T7LAXWQY}},
  note         = {Machine review of arXiv:2411.11227}
}
read the original abstract

Productive Failure (PF) is a learning approach where students initially tackle novel problems targeting concepts they have not yet learned, followed by a consolidation phase where these concepts are taught. Recent application in STEM disciplines suggests that PF can help learners develop more robust conceptual knowledge. However, empirical validation of PF for programming education remains under-explored. In this paper, we investigate the use of PF to teach Python lists to undergraduate students with limited prior programming experience. We designed a novel PF-based learning activity that incorporated the unobtrusive collection of real-time heart-rate data from consumer-grade wearable sensors. This sensor data was used both to make the learning activity engaging and to infer cognitive load. We evaluated our approach with 20 participants, half of whom were taught Python concepts using Direct Instruction (DI), and the other half with PF. We found that although there was no difference in initial learning outcomes between the groups, students who followed the PF approach showed better knowledge retention and performance on delayed but similar tasks. In addition, physiological measurements indicated that these students also exhibited a larger decrease in cognitive load during their tasks after instruction. Our findings suggest that PF-based approaches may lead to more robust learning, and that future work should investigate similar activities at scale across a range of concepts.

Figures

Figures reproduced from arXiv: 2411.11227 by the authors.

Figure 1
Figure 1. A handwritten solution collected during the pilot [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the study procedure for the evaluation [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Distribution of changes in students’ RMSSD during [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 22 canonical work pages

  1. [1]

    Olivier Augereau, Kai Kunze, and Koichi Kise. 2019. Experimental Supplements from Mobile Tools for Cognitive Introspection Towards Cognitive Augmentation. GetMobile: Mobile Comp. and Comm. 23, 2 (Nov. 2019), 22–24. https://doi.org/10. 1145/3372300.3372305

  2. [2]

    Berliner

    David C. Berliner. 2002. Comment: Educational Research:The Hardest Science of All. Educational Researcher 31, 8 (2002), 18–20. https://doi.org/10.3102/ 0013189X031008018

  3. [3]

    Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101. https://doi.org/10.1191/ 1478088706qp063oa

  4. [4]

    Ibrahim Dahlstrom-Hakki, Jodi Asbell-Clarke, and Elizabeth Rowe. 2019. Showing is knowing: The potential and challenges of using neurocognitive measures of implicit learning in the classroom. Mind, Brain, and Education 13, 1 (2019), 30–40. https://doi.org/10.1111/mbe.12177

  5. [5]

    Becker, Michelle Craig, Greg Wilson, and Piotr Ba- naszkiewicz

    Paul Denny, Brett A. Becker, Michelle Craig, Greg Wilson, and Piotr Ba- naszkiewicz. 2019. Research This! Questions that Computing Educators Most Want Computing Education Researchers to Answer. InProceedings of the 2019 ACM Conference on International Computing Education Research (Toronto ON, Canada) (ICER ’19). Association for Computing Machinery, New Yo...

  6. [6]

    Juan Carlos Farah, Arielle Moro, Kristoffer Bergram, Aditya Kumar Purohit, Denis Gillet, and Adrian Holzer. 2020. Bringing computational thinking to non-STEM undergraduates through an integrated notebook application. In 15th European Conference on Technology Enhanced Learning

  7. [7]

    Forlizzi, and Anind K

    Eija Haapalainen, SeungJun Kim, Jodi F. Forlizzi, and Anind K. Dey. 2010. Psycho- physiological measures for assessing cognitive load. In Proceedings of the 12th ACM International Conference on Ubiquitous Computing (Copenhagen, Denmark) (UbiComp ’10) . Association for Computing Machinery, New York, NY, USA, 301–310. https://doi.org/10.1145/1864349.1864395

  8. [8]

    Idit Ed Harel and Seymour Ed Papert. 1991. Constructionism. Ablex Publishing

Show all 38 references
  1. [9]

    Association for Computing Machinery (ACM) Joint Task Force on Comput- ing Curricula and IEEE Computer Society. 2013. Computer Science Curric- ula 2013: Curriculum Guidelines for Undergraduate Degree Programs in Com- puter Science . Association for Computing Machinery, New York...

  2. [10]

    Boonserm Kaewkamnerdpong. 2016. A Framework for Human Learning Ability Study Using Simultaneous EEG/fNIRS and Portable EEG for Learning and Teach- ing Development. In Smart Education and e-Learning 2016 , Vladimir L. Uskov, Robert J. Howlett, and Lakhmi C. Jain (Eds.). Springe...

  3. [11]

    Manu Kapur. 2008. Productive Failure. Cognition and Instruction 26, 3 (2008), 379–424. https://doi.org/10.1080/07370000802212669

  4. [12]

    Manu Kapur. 2016. Examining Productive Failure, Productive Success, Unpro- ductive Failure, and Unproductive Success in Learning. Educational Psychologist 51, 2 (2016), 289–299. https://doi.org/10.1080/00461520.2016.1155457

  5. [13]

    Manu Kapur and Katerine Bielaczyc. 2012. Designing for Productive Failure. Journal of the Learning Sciences 21, 1 (2012), 45–83. https://doi.org/10.1080/ 10508406.2011.591717

  6. [14]

    Manu Kapur and Nikol Rummel. 2012. Productive failure in learning from generation and invention activities. Instructional Science 40 (2012), 645–650. https://doi.org/10.1007/s11251-012-9235-4

  7. [15]

    Katharina Loibl, Ido Roll, and Nikol Rummel. 2017. Towards a theory of when and how problem solving followed by instruction supports learning. Educational psychology review 29 (2017), 693–715. https://doi.org/10.1007/s10648-016-9379-x

  8. [16]

    Lauren Margulieux, Paul Denny, Kathryn Cunningham, Michael Deutsch, and Benjamin R. Shapiro. 2021. When Wrong is Right: The Instructional Power of Multiple Conceptions. In Proceedings of the 17th ACM Conference on International Computing Education Research (Virtual Event, USA)...

  9. [17]

    Olney, and Sidney K

    Caitlin Mills, Igor Fridman, Walid Soussou, Disha Waghray, Andrew M. Olney, and Sidney K. D’Mello. 2017. Put your thinking cap on: detecting cognitive load using EEG during learning. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference (Vancouv...

  10. [18]

    Reinhard Pekrun and Markus Bühner. 2014. Self-report Measures of Academic Emotions. International Handbook of Emotions in Education (2014), 561–579

  11. [19]

    W James Popham. 2001. The truth about testing: An educator’s call to action . ASCD

  12. [20]

    Resnick and Daniel P

    Lauren B. Resnick and Daniel P. Resnick. 1992. Assessing the Thinking Cur- riculum: New Tools for Educational Reform. In Changing Assessments: Al- ternative Views of Aptitude, Achievement and Instruction , Bernard R. Gifford and Mary Catherine O’Connor (Eds.). Springer Netherl...

  13. [21]

    Philip Sands. 2019. Addressing cognitive load in the computer science classroom. ACM Inroads 10, 1 (Feb. 2019), 44–51. https://doi.org/10.1145/3210577

  14. [22]

    Zachary Monroe Savelson. 2020. Student Emotions in a Productive Failure Para- digm. Ph. D. Dissertation. Carleton University

  15. [23]

    Savelson and Kasia Muldner

    Zachary M. Savelson and Kasia Muldner. 2023. How do students feel and collabo- rate during programming activities in the productive failure paradigm?Computer Science Education 0, 0 (2023), 1–34. https://doi.org/10.1080/08993408.2023.2237365 arXiv:https://doi.org/10.1080/089934...

  16. [24]

    Lennart Schalk, Ralph Schumacher, Armin Barth, and Elsbeth Stern. 2018. When problem-solving followed by instruction is superior to the traditional tell-and- practice sequence. Journal of educational psychology 110, 4 (2018), 596

  17. [25]

    Fred Shaffer and Jay P Ginsberg. 2017. An overview of heart rate variability metrics and norms. Frontiers in public health 5 (2017), 258. https://doi.org/10. 3389/fpubh.2017.00258

  18. [26]

    Tanmay Sinha and Manu Kapur. 2021. When problem solving followed by instruction works: Evidence for productive failure.Review of Educational Research 91, 5 (2021), 761–798. https://doi.org/10.3102/00346543211019105

  19. [27]

    Sridhar, Samantha W.T

    Priyashri K. Sridhar, Samantha W.T. Chan, Yvonne Chua, Yow Wei Quin, and Suranga Nanayakkara. 2019. Going beyond performance scores: Understanding cognitive–affective states in Kindergarteners and application of framework in classrooms. International Journal of Child-Computer ...

  20. [28]

    Priyashri Kamlesh Sridhar and Suranga Nanayakkara. 2020. Progression of Cognitive-Affective States During Learning in Kindergarteners: Bringing To- gether Physiological, Observational and Performance Data. Education Sciences 10 (2020), 177. https://doi.org/10.3390/educsci10070177

  21. [29]

    Phil Steinhorst, Christof Duhme, Xiaoyi Jiang, and Jan Vahrenhold. 2024. Recog- nizing Patterns in Productive Failure. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (Portland, OR, USA) (SIGCSE 2024). Association for Computing Machinery, ...

  22. [30]

    Phil Steinhorst, Andrew Petersen, Bogdan Simion, and Jan Vahrenhold. 2023. Ex- ploring Barriers in Productive Failure. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1 (Chicago, IL, USA) (ICER ’23). Association for Computing Ma...

  23. [31]

    Hussel Suriyaarachchi, Paul Denny, Juan Pablo Forero Cortes, Chamod Weeras- inghe, and Suranga Nanayakkara. 2022. Primary School Students Program- ming with Real-Time Environmental Sensor Data. In Proceedings of the 24th Australasian Computing Education Conference (Virtual Eve...

  24. [32]

    John Sweller. 1994. Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction 4, 4 (1994), 295–312. https://doi.org/10.1016/ 0959-4752(94)90003-5

  25. [33]

    Aik Lim Tan, Robyn Gillies, and Azilawati Jamaludin. 2021. A Case Study: Using a Neuro-Physiological Measure to Monitor Students’ Interest and Learning during a Micro:Bit Activity. Education Sciences 11 (2021), 379. https://doi.org/10.3390/ educsci11080379

  26. [34]

    Julian F Thayer, Anita L Hansen, Evelyn Saus-Rose, and Bjorn Helge Johnsen. 2009. Heart rate variability, prefrontal neural function, and cognitive performance: the neurovisceral integration perspective on self-regulation, adaptation, and health. Annals of behavioral medicine ...

  27. [35]

    Sverrir Thorgeirsson, Tanmay Sinha, Felix Friedrich, and Zhendong Su. 2022. Does Deliberately Failing Improve Learning Introductory Computer Science?. In Educating for a New Future: Making Sense of Technology-Enhanced Learning Adoption: 17th European Conference on Technology E...

  28. [36]

    Sverrir Thorgeirsson and Zhendong Su. 2021. Algot: An Educational Program- ming Language with Human-Intuitive Visual Syntax. In 2021 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) (St Louis, MO, USA). IEEE, 1–5. https://doi.org/10.1109/VL/HCC51201.2021.9576166

  29. [37]

    Rebecca J. Trueman. 2014. Productive Failure in Stem Education. Journal of Educational Technology Systems 42, 3 (2014), 199–214. https://doi.org/10.2190/ET. 42.3.b

  30. [38]

    Paul van Gent, Haneen Farah, Nicole van Nes, and Bart van Arem. 2019. HeartPy: A novel heart rate algorithm for the analysis of noisy signals. Transportation Research Part F: Traffic Psychology and Behaviour 66 (2019), 368–378. https: //doi.org/10.1016/j.trf.2019.09.015

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.