Plan More, Debug Less: Applying Metacognitive Theory to AI-Assisted Programming Education

Adish Singla; Christopher Brooks; Heeryung Choi; Mengyan Wu; Tung Phung

REVIEW 3 major objections 4 minor 50 references

Plan More, Debug Less: Applying Metacognitive Theory to AI-Assisted Programming Education

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Plan more, debug less: planning hints track higher grades

desk verdict A useful, honest observational study of metacognitive hint types; the headline association is real but the paper itself scopes it appropriately. read the letter →

arxiv 2509.03171 v1 pith:TBT22AZK submitted 2025-09-03 cs.CY

Tung Phung , Heeryung Choi , Mengyan Wu , Adish Singla , Christopher Brooks This is my paper

classification cs.CY

keywords AI-generatedhintsmetacognitivescaffoldingprogrammingeducationhelp-seekingbehaviorself-regulatedlearningplanninglargelanguagemodelsinclassroomfieldstudy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether grounding AI-generated programming hints in metacognitive theory changes how students seek help and whether it helps them solve problems. In an introductory data-science programming course, 102 students could request three kinds of AI hints—planning, debugging, and optimization—up to five times per question. The paper's central finding is an association: students who requested planning hints solved questions at a higher rate, and this held across easier and harder questions and across higher- and lower-competency students. Students also engaged with planning hints more deeply, spending longer contemplating them, revisiting them more, and rating them higher. At the same time, students requested debugging hints more often overall and increasingly so on harder questions, while optimization hints were rarely used.

What carries the argument

The central mechanism is the AIMS hint system: AI-generated hints based on Metacognitive Scaffolds. The system maps the three metacognitive phases—planning, monitoring, and evaluation—onto three requestable hint types (planning, debugging, optimization), generates each hint from the student's current code plus a student-written reflection using GPT-4o, and caps use at five hints per question in a button-based JupyterLab interface. That design lets the paper observe which metacognitive phase students choose at each moment, generating the behavioral sequences and performance comparisons that carry the argument.

What would settle it

Run the same course with the same hint interface but randomly assign half the students to receive a planning-hint prompt before starting each hard question (or randomly assign which hint types are available per question). If students who are nudged or forced toward planning hints do not solve more problems than controls—holding hint count and timing fixed—then the observed association is selection, not support.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a deliberately metacognitive hint taxonomy makes a measurable difference in an AI-assisted programming classroom. The paper designed hints mapped to the metacognitive phases of planning, monitoring, and evaluation, generated by GPT-4o with guard-rails against revealing solutions, and let students choose among them under a quota of five hints per question. Requesting a planning hint was associated with significantly higher final solving rates than requesting no hint (p = 0.013), and the direction of the association was consistent across all difficulty and competency subgroups, even where not all comparisons reached significance. For higher-compe

Load-bearing premise

The central association could be an artifact of selection: students who ask for planning hints may be the same students who would have earned high grades anyway because of stronger study skills, and the study does not measure or control for that trait.

Editorial extensions

If this is right

If the planning-hint association reflects genuine support, AI tutors can improve outcomes by steering students toward planning hints early, before code is written.
Because planning-hint usage stayed flat while debugging usage rose on harder questions, systems could adaptively prompt for a plan when a student's first attempt fails on a hard problem.
The high engagement and ratings for planning hints suggest a demand for strategic guidance that many current feedback tools, focused on debugging, do not satisfy.
Optimization hints were rarely requested and sometimes misused by students who needed debugging; labeling alone is not enough to align help with need.
Higher-competency students were the ones who requested more hints and showed significant performance associations, suggesting that hint systems may currently benefit students who already have stronger self-regulated learning skills.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension the paper leaves implicit: randomly assign students (or questions) to a planning-hint encouragement condition to separate selection from treatment; if assigned planning hints do not raise solving rates, the association is mostly a marker of better self-regulated learners.
The hint-choice log itself could serve as a low-cost metacognitive assessment: students who never request planning on difficult questions may be signaling a gap that a tutor could address before giving debugging help.
Routing rules could make the system safer: only offer optimization hints when the student's current code passes all tests, and otherwise direct requests to debugging—this would test whether optimization-hint misuse explains the observed negative association on easier questions.
Because lower-competency students requested fewer planning hints and showed no significant benefit, a testable design is to vary Socratic versus directive hint phrasing by competency, predicting that weaker students need more explicit planning scaffolds.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The takeaway: this is a clean, honest field deployment of an AI hint system organized around metacognitive phases. The new bit is the three hint types (planning, debugging, optimization) tied to planning/monitoring/evaluation, plus the behavioral analysis showing students engage most with planning hints, rarely use optimization, and shift to debugging on harder tasks. The paper ships code for hint generation, which is real value.

What it does well: The design is grounded in theory rather than 'let's see what GPT says.' The behavioral findings (contemplation time, revisits, ratings) are genuinely descriptive and useful for building tutors. For RQ2, the authors carefully call their result an association and explicitly flag SRL confound in Section 5.3. They don't overclaim. The limitations section is honest about single course, high grades, no long-term learning.

Soft spots: The main performance claim rests on observational data. Students self-select hint types; the 'No' group mixes strong students who solve unaided and weak students who give up. So comparing planning vs 'No' is not a treatment effect. The subgroup cells in Figure 9 are tiny (n=8–14 for planning in some splits) and the chi-square tests aren't corrected for multiple comparisons. That said, the paper never says 'causal.' The consistent pattern across difficulty/competency is suggestive at best. The optimization-hint lower performance is plausibly just misuse, as they note.

The stress-test note worries about SRL as a confounder, and the paper already concedes this. So the central soft spot is real but disclosed. I'd say the contribution is more the hint system design and behavioral descriptives than the performance association.

Who it's for: people building AI tutoring systems for programming; researchers studying help-seeking. It deserves a serious referee and revision: ideally add within-student comparisons (e.g., same student's performance with vs without planning hints across questions) or at least control for prior ability and time-on-task. As is, it's a useful pilot with code.

Recommendation: engage with it, cite it for the hint taxonomy and behavioral findings, but don't lean on the performance claim.

Referee Report

3 major / 4 minor

Summary. The paper describes a classroom field study of an AI-assisted Jupyter notebook hint system (AIMS) that offers planning, debugging, and optimization hints grounded in metacognitive theory. With 102 students in an introductory data science programming course, the authors analyze logged hint requests and final problem-solving outcomes. They report that planning hints are the most engaged-with and highest-rated hint type; that students request debugging hints more often on harder questions; and that requesting planning hints is associated with higher solving rates across difficulty and competency subgroups. The paper explicitly acknowledges that causality cannot be established and lists several limitations, including the possibility that self-regulated learning (SRL) skill confounds the main association.

Significance. If the central association were robust, the paper would make a useful design contribution: it provides evidence that AI-generated hints organized around metacognitive phases, especially planning, are valued by students and may support performance. The strengths include a real classroom deployment, a publicly released implementation, rich interaction logs, and an unusually candid limitations section. However, the main empirical claim is purely observational. Students self-select which hints to request, SRL/ability is not measured or adjusted for, and the 'No hint' comparison group is heterogeneous. As the paper itself notes in Section 5.3, students' intrinsic SRL skills could drive both planning-hint requests and higher performance, making the headline association potentially spurious. The contribution is therefore more descriptive than the abstract's phrasing ('consistent association ... achieving higher grades') suggests, and the paper needs a stronger identification strategy or a substantially more cautious framing.

major comments (3)

[Section 5.2 / Figure 9] The central association is not identified as a treatment effect. Students self-select hint types, and the 'planning present' vs. 'No' comparison mixes the treated group with a heterogeneous 'No' group (students who solved without help and students who gave up). The paper concedes in Section 5.3 that 'students' high intrinsic SRL skills could be a confounding factor that caused both requesting of planning hints and higher final performance.' Because SRL skill is unmeasured, the observed association is compatible with planning hints being inert or even harmful. This is not a peripheral caveat: it is the main evidence for the paper's headline claim. The authors should either reframe the contribution as descriptive help-seeking behavior, or provide an identification strategy (e.g., within-student fixed-effects across questions, propensity/coarsened-exact matching on prior performance, or an
[Section 5.2 / Figure 9] The 'consistent association' across difficulty and competency rests on many unadjusted chi-square tests. Figure 9 contains nine panels, each with up to five pairwise comparisons against the 'No' group, without multiple-testing correction. Several cells are very small (e.g., P n=8 in Figure 9e; O n=1–2 in Figures 9h/9i), so the few significant effects could easily be false positives. Report all tests with effect sizes, confidence intervals, and a correction (e.g., Benjamini-Hochberg), or pre-specify a smaller set of confirmatory comparisons. As reported, the number of significant subgroups does not substantiate the 'consistency' claim.
[Section 3.5] Question difficulty and competency thresholds are constructed post hoc (four easiest/hardest questions; top/bottom thirds of Assignment 1 attempts) and are not validated. These stratifiers are also correlated with hint-seeking behavior, so the subgroup claims in RQ1c and RQ2b are sensitive to threshold choices. The authors should show robustness to alternative definitions (e.g., continuous difficulty/competency measures, or different quantile cutoffs), or explicitly treat subgroup analyses as exploratory.

minor comments (4)

[Abstract / Section 5.2] The abstract says 'achieving higher grades,' but the analysis uses final solving rates within individual questions. Please align the wording with the measured outcome.
[Sections 1 and 5] The language 'impact' (RQ1) and 'effects' (RQ2) implies causality, but the design is observational. Recommend consistent use of 'association' or 'relation' throughout to match the acknowledged limitations.
[Section 5.2] The sentence 'Other types (debugging, optimization) show no significant effects' is inconsistent with the subsequent report that optimization hints are linked to lower performance on easier questions (p=0.039) and among lower-competency students (p=0.003). Please clarify.
[Figure 9 caption] The caption says 'the five bars represent no hint requested, any type requested, and each type present in the sequence'; for readability, define 'type present' explicitly in the caption rather than relying on the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the planning-hint association is an observed relationship with explicit confounder caveats, not a construction-level reduction.

full rationale

The paper's central empirical claim—that requesting planning hints is associated with higher solving rates—is a measured observational association, not a quantity derived from the hint-generation procedure or from the authors' prior work. Question difficulty is estimated from past course offerings and student competency from Assignment 1 attempts; both are external to the focal outcome on later questions, and Assignment 1 is explicitly excluded from competency-based analyses. The only self-citations in the technical pipeline ([6], [24]) inform how debugging hints are generated, but the behavioral and performance comparisons are based on logged student interactions and would stand or fall independently of that provenance. Section 5.3 explicitly disclaims causality and names intrinsic SRL skills as a possible confounder, and Section 6 lists ceiling effects and the absence of long-term measures; these are validity limitations, not circular reductions. No equation defines a predicted outcome in terms of the data used to fit it, and no uniqueness or theoretical constraint is imported from the authors' prior publications to force the empirical result. Therefore the derivation chain is self-contained with respect to the paper's stated claims, and there is no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new particles, forces, or other invented entities. The hint types are design artifacts, not postulated entities. The main unmeasured construct is student self-regulated learning, which is a known factor, not newly invented.

free parameters (4)

Hint quota per question = 5
Hand-chosen ceiling to prevent over-reliance on AI; shapes the distribution of hint requests and the resulting behavior analyses. Section 3.3.
Competency group thresholds = top third (34) and bottom third (34) by Assignment 1 attempts
Hand-chosen cut into higher/lower competency groups; drives all competency subgroup analyses. Section 3.5.
Question difficulty group selection = four easiest and four hardest questions from prior offerings
Hand-chosen binning of questions into easy/hard; shapes difficulty comparisons in RQ1c and RQ2b. Section 3.5.
Contemplation time cutoff = 1 hour
Hint contemplation times capped at 1 hour to exclude breaks; authors note t=0.5 or t=2 also yield similar results. Section 4.1.

assumptions (5)

domain assumption Metacognitive phases (planning, monitoring, evaluation) map onto the hint types (planning, debugging, optimization).
The entire system design and interpretation assume this mapping is pedagogically valid; invoked in Section 3.2.
domain assumption GPT-4o-generated hints are of sufficient quality and do not reveal solutions.
The study assumes the AI-generated hints are correct and helpful enough to test metacognitive scaffolding; no systematic quality evaluation is reported. Sections 3.2 and 3.3.
domain assumption Question difficulty from prior course iterations transfers to the current cohort.
Difficulty is defined by past students' performance; this assumes stability across cohorts. Section 3.5.
domain assumption Assignment 1 attempts are a valid proxy for student competency.
Competency is measured by Assignment 1 attempts and used to split students into thirds; the proxy is assumed reliable even as Assignment 1 is excluded from competency analyses. Section 3.5.
standard math Student-question pairs are treated as independent observations in chi-square tests.
The chi-square tests assume independence, but multiple pairs come from the same student and may be correlated. Section 5.2.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Plan More, Debug Less: Applying Metacognitive Theory to AI-Assisted Programming Education." pith.science (2026). https://pith.science/paper/TBT22AZK

@misc{pith2026250903171,
  author       = {Pith},
  title        = {Pith review of: Plan More, Debug Less: Applying Metacognitive Theory to AI-Assisted Programming Education},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TBT22AZK}},
  note         = {Machine review of arXiv:2509.03171}
}

read the original abstract

The growing adoption of generative AI in education highlights the need to integrate established pedagogical principles into AI-assisted learning environments. This study investigates the potential of metacognitive theory to inform AI-assisted programming education through a hint system designed around the metacognitive phases of planning, monitoring, and evaluation. Upon request, the system can provide three types of AI-generated hints--planning, debugging, and optimization--to guide students at different stages of problem-solving. Through a study with 102 students in an introductory data science programming course, we find that students perceive and engage with planning hints most highly, whereas optimization hints are rarely requested. We observe a consistent association between requesting planning hints and achieving higher grades across question difficulty and student competency. However, when facing harder tasks, students seek additional debugging but not more planning support. These insights contribute to the growing field of AI-assisted programming education by providing empirical evidence on the importance of pedagogical principles in AI-assisted learning.

Figures

Figures reproduced from arXiv: 2509.03171 by the authors.

**Figure 2.** AIMS hint types with the descriptions provided to students. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Interaction between a student and our hint system for requesting a hint. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Examples of hint types: student reflections and received hints in the study. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Overview of students and hint requests. The orange dots and the right y-axis indicate the number [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Results for RQ1a: Student engagement and perception of hints. (a) demonstrates the amount of [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Results for RQ1b: Patterns of student hint usage. In all plots, y-axis represents the counts of [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 8.** Figure 8: Results for RQ1c: Behaviors by difficulty and competency. In each subfigure, the left shows Hint [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 9.** Figure 9: Results for RQ2: Performance by help-seeking behavior. The dashed line depicts the average [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 48 canonical work pages

[1]

Heffernan, Tanja Käser, Steven Moore, Anna N

Paul Denny, Sumit Gulwani, Neil T. Heffernan, Tanja Käser, Steven Moore, Anna N. Rafferty, and Adish Singla. Generative AI for Education (GAIED): Advances, Opportunities, and Challenges.CoRR, abs/2402.01580, 2024

arXiv 2024
[2]

Enhancing Programming Education with Chatgpt: A Case Study on Student Perceptions and Interactions in a Python Course

Boxuan Ma, Li Chen, and Shin’ichi Konomi. Enhancing Programming Education with Chatgpt: A Case Study on Student Perceptions and Interactions in a Python Course. InProceedings of the Artificial Intelligence in Education (AIED), 2024

work page 2024
[3]

Generative AI for Programming Education: Benchmarking Chatgpt, GPT-4, and Human Tutors

Tung Phung, Victor-Alexandru Pădurean, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Ma- jumdar, Adish Singla, and Gustavo Soares. Generative AI for Programming Education: Benchmarking Chatgpt, GPT-4, and Human Tutors. InProceedings of the Conference on International Computing Education Research (ICER) - Volume 2, 2023. 10

work page 2023
[4]

Mitchell, and Chris Piech

Sierra Wang, John C. Mitchell, and Chris Piech. A Large Scale RCT on Effective Error Messages in CS1. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE), 2024

work page 2024
[5]

You’re (Not) My Type-Can LLMs Generate Feed- back of Specific Types for Introductory Programming Tasks?Journal Of Computer Assisted Learning, 41, 2025

Dominic Lohr, Hieke Keuning, and Natalie Kiesler. You’re (Not) My Type-Can LLMs Generate Feed- back of Specific Types for Introductory Programming Tasks?Journal Of Computer Assisted Learning, 41, 2025

work page 2025
[6]

Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation

Tung Phung, Victor-Alexandru Padurean, Anjali Singh, Christopher Brooks, José Cambronero, Sumit Gulwani, Adish Singla, and Gustavo Soares. Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation. In Proceedings of the International Learning Analytics and Knowledge Co...

work page 2024
[7]

Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices

Ruiwei Xiao, Xinying Hou, and John Stamper. Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices. In Extended Abstracts Of The Conference On Human Factors In Computing Systems (CHI), 2024

work page 2024
[8]

Combining LLM-Generated and Test-based Feedback in a Mooc for Programming

Hagit Gabbay and Anat Cohen. Combining LLM-Generated and Test-based Feedback in a Mooc for Programming. In Proceedings of the Conference on Learning@ Scale (L@S), 2024

work page 2024

Show all 50 references

[9]

Navigating Compiler Errors with ai Assistance - a Study of GPT Hints in an Introductory Programming Course

Maciej Pankiewicz and Ryan Shaun Baker. Navigating Compiler Errors with ai Assistance - a Study of GPT Hints in an Introductory Programming Course. InProceedings of the Innovation and Technology in Computer Science Education (ITiCSE), 2024

2024
[10]

Metacognition and Self-regulation in Programming Education: Theories and Exemplars of Use

Dastyni Loksa, Lauren Margulieux, Brett A Becker, Michelle Craig, Paul Denny, Raymond Pettit, and James Prather. Metacognition and Self-regulation in Programming Education: Theories and Exemplars of Use. ACM Transactions on Computing Education (TOCE), 22, 2022

2022
[11]

The Effects of Worked-out Example and Metacognitive Scaffolding on Problem-solving Programming.Journal of Educational Computing Research, 61, 2023

Yoonhee Shin, Jaewon Jung, Joerg Zumbach, and Eunseon Yi. The Effects of Worked-out Example and Metacognitive Scaffolding on Problem-solving Programming.Journal of Educational Computing Research, 61, 2023

2023
[12]

S. E. Volet and C. Lund. Metacognitive Instruction in Introductory Computer Programming: A Better Explanatory Construct for Performance Than Traditional Factors.Journal of Educational Computing Research, 10, 1994

1994
[13]

Metacognition and Cognitive Monitoring: A New Area of Cognitive–developmental Inquiry

John H Flavell. Metacognition and Cognitive Monitoring: A New Area of Cognitive–developmental Inquiry. American Psychologist, 34, 1979

1979
[14]

Scaffolding and Metacognition.International Journal of Mathematical Education in Science and Technology, 37, 2006

Derek Holton and David Clarke. Scaffolding and Metacognition.International Journal of Mathematical Education in Science and Technology, 37, 2006

2006
[15]

Metacognitive Theories.Educational Psychology Review, 7, 1995

Gregory Schraw and David Moshman. Metacognitive Theories.Educational Psychology Review, 7, 1995

1995
[16]

Taxonomy of Novice Programming Error Patterns with Plan, Web, and Object Solutions.ACM Computing Surveys, 38, 2006

A Ebrahimi, D Kopec, and C Schweikert. Taxonomy of Novice Programming Error Patterns with Plan, Web, and Object Solutions.ACM Computing Surveys, 38, 2006

2006
[17]

Exploring Differences in Planning Between Students with and Without Prior Experience in Programming

Ryan Parsons, Qiang Hao, and Lu Ding. Exploring Differences in Planning Between Students with and Without Prior Experience in Programming. In American Society for Engineering Education Annual Conference & Exposition (ASEE), 2023

2023
[18]

Eunsung Park and Jongpil Cheon. Exploring Debugging Challenges and Strategies Using Structural Topic Model: A Comparative Analysis of High and Low-performing Students.Journal of Educational Computing Research, 62, 2025

2025
[19]

Learning with Style: Improving Student Code-style Through Better Automated Feedback

Liam Saliba, Elisa Shioji, Eduardo Oliveira, Shaanan Cohney, and Jianzhong Qi. Learning with Style: Improving Student Code-style Through Better Automated Feedback. InProceedings of the Technical Symposium on Computer Science Education (SIGCSE), 2024

2024
[20]

The Benefit of Reflection Prompts for Encouraging Learning with Hints in an Online Programming Course.The Internet and Higher Education., 58, 2023

Heeryung Choi, Jelena Jovanovic, Oleksandra Poquet, Christopher Brooks, Srećko Joksimović, and Joseph Jay Williams. The Benefit of Reflection Prompts for Encouraging Learning with Hints in an Online Programming Course.The Internet and Higher Education., 58, 2023

2023
[21]

Metacognitive Support Accelerates Computer Assisted Learning for Novice Programmers.Journal of Educational Technology & Society, 20, 2017

Siti Nurulain Mohd Rum and Maizatul Akmar Ismail. Metacognitive Support Accelerates Computer Assisted Learning for Novice Programmers.Journal of Educational Technology & Society, 20, 2017

2017
[22]

Magana, Anindya Roy, and M

Camilo Vieira, Alejandra J. Magana, Anindya Roy, and M. Falk. Student Explanations in the Context of Computational Science and Engineering Education.Cognition and Instruction, 37, 2019. 11

2019
[23]

Yilmaz and Ramazan Yılmaz

F. Yilmaz and Ramazan Yılmaz. Learning Analytics Intervention Improves Students’ Engagement in Online Learning. Technology, Knowledge and Learning, 27, 2021

2021
[24]

Zamfirescu-Pereira, Laryn Qi, Bjorn Hartmann, John DeNero, and Narges Norouzi

J.D. Zamfirescu-Pereira, Laryn Qi, Bjorn Hartmann, John DeNero, and Narges Norouzi. Conversational Programming with LLM-Powered Interactive Support in an Introductory Computer Science Course. NeurIPS’23 Workshop on Generative AI for Education (GAIED), 2023

2023
[25]

Dan Sun, Azzeddine Boudouaia, Junfeng Yang, and Jie Xu. Investigating Students’ Programming Be- haviors, Interaction Qualities and Perceptions Through Prompt-based Learning in Chatgpt.Humanities and Social Sciences Communications, 11, 2024

2024
[26]

College Student’s Academic Help-seeking Behavior: A Systematic Literature Review.Behavioral Sciences, 13, 2023

Ruihua Li, Norlizah Che Hassan, and Norzihani Saharuddin. College Student’s Academic Help-seeking Behavior: A Systematic Literature Review.Behavioral Sciences, 13, 2023

2023
[27]

Exploring Novice Programmers’ Hint Requests in an Intelligent Block-based Coding Environment

Joseph B Wiggins, Fahmid M Fahid, Andrew Emerson, Madeline Hinckle, Andy Smith, Kristy Elizabeth Boyer, Bradford Mott, Eric Wiebe, and James Lester. Exploring Novice Programmers’ Hint Requests in an Intelligent Block-based Coding Environment. In Proceedings of the Technical Sy...

2021
[28]

An Evaluation of the Impact of Automated Programming Hints on Performance and Learning

Samiha Marwan, Joseph Jay Williams, and Thomas Price. An Evaluation of the Impact of Automated Programming Hints on Performance and Learning. InProceedings of the Conference on International Computing Education Research (ICER), 2019

2019
[29]

Hint Generation Under Uncertainty: The Effect of Hint Quality on Help-seeking Behavior

Thomas W Price, Rui Zhi, and Tiffany Barnes. Hint Generation Under Uncertainty: The Effect of Hint Quality on Help-seeking Behavior. InArtificial Intelligence in Education (AIED), 2017

2017
[30]

Do Hints Enhance Learning in Programming Exercises? Exploring Students’ Problem-solving and Interactions

Giang Bui, Nicholas Susanto, Naaz Sibia, Angela Zavaleta Bernuy, Michael Liut, and Andrew Petersen. Do Hints Enhance Learning in Programming Exercises? Exploring Students’ Problem-solving and Interactions. In Proceedings of the Technical Symposium on Computer Science Education...

2024
[31]

Beware of Metacognitive Laziness: Effects of Generative Artificial Intelligence on Learning Motivation, Processes, and Performance.British Journal of Educational Technology, 2024

Yizhou Fan, Luzhen Tang, Huixiao Le, Kejie Shen, Shufang Tan, Yueying Zhao, Yuan Shen, Xinyu Li, and Dragan Gašević. Beware of Metacognitive Laziness: Effects of Generative Artificial Intelligence on Learning Motivation, Processes, and Performance.British Journal of Educationa...

2024
[32]

Codehelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes

Mark Liffiton, Brad E Sheese, Jaromir Savelka, and Paul Denny. Codehelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes. In Proceedings of the Koli Calling International Conference on Computing Education Research, 2023

2023
[33]

Gpt-4o System Card.Arxiv Preprint Arxiv:2410.21276, 2024

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, Aj Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, and Others. Gpt-4o System Card.Arxiv Preprint Arxiv:2410.21276, 2024

2024 arXiv
[34]

Chain-of-thought Prompting Elicits Reasoning in Large Language Models.Advances in Neural Information Processing Systems (NeurIPS), 35, 2022

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and Others. Chain-of-thought Prompting Elicits Reasoning in Large Language Models.Advances in Neural Information Processing Systems (NeurIPS), 35, 2022

2022
[35]

Alexander Mcfarlane Mood.Introduction to the Theory of Statistics.McGraw-hill, 1950

1950
[36]

Teoria Statistica Delle Classi E Calcolo Delle Probabilita.Pubblicazioni Del R Istituto Superiore Di Scienze Economiche E Commericiali Di Firenze, 8, 1936

Carlo Bonferroni. Teoria Statistica Delle Classi E Calcolo Delle Probabilita.Pubblicazioni Del R Istituto Superiore Di Scienze Economiche E Commericiali Di Firenze, 8, 1936

1936
[37]

Use of Ranks in One-criterion Variance Analysis.Journal of the American Statistical Association, 47, 1952

William H Kruskal and W Allen Wallis. Use of Ranks in One-criterion Variance Analysis.Journal of the American Statistical Association, 47, 1952

1952
[38]

Karl Pearson. X. On the Criterion That a Given System of Deviations From the Probable in the Case of a Correlated System of Variables Is Such That It Can Be Reasonably Supposed to Have Arisen From Random Sampling.The London, Edinburgh, and Dublin Philosophical Magazine and Jou...

1900
[39]

Teachers’ Practices of Proactive and Reac- tive Classroom Management Strategies and the Relationship to Their Self-efficacy.Language Teaching Research, 28, 2024

Nada Jaber Alasmari and Abeer Sultan Ahmed Althaqafi. Teachers’ Practices of Proactive and Reac- tive Classroom Management Strategies and the Relationship to Their Self-efficacy.Language Teaching Research, 28, 2024. 12

2024
[40]

Do Students Learn More From Erroneous Code? Exploring Student Performance and Satisfaction in an Error-free Versus an Error-full Sas® Programming Envi- ronment

Heather J Hoffman and Angelo F Elmi. Do Students Learn More From Erroneous Code? Exploring Student Performance and Satisfaction in an Error-free Versus an Error-full Sas® Programming Envi- ronment. Journal of Statistics and Data Science Education, 29, 2021

2021
[41]

Proactive and Reactive Engagement of Artificial Intelligence Methods for Education: a Review.Frontiers Artificial Intelligence, 6, 2023

Sruti Mallik and Ahana Gangopadhyay. Proactive and Reactive Engagement of Artificial Intelligence Methods for Education: a Review.Frontiers Artificial Intelligence, 6, 2023

2023
[42]

Learning for Mastery

Benjamin S Bloom. Learning for Mastery. Instruction and Curriculum. Regional Education Laboratory for the Carolinas and Virginia, Topical Papers and Reprints, Number 1.Evaluation Comment, 1, 1968

1968
[43]

Fostering Metacognition to Support Student Learning and Performance.CBE—Life Sciences Education, 20, 2021

Julie Dangremond Stanton, Amanda J Sebesta, and John Dunlosky. Fostering Metacognition to Support Student Learning and Performance.CBE—Life Sciences Education, 20, 2021

2021
[44]

The Role of Planning in Complex Problem Solving.Computers & Education, 128, 2019

Beate Eichmann, Frank Goldhammer, Samuel Greiff, Liene Pucite, and Johannes Naumann. The Role of Planning in Complex Problem Solving.Computers & Education, 128, 2019

2019
[45]

Effectiveness of Reflection on Programming Problem Solving Self-assessments

Mohammed Alzaid and I-han Hsiao. Effectiveness of Reflection on Programming Problem Solving Self-assessments. In Frontiers in Education Conference (FIE), 2018

2018
[46]

Evaluation of Animalwatch: An Intelligent Tutoring System for Arithmetic and Fractions.Journal of Interactive Online Learning, 9, 2010

Carole R Beal, Ivon M Arroyo, Paul R Cohen, and Beverly P Woolf. Evaluation of Animalwatch: An Intelligent Tutoring System for Arithmetic and Fractions.Journal of Interactive Online Learning, 9, 2010

2010
[47]

On-line Tutoring for Math Achievement Testing: A Controlled Evaluation.Journal of Interactive Online Learning, 6, 2007

Carole R Beal, Rena Walles, Ivon Arroyo, and Beverly P Woolf. On-line Tutoring for Math Achievement Testing: A Controlled Evaluation.Journal of Interactive Online Learning, 6, 2007

2007
[48]

Department of Computer Science, Stanford University, 1982

Paul R Cohen and Ea Feigenbaum.Planning and Problem Solving. Department of Computer Science, Stanford University, 1982

1982
[49]

Problem Solving: Increased Planning with Practice.Cogni- tive Systems Research, 4, 2003

Glenn Gunzelmann and John R Anderson. Problem Solving: Increased Planning with Practice.Cogni- tive Systems Research, 4, 2003

2003
[50]

The Effects of Dynamic and Static Feedback Under Tasks with Different Difficulty Levels in Digital Game-based Learning.The Internet and Higher Education, 60, 2024

Peipei Mao, Zhihui Cai, Zhikeng Wang, Xin Hao, Xitao Fan, and Xiaojun Sun. The Effects of Dynamic and Static Feedback Under Tasks with Different Difficulty Levels in Digital Game-based Learning.The Internet and Higher Education, 60, 2024. 13

2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Heffernan, Tanja Käser, Steven Moore, Anna N

Paul Denny, Sumit Gulwani, Neil T. Heffernan, Tanja Käser, Steven Moore, Anna N. Rafferty, and Adish Singla. Generative AI for Education (GAIED): Advances, Opportunities, and Challenges.CoRR, abs/2402.01580, 2024

arXiv 2024

[2] [2]

Enhancing Programming Education with Chatgpt: A Case Study on Student Perceptions and Interactions in a Python Course

Boxuan Ma, Li Chen, and Shin’ichi Konomi. Enhancing Programming Education with Chatgpt: A Case Study on Student Perceptions and Interactions in a Python Course. InProceedings of the Artificial Intelligence in Education (AIED), 2024

work page 2024

[3] [3]

Generative AI for Programming Education: Benchmarking Chatgpt, GPT-4, and Human Tutors

Tung Phung, Victor-Alexandru Pădurean, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Ma- jumdar, Adish Singla, and Gustavo Soares. Generative AI for Programming Education: Benchmarking Chatgpt, GPT-4, and Human Tutors. InProceedings of the Conference on International Computing Education Research (ICER) - Volume 2, 2023. 10

work page 2023

[4] [4]

Mitchell, and Chris Piech

Sierra Wang, John C. Mitchell, and Chris Piech. A Large Scale RCT on Effective Error Messages in CS1. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE), 2024

work page 2024

[5] [5]

You’re (Not) My Type-Can LLMs Generate Feed- back of Specific Types for Introductory Programming Tasks?Journal Of Computer Assisted Learning, 41, 2025

Dominic Lohr, Hieke Keuning, and Natalie Kiesler. You’re (Not) My Type-Can LLMs Generate Feed- back of Specific Types for Introductory Programming Tasks?Journal Of Computer Assisted Learning, 41, 2025

work page 2025

[6] [6]

Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation

Tung Phung, Victor-Alexandru Padurean, Anjali Singh, Christopher Brooks, José Cambronero, Sumit Gulwani, Adish Singla, and Gustavo Soares. Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation. In Proceedings of the International Learning Analytics and Knowledge Co...

work page 2024

[7] [7]

Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices

Ruiwei Xiao, Xinying Hou, and John Stamper. Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices. In Extended Abstracts Of The Conference On Human Factors In Computing Systems (CHI), 2024

work page 2024

[8] [8]

Combining LLM-Generated and Test-based Feedback in a Mooc for Programming

Hagit Gabbay and Anat Cohen. Combining LLM-Generated and Test-based Feedback in a Mooc for Programming. In Proceedings of the Conference on Learning@ Scale (L@S), 2024

work page 2024