Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Hints Help Finding and Fixing Bugs Differently in Python and Text-based Program Representations

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Plain text beats Python code for bug finding in a 753-person debugging study.

desk verdict Well-run study with an interesting interaction result, but the text-vs-Python comparison rests on an unvalidated assumption of matched bug salience. read the letter →

arxiv 2412.12471 v1 pith:2YRNBTOQ submitted 2024-12-17 cs.SE

Ruchit Rawal , Victor-Alexandru Pădurean , Sven Apel , Adish Singla , Mariya Toneva This is my paper · ORCID
classification cs.SE
keywords programcomprehensiondebuggingtext-basedrepresentationprogramminghintscrowdsourcedexperimentPythonalgorithmicunderstandingpersonalizedtools
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a large-scale crowdsourced experiment testing whether the way a buggy program is presented, as Python source code or as a natural-language description, and the type of hint offered change a user's ability to find and fix bugs. The central claim is that representation matters: for participants who demonstrated a clear understanding of the algorithmic task, the text-based representation led to significantly better debugging accuracy than the Python representation (p = 0.004, Cohen's d = 0.52, mean difference 0.129). The paper further argues that hints are not uniformly helpful: they improve accuracy for Python programs across both clear and confused user groups, while for text descriptions they mainly reduce response time rather than error. These findings matter because they suggest programming tools could personalize both the modality of code presentation and the kind of hint, depending on the user's level of algorithmic understanding.

What carries the argument

The load-bearing machinery is the study's four-question protocol: Q1 tests understanding of the task and partitions users into 'clear' and 'confused' groups; Q2, Q3, and Q4 measure respectively the ability to predict the buggy program's output, localize the bug, and select the correct fix, and their average forms the accuracy metric. The other central object is the handcrafted text-based representation, a natural-language paraphrase of the Python program that avoids programming terminology and is claimed to be faithful to the original code. This design lets the authors attribute accuracy differences to representation and hint condition rather than to task knowledge.

What would settle it

Run the same debugging task with text descriptions that conceal the bug, for example describing the odd-length check as 'the algorithm treats strings of certain lengths specially' rather than 'returns False if the length is odd'; if accuracy on text drops to Python levels, the advantage is driven by the bug being stated explicitly, not by the text format.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that program representation and hint type interact with the user's understanding of the task to determine debugging success. For users with a clear understanding who receive no hint, viewing the program as natural text yields higher accuracy on questions about the bug's output, location, and fix than viewing the same program as Python code (mean difference 0.129, p = 0.004, Cohen's d = 0.52). Hints erase this gap: giving any hint to clear users working in Python raises their accuracy to the level of text users without hints, and gives confused Python users a similar boost, making them statistically indistinguishable from clear users without hints. Among hint types, detailed fixes are generally the most accurate, while conceptual hints specifically help confused users with Python; test cases do not improve accuracy but reduce response time for clear users on text. The authors interpret the text advantage through top-down versus bottom-up comprehension, proposing that Python invites schema-driven reading that can miss bugs, whereas text forces sequential attention.

Load-bearing premise

The study assumes the handcrafted text-based descriptions are no easier or harder to debug than the Python programs apart from the format itself, so any accuracy difference is due to representation rather than to bugs being more conspicuous when spelled out in words.

Editorial extensions

If this is right

  • Debugging tools could adapt program representation to the user: showing natural-language descriptions to users who grasp the task, and Python with hints to users who do not.
  • Hints can serve as a leveler: a conceptual hint for a confused user on Python code closes the accuracy gap with a clear user who receives no hint.
  • Detailed fixes are the safest default hint, improving accuracy for clear users on Python and confused users on text; test cases should be used primarily to speed up accurate responses on text representations.
  • The accuracy gap between representations is not fixed: providing hints for Python removes the statistical difference between Python and text for clear users.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the text-based advantage stems from the bug being spelled out in words rather than from natural language per se, then rewriting Python identifiers or adding comments that name the faulty condition should reproduce the effect—a testable prediction beyond the paper.
  • The paper's preliminary analysis by self-reported experience points toward text helping inexperienced programmers; an inference is that experienced developers might show the opposite or no advantage, a population the study did not have power to settle.
  • The finding that test cases reduce response time on text but not accuracy suggests that mixed-modality hints (code plus natural language) might act as comprehension scaffolds; this could be tested by giving cross-modal hints, such as a conceptual hint in text while the program is in Python.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a large-scale crowdsourced study (753 participants) investigating how program representation (Python vs. a handcrafted text-based description) and three hint types (test cases, conceptual, detailed fix) affect users' ability to find and fix bugs in short algorithmic programs. Participants were assigned to one of eight representation-hint conditions, answered four multiple-choice questions per task (understanding, bug output, bug location, bug fix), and were grouped post hoc into 'clear' and 'confused' understanding based on Q1 accuracy. The main claims are: (RQ1) among clear-understanding users, text-based representations yield significantly higher Q2–Q4 accuracy than Python in the no-hint condition; (RQ2) hints significantly improve accuracy for Python representations for both clear and confused users, while for text representations hints mainly reduce response time; and (RQ3) detailed fixes are generally the most effective hint type, with conceptual hints especially helpful for confused Python users. The paper formulates four hypotheses (H1–H4) to interpret these results and makes raw data and analysis scripts available.

Significance. If the central claims hold, the paper makes a useful contribution to the emerging literature on non-classical programming modalities and personalized debugging support: it is, to my knowledge, the first large-scale study to cross program representation with hint type and user understanding. The authors deserve credit for a carefully specified experimental flow, a relatively large participant pool, an open data repository, and explicit falsifiable hypotheses (H1–H4) that go beyond the immediate findings. However, the significance is currently tempered by a potentially load-bearing stimulus confound in the text-based representations and by statistical reporting that does not account for the number of comparisons made. The direction of the main RQ1 effect is surprising and interesting, but the evidence as presented does not yet rule out the alternative explanation that the text descriptions make bugs more conspicuous.

major comments (4)
  1. [IV, RQ1] The central RQ1 claim (text > Python for clear users, p=0.004, d=0.52) rests on the assumption that the handcrafted text representations and the Python programs differ only in format, not in bug salience or difficulty. The example in Fig. 1 raises a concrete concern: the text explicitly describes the faulty odd-length check as a standalone step ('it checks whether the calculated length is odd. If so, it returns False'), whereas the Python version requires the reader to parse 'if length % 2 == 1: return False' within control flow. No validation is reported that the 25 text-Python pairs are matched on bug conspicuity, ambiguity, or reading difficulty. If this explicitness pattern generalizes, the RQ1 advantage may reflect the text stating the bug more directly rather than a general benefit of natural-language format, and the RQ2 Python-hint comparisons inherit the same confound because their baseline is inflated. Please provide an item-level analysis showing the effect is consistent across pairs, a rating study for bug salience, or an alternative argument that the stimulus pairs are matched on this dimension.
  2. [III-H, IV] The paper performs a large number of pairwise Wilcoxon tests across representations, hint types, user groups, and time metrics, but reports raw p-values without any multiple-comparison correction. For instance, the RQ2 result that hints help clear Python users is reported at p=0.049, and the confused Python result at p=0.043; these would not survive even a simple Bonferroni correction across the several tests in Figure 4. The RQ3 tests in Figure 5 involve 12 pairwise comparisons against no-hint (2 representations × 2 groups × 3 hint types), yet the reported significant results (p=0.006, p=0.047, p=0.009) are not adjusted. Please report corrected p-values or an explicit per-comparison error rate with justification, and clarify which findings survive correction.
  3. [III-G, IV] The 'clear' vs. 'confused' grouping is derived from Q1 accuracy in the same experiment and is therefore not a randomized factor. The authors acknowledge this, but the subsequent analysis treats it as a quasi-independent variable. Because participants who switch groups across their two tasks are handled by randomly discarding one response, the effective sample size is reduced and the discarded response may not be missing at random (e.g., the second task may be harder or presented after fatigue). To support the load-bearing claims about clear vs. confused users, please add sensitivity analyses: e.g., use both responses with a mixed model, or re-run the main comparisons using an alternative grouping (such as self-reported programming experience) and show that the RQ1/RQ2 conclusions are robust.
  4. [IV, RQ2] The paper claims that hints 'bridged' the accuracy gap between Python and text representations and between confused and clear users, but the evidence offered is the absence of a significant difference in the hinted conditions (p > 0.05). A non-significant difference is not evidence of equivalence, especially with the sample sizes used here. To support the bridging interpretation, please report a formal test of the interaction between representation and hint presence (or between understanding group and hint presence), or an equivalence test with a pre-specified margin.
minor comments (5)
  1. [Figures] Several figures contain typos: 'Avgerage Accuracy' in Figures 4 and 5, 'T est vs. Python' in Figure 2, and 'inexpierenced' in Section VI. Please correct these throughout.
  2. [III-C.2] The text claims the text-based representations avoid 'programming concepts such as variables, loops', but the example in Figure 1 uses words like 'algorithm', 'returns', and 'terminates', which are themselves programming-related. Please clarify what counts as a programming concept in this design, or acknowledge that the text representation is not free of programming vocabulary.
  3. [III-G] The figures report means and standard errors but not the number of participants per condition. Because the sample is split into eight conditions and then further split by the post-hoc clear/confused grouping, some cells may be small. Please include n per bar or report the cell sizes in a table.
  4. [III-B] The sentence 'The range of average accuracy ranges from 0 to 1' is redundant; please rephrase. Also, the theoretical chance accuracy of 0.305 is correctly derived from 0.25, 0.333, and 0.333, but the derivation could be stated explicitly.
  5. [IV, RQ1] When reporting the time analysis for Q2–Q4, the paper says text representations take 'significantly longer time' with p=0.047, but then notes this is likely due to reading time and finds no Q2-only difference. This nuance is appropriate; consider moving the Q2-only result into the main text rather than only mentioning it in the discussion of Figure 6.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical participant measurements, not derivations from their own inputs.

full rationale

The paper is a crowd-sourced experiment rather than a formal derivation. Its central claims—that text representations improve bug-finding accuracy for clear users, that hints improve accuracy for Python representations, and that detailed fixes are the most helpful hint type—are statistical comparisons of measured Q2–Q4 accuracy across randomly assigned conditions. There is no fitted parameter later renamed as a prediction, no outcome variable defined in terms of itself, and no uniqueness theorem or prior author-derived result invoked to force the conclusions. The Q1-based grouping into clear versus confused users is a separate comprehension check, not the outcome metric, so the grouping is not equivalent to the bug-finding/fixing scores. The self-citations in the related work and stimulus design (e.g., [12], [13], [34], [40]) are used as sources of publicly available buggy programs and as background literature; they are not load-bearing for the statistical results, which come from the 753 participants and are released with raw data and scripts. The handcrafted text representations are a potential construct-validity threat—for instance, the Figure 1 text spells out the faulty odd-length check explicitly, which could make the bug more conspicuous than in the Python version—but this is a possible confound about stimulus comparability, not circularity in the paper's argument. Under the stated rubric, no circular step can be exhibited with a specific reduction of a claimed result to its inputs.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No fitted parameters or invented entities are present; the study reports measured accuracy and response times. Its design choices, such as the set of algorithmic tasks, the three hint types, the Q1-based grouping rule, and the exclusion thresholds, are assumptions about construct validity and generalizability rather than fitted quantities.

assumptions (5)
  • domain assumption The average accuracy on Q2, Q3, and Q4 is a valid measure of a user's ability to find and fix bugs.
    The paper defines utility as accuracy on multiple-choice questions, but does not validate that the questions map to real debugging behavior (Section III-B, III-D).
  • domain assumption The handcrafted text-based representations are faithful, unbiased translations of the Python programs.
    Authors crafted descriptions with no programming terms; no inter-rater reliability or difficulty matching is reported (Section III-A.2).
  • domain assumption The handcrafted hints are equally applicable across representations and do not leak information differently.
    Hints are stated to be independent of representation, but no checks are reported for consistency across the two formats (Section III-A.3).
  • standard math Wilcoxon rank-sum test assumptions (independence, same shape) are met after the data processing.
    The paper checks shape with KS tests and uses one datapoint per participant; this is a reasonable but unverified application (Section III-H).
  • domain assumption Crowdsourced participants are representative of the target end-user population.
    Authors acknowledge limited programming experience; generalizability to other populations is not established (Section V-E).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hints Help Finding and Fixing Bugs Differently in Python and Text-based Program Representations." pith.science (2026). https://pith.science/paper/2YRNBTOQ

@misc{pith2026241212471,
  author       = {Pith},
  title        = {Pith review of: Hints Help Finding and Fixing Bugs Differently in Python and Text-based Program Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2YRNBTOQ}},
  note         = {Machine review of arXiv:2412.12471}
}
read the original abstract

With the recent advances in AI programming assistants such as GitHub Copilot, programming is not limited to classical programming languages anymore--programming tasks can also be expressed and solved by end-users in natural text. Despite the availability of this new programming modality, users still face difficulties with algorithmic understanding and program debugging. One promising approach to support end-users is to provide hints to help them find and fix bugs while forming and improving their programming capabilities. While it is plausible that hints can help, it is unclear which type of hint is helpful and how this depends on program representations (classic source code or a textual representation) and the user's capability of understanding the algorithmic task. To understand the role of hints in this space, we conduct a large-scale crowd-sourced study involving 753 participants investigating the effect of three types of hints (test cases, conceptual, and detailed), across two program representations (Python and text-based), and two groups of users (with clear understanding or confusion about the algorithmic task). We find that the program representation (Python vs. text) has a significant influence on the users' accuracy at finding and fixing bugs. Surprisingly, users are more accurate at finding and fixing bugs when they see the program in natural text. Hints are generally helpful in improving accuracy, but different hints help differently depending on the program representation and the user's understanding of the algorithmic task. These findings have implications for designing next-generation programming tools that provide personalized support to users, for example, by adapting the programming modality and providing hints with respect to the user's skill level and understanding.

Figures

Figures reproduced from arXiv: 2412.12471 by the authors.

Figure 1
Figure 1. An illustrative example from the study showcasing an algorithmic task. After showing a task, the user is asked to [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Visual summary of main findings to our Research Questions. Circular nodes represent main factors of variation (Program [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Accuracy of participants when presented with text [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Accuracy of participants when presented with hints [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Accuracy of participants when presented with different hints or no hint, across different program representations and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Time taken by participants to correctly answer Q2, Q3, and Q4 when presented with different hint types or no hint, [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A survey of 80+ Deep Research systems that proposes a four-layer taxonomy (foundation models, tool use, planning, synthesis) and compares commercial and open-source implementations.

Reference graph

Works this paper leans on

47 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    GitHub Copilot,

    GitHub, “GitHub Copilot,” https://github.com/features/copilot, 2021

    work page 2021

  2. [2]

    ChatGPT,

    OpenAI, “ChatGPT,” https://openai.com/blog/chatgpt, 2023

    work page 2023

  3. [3]

    Game-changing top 60 no-code low-code & citizen development statistics,

    Quixy Editorial Team, “Game-changing top 60 no-code low-code & citizen development statistics,” Feb 2024. [Online]. Available: https:// quixy.com/blog/no-code-low-code-citizen-development-statistics-facts/

    work page 2024

  4. [4]

    Debugging: Finding, Fixing and Flailing, a Multi-institutional Study of Novice Debuggers,

    S. Fitzgerald, G. Lewandowski, R. McCauley, L. Murphy, B. Simon, L. Thomas, and C. Zander, “Debugging: Finding, Fixing and Flailing, a Multi-institutional Study of Novice Debuggers,” Computer Science Education, vol. 18, 2008

    work page 2008

  5. [5]

    Debugging: A Review of the Literature from an Educational Perspective,

    R. McCauley, S. Fitzgerald, G. Lewandowski, L. Murphy, B. Simon, L. Thomas, and C. Zander, “Debugging: A Review of the Literature from an Educational Perspective,” Computer Science Education , 2008

    work page 2008

  6. [6]

    Towards a Framework for Teaching Debugging,

    C. Li, E. Chan, P. Denny, A. Luxton-Reilly, and E. D. Tempero, “Towards a Framework for Teaching Debugging,” in Proceedings of the Australasian Computing Education Conference (ACE) , 2019

    work page 2019

  7. [7]

    Novice Reflections on Debugging,

    J. Whalley, A. Settle, and A. Luxton-Reilly, “Novice Reflections on Debugging,” in Proceedings of the Technical Symposium on Computer Science Education (SIGCSE) , 2021

    work page 2021

  8. [8]

    The Robots Are Here: Navigating the Generative AI Revolution in Computing Education,

    J. Prather et al., “The Robots Are Here: Navigating the Generative AI Revolution in Computing Education,” in Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education (ITiCSE-WGR), 2023

    work page 2023

Show all 47 references
  1. [9]

    Generative AI for Education (GAIED): Advances, Opportunities, and Challenges,

    P. Denny, S. Gulwani, N. T. Heffernan, T. K ¨aser, S. Moore, A. N. Raf- ferty, and A. Singla, “Generative AI for Education (GAIED): Advances, Opportunities, and Challenges,” CoRR, vol. abs/2402.01580, 2024

    2024 arXiv

  2. [10]

    Using Large Language Models to Enhance Program- ming Error Messages,

    J. Leinonen, A. Hellas, S. Sarsa, B. N. Reeves, P. Denny, J. Prather, and B. A. Becker, “Using Large Language Models to Enhance Program- ming Error Messages,” in Proceedings of the Technical Symposium on Computer Science Education (SIGCSE) , 2023

    2023

  3. [11]

    A Large Scale RCT on Effective Error Messages in CS1,

    S. Wang, J. C. Mitchell, and C. Piech, “A Large Scale RCT on Effective Error Messages in CS1,” in Proceedings of the Technical Symposium on Computer Science Education (SIGCSE) . ACM, 2024, pp. 1395–1401

    2024

  4. [12]

    Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors,

    T. Phung, V . Padurean, J. Cambronero, S. Gulwani, T. Kohn, R. Ma- jumdar, A. Singla, and G. Soares, “Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors,” in Proceedings of the Conference on International Computing Education Research (ICER) ...

    2023

  5. [13]

    Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT- 3.5 Student Model for Hint Validation,

    T. Phung, V . Padurean, A. Singh, C. Brooks, J. Cambronero, S. Gulwani, A. Singla, and G. Soares, “Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT- 3.5 Student Model for Hint Validation,” in Proceedings of the Learnin...

    2024

  6. [14]

    Using Large Language Models for Student- Code Guided Test Case Generation in Computer Science Education,

    N. A. Kumar and A. S. Lan, “Using Large Language Models for Student- Code Guided Test Case Generation in Computer Science Education,” AI4ED Workshop at AAAI , 2024

    2024

  7. [15]

    Generating Feedback-Ladders for Logical Errors in Programming using Large Language Models,

    H. Heicka and A. S. Lan, “Generating Feedback-Ladders for Logical Errors in Programming using Large Language Models,” in Proceedings of International Conference on Educational Data Mining (EDM) , 2024

    2024

  8. [16]

    Computer code com- prehension shares neural resources with formal logical inference in the fronto-parietal network,

    Y .-F. Liu, J. Kim, C. Wilson, and M. Bedny, “Computer code com- prehension shares neural resources with formal logical inference in the fronto-parietal network,” Elife, vol. 9, p. e59340, 2020

    2020

  9. [17]

    Contribution of the language network to the comprehension of python programming code,

    Y .-F. Liu, C. Wilson, and M. Bedny, “Contribution of the language network to the comprehension of python programming code,” Brain and Language, vol. 251, p. 105392, 2024

    2024

  10. [18]

    Comprehension of computer code relies primarily on domain-general executive brain regions,

    A. A. Ivanova, S. Srikant, Y . Sueoka, H. H. Kean, R. Dhamala, U.-M. O’reilly, M. U. Bers, and E. Fedorenko, “Comprehension of computer code relies primarily on domain-general executive brain regions,” elife, vol. 9, p. e58906, 2020

    2020

  11. [19]

    Decoding the representation of code in the brain: An fmri study of code review and expertise,

    B. Floyd, T. Santander, and W. Weimer, “Decoding the representation of code in the brain: An fmri study of code review and expertise,” in 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 2017, pp. 175–186

    2017

  12. [20]

    Eye movements in code reading: Relaxing the linear order,

    T. Busjahn, R. Bednarik, A. Begel, M. Crosby, J. H. Paterson, C. Schulte, B. Sharif, and S. Tamm, “Eye movements in code reading: Relaxing the linear order,” in 2015 ieee 23rd international conference on program comprehension. IEEE, 2015, pp. 255–265

    2015

  13. [21]

    Connecting the dots: rethinking the relationship between code and prose writing with func- tional connectivity,

    Z. Karas, A. Jahn, W. Weimer, and Y . Huang, “Connecting the dots: rethinking the relationship between code and prose writing with func- tional connectivity,” in Proceedings of the ACM Joint Meeting on European Software Eng. Conference and Symposium on the Foundations of Softw...

    2021

  14. [22]

    An empirical study on the impact of C++ lambdas and programmer experience,

    P. Uesbeck, A. Stefik, S. Hanenberg, J. Pedersen, and P. Daleiden, “An empirical study on the impact of C++ lambdas and programmer experience,” in Proc. Int. Conf. Software Engineering (ICSE) . ACM, 2016, pp. 760–771

    2016

  15. [23]

    An empirical investigation into programming language syntax,

    A. Stefik and S. Siebert, “An empirical investigation into programming language syntax,” ACM Transactions on Computing Education , vol. 13, no. 4, pp. 19:1–19:40, 2013

    2013

  16. [24]

    Object-oriented program comprehension: Effect of expertise, task and phase,

    J.-M. Burkhardt, F. D ´etienne, and S. Wiedenbeck, “Object-oriented program comprehension: Effect of expertise, task and phase,” Empirical Software Engineering, vol. 7, no. 2, pp. 115–156, 2002

    2002

  17. [25]

    Expertise in debugging computer programs: A process analysis,

    I. Vessey, “Expertise in debugging computer programs: A process analysis,” International Journal of Man–Machine Studies, vol. 23, no. 5, pp. 459–494, 1985

    1985

  18. [26]

    D ´etienne, Software Design—Cognitive Aspects

    F. D ´etienne, Software Design—Cognitive Aspects . Springer, 2002

    2002

  19. [27]

    Empirical evaluation of the effects of expe- rience on code quality and programmer productivity: An exploratory study,

    O. Dieste, A. Aranda, F. U. Uyaguari, B. Turhan, A. Tosun, D. Fucci, M. Oivo, and N. Juristo, “Empirical evaluation of the effects of expe- rience on code quality and programmer productivity: An exploratory study,” Empirical Software Engineering , vol. 22, no. 5, pp. 2457–2542, 2017

    2017

  20. [28]

    Automated Feedback Gen- eration for Introductory Programming Assignments,

    R. Singh, S. Gulwani, and A. Solar-Lezama, “Automated Feedback Gen- eration for Introductory Programming Assignments,” in Proceedings of the Conference on Programming Language Design and Implementation (PLDI). ACM, 2013, pp. 15–26

    2013

  21. [29]

    Automated Clustering and Program Repair for Introductory Programming Assignments,

    S. Gulwani, I. Radicek, and F. Zuleger, “Automated Clustering and Program Repair for Introductory Programming Assignments,” in Pro- ceedings of the Conference on Programming Language Design and Implementation (PLDI). ACM, 2018, pp. 465–480

    2018

  22. [30]

    A Feasibility Study of Using Automated Program Repair for Introductory Programming Assignments,

    J. Yi, U. Z. Ahmed, A. Karkare, S. H. Tan, and A. Roychoudhury, “A Feasibility Study of Using Automated Program Repair for Introductory Programming Assignments,” in Proceedings of the Joint Meeting on Foundations of Software Engineering (FSE) , 2017

    2017

  23. [31]

    Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis,

    A. Head, E. L. Glassman, G. Soares, R. Suzuki, L. Figueredo, L. D’Antoni, and B. Hartmann, “Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis,” in Proceedings of the Conference on Learning @ Scale (L@S) . ACM, 2017, pp. 89–98

    2017

  24. [32]

    What Would Other Programmers Do: Suggesting Solutions to Error Mes- sages,

    B. Hartmann, D. MacDougall, J. Brandt, and S. R. Klemmer, “What Would Other Programmers Do: Suggesting Solutions to Error Mes- sages,” in Proceedings of the International Conference on Human Factors in Computing Systems (CHI) , 2010

    2010

  25. [33]

    A Review on Recommending Solutions for Bugs Using Crowdsourcing,

    A. Al-batlaa, M. Abdullah-Al-Wadud, and M. A. Hossain, “A Review on Recommending Solutions for Bugs Using Crowdsourcing,” in Saudi Computer Society National Computer Conference (NCC) , 2018

    2018

  26. [34]

    Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models,

    T. Phung, J. Cambronero, S. Gulwani, T. Kohn, R. Majumdar, A. Singla, and G. Soares, “Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models,” in Proceedings of the Conference on Educational Data Mining (EDM) , 2023

    2023

  27. [35]

    Correlates of programmer efficacy and their link to experience: A combined eeg and eye-tracking study,

    N. Peitek, A. Bergum, M. Rekrut, J. Mucke, M. Nadig, C. Parnin, J. Siegmund, and S. Apel, “Correlates of programmer efficacy and their link to experience: A combined eeg and eye-tracking study,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and ...

    2022

  28. [36]

    Barriers for students during code change comprehension,

    J. Middleton, J.-P. Ore, and K. T. Stolee, “Barriers for students during code change comprehension,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13

    2024

  29. [37]

    GeeksforGeeks: A Computer Science Portal for Geeks,

    geeksforgeeks.org, “GeeksforGeeks: A Computer Science Portal for Geeks,” https://www.geeksforgeeks.org/, 2009

    2009

  30. [38]

    The recurring rainfall problem,

    K. Fisler, “The recurring rainfall problem,” in Proc. of the Conference on International Computing Education Research (ICER) , 2014

    2014

  31. [39]

    Explaining Code with a Purpose: An Integrated Approach for Developing Code Comprehension and Prompting Skills,

    P. Denny, D. H. S. IV , M. Fowler, J. Prather, B. A. Becker, and J. Leinonen, “Explaining Code with a Purpose: An Integrated Approach for Developing Code Comprehension and Prompting Skills,” CoRR, vol. abs/2403.06050, 2024

    2024 arXiv

  32. [40]

    BugSpotter: Automated Genera- tion of Code Debugging Exercises,

    V . Padurean, P. Denny, and A. Singla, “BugSpotter: Automated Genera- tion of Code Debugging Exercises,” CoRR, vol. abs/2411.14303, 2024

    2024 arXiv

  33. [41]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,” in Break- throughs in statistics: Methodology and distribution . Springer, 1992, pp. 196–202

    1992

  34. [42]

    The kolmogorov-smirnov test for goodness of fit,

    F. J. Massey Jr, “The kolmogorov-smirnov test for goodness of fit,” Journal of the American statistical Association , vol. 46, no. 253, pp. 68–78, 1951

    1951

  35. [43]

    Cohen, Statistical power analysis for the behavioral sciences

    J. Cohen, Statistical power analysis for the behavioral sciences . rout- ledge, 2013

    2013

  36. [44]

    Towards a theory of the comprehension of computer programs,

    R. Brooks, “Towards a theory of the comprehension of computer programs,” Journal of Man–Machine Studies , vol. 18, no. 6, pp. 543– 554, 1983

    1983

  37. [45]

    Empirical studies of programming knowl- edge,

    E. Soloway and K. Ehrlich, “Empirical studies of programming knowl- edge,” IEEE Transactions on Software Engineering , vol. 10, no. 5, pp. 595–609, 1984

    1984

  38. [46]

    Syntactic/semantic interactions in programmer behavior: A model and experimental results,

    B. Shneiderman and R. Mayer, “Syntactic/semantic interactions in programmer behavior: A model and experimental results,” Journal of Parallel Programming, vol. 8, no. 3, pp. 219–238, 1979

    1979

  39. [47]

    Stimulus structures and mental representations in expert comprehension of computer programs,

    N. Pennington, “Stimulus structures and mental representations in expert comprehension of computer programs,” Cognitive Psychology, vol. 19, no. 3, pp. 295–341, 1987

    1987

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.