Pith. sign in

REVIEW 2 major objections 3 minor 51 references

A Survey of Automated Programming Hint Generation -- The HINTS Framework

T0 review · 2 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that all surveyed automated programming hint techniques reduce to the same iterative pattern: a narrow-down step that selects a subset of hint data by relevance or quality, and a transformation step that changes the…

desk verdict HINTS gives the field a useful component-level vocabulary, but the paper's universal claim is too elastic to be falsifiable and the survey doesn't test it. read the letter →

arxiv 1908.11566 v1 pith:7TSSUONI submitted 2019-08-30 cs.HC cs.AIcs.CY

Jessica McBroom , Irena Koprinska , Kalina Yacef This is my paper · ORCID
classification cs.HCcs.AIcs.CY
keywords automatedprogramminghintsframeworkhintgenerationsurveynarrow-downsteptransformationeducationintelligenttutoringsystemsdata-drivenevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the many apparently different techniques for automatically generating programming hints are not truly disparate: each can be described as a sequence of just two kinds of operations on hint data. The first is a narrow-down step, which selects a subset of the available data using relevance to the student's program and/or some quality criterion. The second is a transformation step, which changes how the data is represented, for instance by dividing a solution into functions or converting programs to abstract syntax trees. Applying these steps iteratively lets a system move from raw input data (peer submissions, teacher hints, test cases) to a final hint. The paper surveys 2014-2018 hint-generation work under this HINTS framework and argues that even sophisticated systems such as the Hint Factory, AskElle, and SourceCheck reduce to this common structure, which would let researchers compare, reuse, and compose components rather than treating each technique as a monolithic black box.

What carries the argument

The HINTS framework is the central object: a description of hint generation as a sequence of steps applied iteratively to a pool of 'hint data' (peer submissions, teacher-written hints, model solutions, test cases, or the student's own program), ending when a selected set of hint data is offered to the student as a hint. The framework's two operations are the narrow-down step, which selects a subset of the current data using a relevance criterion and/or a quality criterion, and the transformation step, which changes the data's representation—e.g., converting programs to abstract syntax trees, collecting programs into states of a Markov decision process, or normalizing programs to a canonical form. The framework does the argument's work by providing a uniform diagram language: every surveyed technique is redrawn as a chain of these steps, which is what makes the 'all techniques have the same structure' claim visible and checkable.

What would settle it

Run a systematic literature search of automated hint generation for programming exercises from 2014-2018 and attempt to map every technique onto the HINTS framework; any technique whose generation process cannot be expressed as a finite sequence of narrow-down steps (selecting a subset of current hint data by a relevance or quality criterion) and transformation steps (changing the representation of the data) refutes the claim. Simpler still: find one published hint system whose output is produced with no step that chooses among alternatives and no step that re-represents its input.

Watch

Extended reading notes

Core claim

The paper's central claim is that every automated hint-generation technique within its scope—recent (2014-2018) methods for producing hints for programming exercises—can be understood as an iterative application of just two operations. A narrow-down step takes some set of hint data and selects a subset according to a relevance criterion (ties the data to the student's program) and/or a quality criterion (some measure of correctness, popularity, or usefulness). A transformation step changes how the hint data is represented, for example by splitting a program into functions, converting it to an abstract syntax tree, or grouping peer submissions into states. The paper shows through a series of worked examples—MistakeBrowser, spectrum-based fault localization, SourceCheck, the Hint Factory, AskElle, Codewebs, and SYNFIX—that systems which seem completely different can be drawn as the same kind of flowchart, and it argues that these two operations therefore characterize the nature of hint generation. It concludes that hint systems should be designed, communicated, evaluated, and compared at the level of these components, and that this perspective reveals links between hint generation and data-driven evaluation.

Load-bearing premise

The load-bearing premise is that the example techniques selected to guide the survey represent the full space of automated programming hint methods published in 2014-2018; if a published technique cannot be split into selecting a subset of data and changing how data is represented, the paper's universal claim fails.

Editorial extensions

If this is right

  • If the HINTS claim is right, hint techniques can be compared step-by-step rather than as monolithic systems, so a component shown to work in one system can be reused in another.
  • The choice of relevance and quality criteria at each narrow-down step becomes a first-class object of study; evaluating those criteria separately could reveal which choices drive hint quality and availability.
  • The framework implies a large combinatorial space of possible hint systems built by recombining existing steps, which strengthens the paper's call for scalable evaluation methods.
  • The noted correspondence between narrow-down steps and data-driven evaluation suggests that an evaluation metric (e.g., distance-to-solution) could be reused as a selection criterion inside a hint generator, and vice versa.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A systematic coding of the full 2014-2018 hint-generation literature—rather than the paper's guiding examples—would test whether the two operations are jointly sufficient, and would likely reveal whether any technique needs a third kind of step.
  • If the two-operation characterization holds as a definitional law, then novelty in future hint systems will usually be a new transformation or a new relevance/quality criterion, making the design space enumerable rather than open-ended.
  • The framework could generalize to other tutoring domains, such as logic or mathematics, since a hint that selects among alternatives and re-represents the problem state would fit the same two operations; the paper mentions this possibility only as future work.
  • Adopting the component view would change empirical reporting: studies could state which steps were held fixed and which varied, making conflicting results across hint systems easier to reconcile.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper claims that all automated programming hint generation techniques can be understood as iterative applications of two operations on hint data: a narrow-down step that selects a subset by relevance and/or quality criteria, and a transformation step that changes how the data is represented. It introduces the HINTS framework, presents an example-guided survey of 2014-2018 hint generation techniques organized around selecting next steps, generating steps toward a goal, comparing program features, and repairing programs, and concludes with implications for designing, communicating, and evaluating hint systems.

Significance. If the universal claim is sound, the framework would provide a genuinely useful unifying vocabulary for a fragmented literature, enabling component reuse, more focused evaluation questions, and links between hint generation and data-driven evaluation. The paper's strengths include its concrete diagrams mapping representative systems (MistakeBrowser, Hint Factory, AskElle, Codewebs, SYNFIX, and others) into the framework, its clear presentation of a wide range of recent work, and its thoughtful discussion of evaluation implications. The load-bearing weakness is that the central universality claim is not well-posed or adequately tested; as stated, the two operations are broad enough to make the claim nearly unfalsifiable, and the survey method is explicitly example-guided rather than systematic. These issues are fixable by either formalizing the operations or weakening the claim, so the paper is a credible candidate for major revision.

major comments (2)
  1. [§4.2, §5.1.2, §6.4] The central claim that all hint techniques consist of iterative narrow-down and transformation steps is not well-posed as stated. The definition of transformation as 'changing the way hint data is represented' and narrow-down as 'selecting a subset' is broad enough that almost any pipeline can be redescribed in these terms, making the universal claim near-tautological. The Continuous Hint Factory (Section 5.1.2) illustrates the problem: its next state is a weighted sum of peer edits, which is not a subset of existing states and, under the examples given in Section 4.2, is not a representation change either. The paper does not say which HINTS operation produces this synthetic state. Since the abstract and Section 6.4 assert that all techniques are constrained to exactly these two operations, this is a load-bearing gap. The authors should either give precise, exclusionary definitions of the two operations or replace the universal claim with a weaker 'usefully viewed as' claim and state what would count as a counterexample.
  2. [§5 introduction, §3, §6.4] The survey is explicitly example-guided rather than systematic: Section 5 states that the review 'progresses through a series of stages, guided by example hint techniques' and that the sections 'do not represent a general categorisation.' No search strategy, inclusion/exclusion criteria, or screening counts are reported in Section 3 or elsewhere. Nevertheless, Section 6.4 makes the universal empirical claim that 'all automated hint techniques exhibit remarkable similarities in structure' and are described by the two operations. The curated examples are insufficient evidence for universality; if a 2014-2018 technique does not decompose into narrow-down and transformation steps, the central claim fails. The authors should either conduct a systematic literature search and report its protocol, or explicitly restrict the framework's scope to the surveyed techniques.
minor comments (3)
  1. [Table 7] The header 'MisktakeBrowser' is a typo and should read 'MistakeBrowser.'
  2. [Figure 5 caption] The caption says 'A visualisation of how the program strategy hint technique in SYNFIX fits into the HINTS framework,' but SYNFIX uses an RNN-based correction model rather than the program-strategy technique described in Figure 3; the caption appears to be a copy-paste error.
  3. [Section 5.1.2] There is a typo in the sentence 'if output is used, the the hints would suggest the next output to aim for'; the duplicated 'the' should be removed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HINTS is a descriptive survey framework, not a derivation that feeds its own conclusion.

full rationale

The paper proposes a descriptive scheme (Sec. 4.2) and uses it to organize a survey (Sec. 5); it does not derive a quantitative prediction from fitted parameters, nor does it rely on a self-citation chain to justify its central claim. The two self-citations that occur ([16], [50]) are used as one surveyed hint method and one evaluation example, not as load-bearing justification. The HINTS universality statement in Sec. 6.4 is broad and definitionally flexible — 'transformation' is defined as any representational change — but this is a falsifiability/scope weakness, not a case of the framework's output being identical to its input by construction. The paper itself notes in Sec. 5 that the review is example-guided and 'does not represent a general categorisation,' which tempers the universal claim without making it circular.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No numeric parameters are fitted. The paper's contribution is conceptual: it introduces the HINTS framework and applies it to selected systems. The main assumptions are the representativeness of the surveyed examples and the sufficiency of the two operations; both are reasonable for an organizing framework but are not empirically established.

assumptions (3)
  • domain assumption The example techniques surveyed in Section 5 are representative of all automated programming hint techniques published in 2014-2018.
    Section 3 sets the scope; Section 5 is explicitly example-guided, and Section 6.4 generalizes from these examples to all hint techniques.
  • domain assumption Narrow-down and transformation are sufficient primitives for describing hint generation.
    This is the central claim of the HINTS framework in Section 4.2, asserted as the basis of the framework rather than derived from external constraints.
  • domain assumption Hints are defined as feedback that improves a student's knowledge of how to complete a programming exercise, excluding encouragement and timing-based decisions.
    Section 3 defines this scope; the universal claim applies only within this definition, so the definition is a load-bearing premise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Survey of Automated Programming Hint Generation -- The HINTS Framework." pith.science (2026). https://pith.science/paper/7TSSUONI

@misc{pith2026190811566,
  author       = {Pith},
  title        = {Pith review of: A Survey of Automated Programming Hint Generation -- The HINTS Framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7TSSUONI}},
  note         = {Machine review of arXiv:1908.11566}
}
read the original abstract

Automated tutoring systems offer the flexibility and scalability necessary to facilitate the provision of high quality and universally accessible programming education. In order to realise the full potential of these systems, recent work has proposed a diverse range of techniques for automatically generating hints to assist students with programming exercises. This paper integrates these apparently disparate approaches into a coherent whole. Specifically, it emphasises that all hint techniques can be understood as a series of simpler components with similar properties. Using this insight, it presents a simple framework for describing such techniques, the Hint Iteration by Narrow-down and Transformation Steps (HINTS) framework, and it surveys recent work in the context of this framework. It discusses important implications of the survey and framework, including the need to further develop evaluation methods and the importance of considering hint technique components when designing, communicating and evaluating hint systems. Ultimately, this paper is designed to facilitate future opportunities for the development, extension and comparison of automated programming hint techniques in order to maximise their educational potential.

Figures

Figures reproduced from arXiv: 1908.11566 by the authors.

Figure 1
Figure 1. The HINTS framework for describing automated hint generation techniques. A small pool of [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. A visualisation of the Hint Factory technique, using the HINTS framework (the colours corre [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. A visualisation of how the program strategy hint technique in AskElle [9] fits into the HINTS [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: A visualisation of how the hint techniques in Codewebs [38] fits into the HINTS framework. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: A visualisation of how the program strategy hint technique in SYNFIX [45] fits into the HINTS [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: A general diagram summarising the main ideas of the surveyed papers and the relationships [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 48 canonical work pages

  1. [1]

    Writing reusable code feedback at scale with mixed-initiative program synthesis,

    A. Head, E. Glassman, G. Soares, R. Suzuki, L. Figueredo, L. D’Antoni, and B. Hartmann, “Writing reusable code feedback at scale with mixed-initiative program synthesis,” inProceedings of the Fourth (2017) ACM Conference on Learning@ Scale , pp. 89–98, ACM, 2017

    work page 2017

  2. [2]

    Vida: A virtual debugging advisor for supporting learning in computer programming courses,

    V. C. Lee, Y.-T. Yu, C. M. Tang, T.-L. Wong, and C. K. Poon, “Vida: A virtual debugging advisor for supporting learning in computer programming courses,” Journal of Computer Assisted Learning, vol. 34, no. 3, pp. 243–258, 2018

    work page 2018

  3. [3]

    Providing meaningful feedback for autograding of programming assignments,

    G. Haldeman, A. Tjang, M. Babe¸ s-Vroman, S. Bartos, J. Shah, D. Yucht, and T. D. Nguyen, “Providing meaningful feedback for autograding of programming assignments,” in Proceedings of the 49th ACM Technical Symposium on Computer Science Education , pp. 278–283, ACM, 2018

    work page 2018

  4. [4]

    Automatic extraction of ast patterns for debugging stu- dent programs,

    T. Lazar, M. Moˇ zina, and I. Bratko, “Automatic extraction of ast patterns for debugging stu- dent programs,” in International Conference on Artificial Intelligence in Education , pp. 162–174, Springer, 2017

    work page 2017

  5. [5]

    Generating hints for programming problems using inter- mediate output,

    B. Peddycord Iii, A. Hicks, and T. Barnes, “Generating hints for programming problems using inter- mediate output,” in Proceedings of the 7th International Conference on Educational Data Mining , pp. 92–98, International Educational Data Mining Society, 2014

    work page 2014

  6. [6]

    Rewrite rules for debugging student programs in programming tutors,

    T. Lazar, A. Sadikov, and I. Bratko, “Rewrite rules for debugging student programs in programming tutors,” IEEE Transactions on Learning Technologies, vol. 11, no. 4, pp. 429–440, 2017

    work page 2017

  7. [7]

    Applying spectrum-based fault localization to generate debugging suggestions for student programmers,

    B. Edmison and S. H. Edwards, “Applying spectrum-based fault localization to generate debugging suggestions for student programmers,” in Software Reliability Engineering Workshops (ISSREW), 2015 IEEE International Symposium on , pp. 93–99, IEEE, 2015

    work page 2015

  8. [8]

    Strategy-based feedback in a programming tutor,

    H. Keuning, B. Heeren, and J. Jeuring, “Strategy-based feedback in a programming tutor,” in Proceedings of the Computer Science Education Research Conference, pp. 43–54, ACM, 2014

    work page 2014

Show all 51 references
  1. [9]

    Ask-elle: an adaptable programming tutor for haskell giving automated feedback,

    A. Gerdes, B. Heeren, J. Jeuring, and L. T. van Binsbergen, “Ask-elle: an adaptable programming tutor for haskell giving automated feedback,” International Journal of Artificial Intelligence in Education, vol. 27, no. 1, pp. 65–100, 2017

    2017

  2. [10]

    Evaluation of a data-driven feedback algorithm for open-ended programming.,

    T. W. Price, R. Zhi, and T. Barnes, “Evaluation of a data-driven feedback algorithm for open-ended programming.,” in Proceedings of the 10th International Conference on Educational Data Mining , pp. 192–197, International Educational Data Mining Society, 2017

    2017

  3. [11]

    Data-driven hint generation in vast solution spaces: a self- improving python programming tutor,

    K. Rivers and K. R. Koedinger, “Data-driven hint generation in vast solution spaces: a self- improving python programming tutor,” International Journal of Artificial Intelligence in Education, vol. 27, no. 1, pp. 37–64, 2017

    2017

  4. [12]

    Autonomously generating hints by inferring problem solving policies,

    C. Piech, M. Sahami, J. Huang, and L. Guibas, “Autonomously generating hints by inferring problem solving policies,” in Proceedings of the Second (2015) ACM Conference on Learning@ Scale, pp. 195– 204, ACM, 2015

    2015

  5. [13]

    Generating data-driven hints for open-ended programming.,

    T. W. Price, Y. Dong, and T. Barnes, “Generating data-driven hints for open-ended programming.,” in Proceedings of the 9th International Conference on Educational Data Mining , pp. 191–198, Inter- national Educational Data Mining Society, 2016

    2016

  6. [14]

    An incremental hint system for automated programming assignments,

    P. Antonucci, C. Estler, D. Nikoli´ c, M. Piccioni, and B. Meyer, “An incremental hint system for automated programming assignments,” in Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education , pp. 320–325, ACM, 2015

    2015

  7. [15]

    Automated personalized feedback in introductory java programming moocs,

    V. J. Marin, T. Pereira, S. Sridharan, and C. R. Rivero, “Automated personalized feedback in introductory java programming moocs,” inData Engineering (ICDE), 2017 IEEE 33rd International Conference on, pp. 1259–1270, IEEE, 2017. 23

    2017

  8. [16]

    Automated data-driven hints for computer pro- gramming students,

    S. Chow, K. Yacef, I. Koprinska, and J. Curran, “Automated data-driven hints for computer pro- gramming students,” in Adjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization, pp. 5–10, ACM, 2017

    2017

  9. [17]

    A comparison of the quality of data-driven programming hint generation algorithms,

    T. W. Price, Y. Dong, R. Zhi, B. Paaßen, N. Lytle, V. Catet´ e, and T. Barnes, “A comparison of the quality of data-driven programming hint generation algorithms,” International Journal of Artificial Intelligence in Education , pp. 1–28, 2019

    2019

  10. [18]

    A classification of adaptive feedback in educational systems for programming,

    N.-T. Le, “A classification of adaptive feedback in educational systems for programming,” Systems, vol. 4, no. 2, p. 22, 2016

    2016

  11. [19]

    A review of static analysis approaches for programming exercises,

    M. Striewe and M. Goedicke, “A review of static analysis approaches for programming exercises,” in International Computer Assisted Assessment Conference , pp. 100–113, Springer, 2014

    2014

  12. [20]

    A systematic literature review of automated feedback gen- eration for programming exercises,

    H. Keuning, J. Jeuring, and B. Heeren, “A systematic literature review of automated feedback gen- eration for programming exercises,” ACM Transactions on Computing Education (TOCE) , vol. 19, no. 1, p. 3, 2018

    2018

  13. [21]

    Towards a systematic review of automated feedback gen- eration for programming exercises,

    H. Keuning, J. Jeuring, and B. Heeren, “Towards a systematic review of automated feedback gen- eration for programming exercises,” in Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education , pp. 41–46, ACM, 2016

    2016

  14. [22]

    A survey of automated assessment approaches for programming assignments,

    K. M. Ala-Mutka, “A survey of automated assessment approaches for programming assignments,” Computer science education, vol. 15, no. 2, pp. 83–102, 2005

    2005

  15. [23]

    Review of recent systems for automatic assessment of programming assignments,

    P. Ihantola, T. Ahoniemi, V. Karavirta, and O. Sepp¨ al¨ a, “Review of recent systems for automatic assessment of programming assignments,” in Proceedings of the 10th Koli calling international con- ference on computing education research, pp. 86–93, ACM, 2010

    2010

  16. [24]

    A review of ai-supported tutoring approaches for learning programming,

    N.-T. Le, S. Strickroth, S. Gross, and N. Pinkwart, “A review of ai-supported tutoring approaches for learning programming,” in Advanced Computational Methods for Knowledge Engineering , pp. 267– 279, Springer, 2013

    2013

  17. [25]

    Intelligent tutoring systems for programming educa- tion: a systematic review,

    T. Crow, A. Luxton-Reilly, and B. Wuensche, “Intelligent tutoring systems for programming educa- tion: a systematic review,” in Proceedings of the 20th Australasian Computing Education Conference, pp. 53–62, ACM, 2018

    2018

  18. [26]

    Automatic software repair: a bibliography,

    M. Monperrus, “Automatic software repair: a bibliography,” ACM Computing Surveys (CSUR) , vol. 51, no. 1, p. 17, 2018

    2018

  19. [27]

    A survey on algorithmic debugging strategies,

    J. Silva, “A survey on algorithmic debugging strategies,” Advances in engineering software, vol. 42, no. 11, pp. 976–991, 2011

    2011

  20. [28]

    Adaptive feedback based on student emotion in a system for program- ming practice,

    T. J. Tiam-Lee and K. Sumi, “Adaptive feedback based on student emotion in a system for program- ming practice,” in International Conference on Intelligent Tutoring Systems , pp. 243–255, Springer, 2018

    2018

  21. [29]

    An affective and cognitive tutoring system for learning programming,

    M. L. Barr´ on-Estrada, R. Zatarain-Cabada, F. G. Hern´ andez, R. O. Bustillos, and C. A. Reyes- Garc´ ıa, “An affective and cognitive tutoring system for learning programming,” in Mexican Inter- national Conference on Artificial Intelligence , pp. 171–182, Springer, 2015

    2015

  22. [30]

    Comparing different styles of automated feedback for programming exercises,

    A. Annamaa, R. Suviste, and V. Vene, “Comparing different styles of automated feedback for programming exercises,” in Proceedings of the 17th Koli Calling Conference on Computing Education Research, pp. 183–184, ACM, 2017

    2017

  23. [31]

    The hint factory: Automatic generation of contextualized help for existing computer aided instruction,

    J. Stamper, T. Barnes, L. Lehmann, and M. Croy, “The hint factory: Automatic generation of contextualized help for existing computer aided instruction,” in Proceedings of the 9th International Conference on Intelligent Tutoring Systems Young Researchers Track , pp. 71–78, 2008

    2008

  24. [32]

    Building games to learn from their players: Generating hints in a serious game,

    A. Hicks, B. Peddycord, and T. Barnes, “Building games to learn from their players: Generating hints in a serious game,” in International Conference on Intelligent Tutoring Systems , pp. 312–317, Springer, 2014. 24

    2014

  25. [33]

    Automating hint generation with solution space path construction,

    K. Rivers and K. R. Koedinger, “Automating hint generation with solution space path construction,” in International Conference on Intelligent Tutoring Systems , pp. 329–339, Springer, 2014

    2014

  26. [34]

    The continuous hint factory-providing hints in vast and sparsely populated edit distance spaces,

    B. Paaßen, B. Hammer, T. W. Price, T. Barnes, S. Gross, and N. Pinkwart, “The continuous hint factory-providing hints in vast and sparsely populated edit distance spaces,” arXiv preprint arXiv:1708.06564, 2017

    2017 arXiv

  27. [35]

    Tipsc: tips and corrections for programming moocs,

    S. Sharma, P. Agarwal, P. Mor, and A. Karkare, “Tipsc: tips and corrections for programming moocs,” in International Conference on Artificial Intelligence in Education , pp. 322–326, Springer, 2018

    2018

  28. [36]

    Gerdes, Ask-Elle: a Haskell Tutor

    A. Gerdes, Ask-Elle: a Haskell Tutor . PhD thesis, Universiteit Utrecht, 2012

    2012

  29. [37]

    Hint generation under uncertainty: the effect of hint quality on help-seeking behavior,

    T. W. Price, R. Zhi, and T. Barnes, “Hint generation under uncertainty: the effect of hint quality on help-seeking behavior,” in International Conference on Artificial Intelligence in Education, pp. 311– 322, Springer, 2017

    2017

  30. [38]

    Codewebs: scalable homework search for massive open online programming courses,

    A. Nguyen, C. Piech, J. Huang, and L. Guibas, “Codewebs: scalable homework search for massive open online programming courses,” in Proceedings of the 23rd international conference on World wide web, pp. 491–502, ACM, 2014

    2014

  31. [39]

    Model solutions and properties for diagnosing student programs in ask-elle,

    J. Jeuring, L. T. van Binsbergen, A. Gerdes, and B. Heeren, “Model solutions and properties for diagnosing student programs in ask-elle,” in Proceedings of the Computer Science Education Research Conference, pp. 31–40, ACM, 2014

    2014

  32. [40]

    Fifteen years of constraint-based tutors: what we have achieved and where we are going,

    A. Mitrovic, “Fifteen years of constraint-based tutors: what we have achieved and where we are going,” User modeling and user-adapted interaction , vol. 22, no. 1-2, pp. 39–72, 2012

    2012

  33. [41]

    Execution traces as a powerful data representation for intelligent tutoring systems for programming.,

    B. Paaßen, J. Jensen, and B. Hammer, “Execution traces as a powerful data representation for intelligent tutoring systems for programming.,” in Proceedings of the 9th International Conference on Educational Data Mining , pp. 183–190, International Educational Data Mining Society, 2016

    2016

  34. [42]

    Feedback generation for performance problems in in- troductory programming assignments,

    S. Gulwani, I. Radiˇ cek, and F. Zuleger, “Feedback generation for performance problems in in- troductory programming assignments,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering , pp. 41–51, ACM, 2014

    2014

  35. [43]

    Real-time programming exercise feedback in moocs.,

    Z. Chen, A. Nguyen, A. Schlender, and J. Ngiam, “Real-time programming exercise feedback in moocs.,” in Proceedings of the 10th International Conference on Educational Data Mining , pp. 414– 415, International Educational Data Mining Society, 2017

    2017

  36. [44]

    Learning program embeddings to propagate feedback on student code,

    C. Piech, J. Huang, A. Nguyen, M. Phulsuksombati, M. Sahami, and L. Guibas, “Learning program embeddings to propagate feedback on student code,” arXiv preprint arXiv:1505.05969 , 2015

    2015 arXiv

  37. [45]

    Automated correction for syntax errors in programming assignments using recurrent neural networks,

    S. Bhatia and R. Singh, “Automated correction for syntax errors in programming assignments using recurrent neural networks,” arXiv preprint arXiv:1603.06129 , 2016

    2016 arXiv

  38. [46]

    Deep reinforcement learning for programming language correction,

    R. Gupta, A. Kanade, and S. Shevade, “Deep reinforcement learning for programming language correction,” arXiv preprint arXiv:1801.10467 , 2018

    2018 arXiv

  39. [47]

    Identifying typical approaches and errors in prolog pro- gramming with argument-based machine learning,

    M. Moˇ zina, T. Lazar, and I. Bratko, “Identifying typical approaches and errors in prolog pro- gramming with argument-based machine learning,” Expert Systems with Applications , vol. 112, pp. 110–124, 2018

    2018

  40. [48]

    Test-driven synthesis for automated feedback for in- troductory computer science assignments,

    D. Perelman, S. Gulwani, and D. Grossman, “Test-driven synthesis for automated feedback for in- troductory computer science assignments,” Proceedings of Data Mining for Educational Assessment and Feedback (ASSESS 2014), 2014

    2014

  41. [49]

    Exploring the design space of automatically synthesized hints for introductory programming assignments,

    R. Suzuki, G. Soares, E. Glassman, A. Head, L. D’Antoni, and B. Hartmann, “Exploring the design space of automatically synthesized hints for introductory programming assignments,” in Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing System...

    2017

  42. [50]

    A data-driven method for helping teachers improve feedback in computer programming automated tutors,

    J. McBroom, K. Yacef, I. Koprinska, and J. R. Curran, “A data-driven method for helping teachers improve feedback in computer programming automated tutors,” in International Conference on Artificial Intelligence in Education , pp. 324–337, Springer, 2018

    2018

  43. [51]

    The impact of data quantity and source on the quality of data-driven hints for programming,

    T. W. Price, R. Zhi, Y. Dong, N. Lytle, and T. Barnes, “The impact of data quantity and source on the quality of data-driven hints for programming,” in International Conference on Artificial Intelligence in Education , pp. 476–490, Springer, 2018. 26

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.