Bridging Gaps Between Student and Expert Evaluations of AI-Generated Programming Hints

Adish Singla; Christopher Brooks; Gustavo Soares; Heeryung Choi; Mengyan Wu; Sumit Gulwani; Tung Phung

REVIEW 3 major objections 4 minor 37 references

Bridging Gaps Between Student and Expert Evaluations of AI-Generated Programming Hints

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that expert rubric ratings of AI-generated programming hints are systematically more optimistic than student helpfulness ratings, identifies five reasons for the mismatch, and proposes rubric extensions and prompt…

desk verdict A useful small-study comparison of student vs. expert ratings of AI hints, but the headline mismatch is statistically underspecified and the fixes are validated on the cases that motivated them. read the letter →

arxiv 2509.03269 v1 pith:DYXNJDGI submitted 2025-09-03 cs.CY

Tung Phung , Mengyan Wu , Heeryung Choi , Gustavo Soares , Sumit Gulwani , Adish Singla , Christopher Brooks This is my paper · ORCID

classification cs.CY

keywords generativeAIineducationprogrammingfeedbackhintgenerationexpertrubricsstudenthelpfulnessratingsmismatchtaxonomypersonalizedlearningatscale

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that expert rubric ratings of AI-generated programming hints are systematically more optimistic than the ratings of the students who receive them. In a Python course, students and experts disagreed on 48 of 139 rated hints, and in 37 cases the experts called a hint high-quality while the student called it unhelpful—a pattern the paper judges unlikely to be chance ($p \le 0.001$ by a $\chi^2$ test). It names five reasons for these over-optimistic ratings, such as hints that ignore the student's stated concern or previous hints, and it proposes an extended rubric plus prompt changes that use the student's thoughts and request history. If the paper is right, offline expert-only evaluation pipelines can overstate the quality of AI hints, and the new rubric attributes would make expert ratings a better stand-in for student helpfulness.

What carries the argument

The argument runs on a $2\times2$ contingency table of expert versus student ratings, tested with a $\chi^2$ test, plus a qualitative taxonomy built by hand-coding the 37 high-quality-but-unhelpful cases. The deployed hint generator is a two-stage pipeline: it runs the student's buggy program to obtain the buggy output and a fixed program, then asks a large language model for a chain-of-thought explanation followed by a Socratic-style hint. The taxonomy of five mismatch categories is the central explanatory device, and the proposed remedy is an augmented rubric whose four new attributes—Accounting student's concern, Informative given history, Tackling overarching bug, and Guiding—are designed to capture what the original four attributes miss.

What would settle it

Have two or more independent expert pairs rate the same 139 hints with the four-attribute rubric; if expert-pair agreement is low, the 37 'high-quality but unhelpful' count is an artifact of one pair's judgments. A complementary test: in a fresh course, ask students who rate a hint unhelpful to choose from the five mismatch categories, and check whether the categories actually account for their reasons.

Watch

Extended reading notes

Core claim

Comparing a four-attribute expert rubric (Correct, Informative, Conceal, Comprehensible) with student Helpful/Unhelpful ratings on 139 hints from a deployed AI hint generator yielded agreement in 91 cases (65.5%) and disagreement in 48 cases (34.5%), with the disagreement pattern significant by a $\chi^2$ test. Of the 48 disagreements, 37 were hints the experts rated high-quality but the student rated unhelpful; the paper concludes that rubric-based expert evaluations may be overly optimistic. Coding those 37 cases produces five mismatch categories: mismatched pedagogical objectives, the student's concern not addressed, the student's trajectory ignored, the student's struggle in approach ignored, and the student's progress not acknowledged, with 11 cases of unknown reason. The paper then proposes four new rubric attributes and five prompt adjustments aimed at the four explainable categories, and reports preliminary results that the extended rubric re-rates all 16 targeted mismatch cases as low-quality.

Load-bearing premise

The expert side of the comparison is trusted as ground truth: two experts jointly scored every hint with one rubric and no inter-rater agreement statistic is reported, so if their ratings are idiosyncratic, the mismatch counts and the five categories lose their foundation.

Editorial extensions

If this is right

Rubric-only offline evaluation of hint generators is likely to overstate quality; techniques tuned to the original four attributes may be overfitting to criteria students do not weigh heavily.
The five mismatch categories give a concrete diagnostic checklist: a hint can fail by ignoring the student's concern, earlier hints, the overarching bug, the student's current approach, or the student's progress.
The extended rubric, with the four new attributes, should make expert ratings a closer proxy for student helpfulness, reducing the need to deploy unfinished techniques in classrooms.
Feeding pre-hint thoughts, post-hint thoughts, and request history into the hint-generation prompt should produce hints that target the student's actual difficulty rather than a generic bug in the code.
The extended rubric's effect on hints where students and experts already agree is not yet measured, so the authors' own caution is that the rubric needs validation on those cases before being adopted.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the over-optimism bias generalizes, earlier benchmark comparisons of hint generators that relied only on expert rubrics may need to be re-run with student-helpfulness data, since their relative rankings could shift.
The five-category taxonomy could be turned into a lightweight classifier that flags likely-unhelpful hints at generation time, trained on the student code, hint text, and interaction history.
The same student-versus-expert gap may appear in other formative AI-feedback settings, such as essay comments or math step hints, because the root cause is a rubric describing content quality rather than situational relevance.
An online protocol that asks students who rate a hint unhelpful to select a reason from the five categories would collect ground-truth labels at scale, replacing the hand-coding that currently limits the taxonomy.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

First thing to know: this paper does the direct comparison we've been missing. Prior work used either expert rubrics or student ratings; here the same 139 AI-generated hints get both, from 34 students in a real Python course, and experts rate them on a standard four-attribute rubric. The mismatch is visible: 37 hints experts called high-quality were rated unhelpful by the student, while only 11 went the other way. That asymmetry is the paper's main event.

What's good: the five-category taxonomy for why experts over-rate is plausible and grounded in examples. The extended rubric attributes (accounting for student's concern, informative given history, etc.) follow directly from the failure modes and are practical for anyone building hint generators. The paper is honest about being in-progress, small-sample, and single-course; the repo is public.

Now the soft spots. The chi-square test in RQ1 is the one real load-bearing issue. The comparison is paired—student and expert on the same hint—and the claim of 'overly optimistic' rests on the 37-vs-11 asymmetry. That needs McNemar or an exact binomial test on discordant pairs, not a standard chi-square of association. They cite Cochran 1952 but give no statistic or df, and with 139 hints nested in 34 students, the independence assumption is violated anyway. So p≤0.001 likely isn't the right test, and the conclusion that expert evaluations are 'overly optimistic' is, statistically speaking, still up in the air. The descriptive mismatch count stands, but the confidence level should be lower.

Other issues are softer. Two experts rate together with no IRR reported, so we don't know how much the expert side is a single opinion. The RQ2 categories are single-coder. And the preliminary validation of the extended rubric scores the same 16 cases it was built from—no surprise that it catches them. The authors label this as preliminary, so this is a minor over-claim, not a fatal one.

Bottom line: worth a serious referee. The comparative design and taxonomy are useful for the AI-tutoring community, and the questions are the right ones. The authors should be pushed to report a proper paired test with clustering, get a second coder on the taxonomy, and ideally validate the extended rubric on held-out data. If they do that, this becomes a genuinely useful reference point.

Referee Report

3 major / 4 minor

Summary. This paper reports on a classroom deployment of GPT-4-generated programming hints in a Python data science course. Of 74 enrolled students, 34 requested and rated hints; 139 rated hints are compared with ratings given by two experts who jointly applied a four-attribute rubric. The authors report 65.5% agreement between students and experts, with 37 cases in which experts rated a hint high-quality while the student rated it unhelpful, and 11 reverse mismatches. A chi-square test is reported as significant at p≤0.001, and the authors interpret the imbalance as evidence that rubric-based expert evaluations may be overly optimistic. They then classify 26 of the 37 mismatch cases into five qualitative categories and propose four additional rubric attributes and five adjustments to the hint-generation process, with preliminary validation on the same 16 mismatch cases that motivated the new rubric.

Significance. If the descriptive mismatch result is robust, the paper offers a useful, actionable contribution to the learning-at-scale community: it provides concrete evidence that expert-only rubric evaluation can diverge from students' perceived helpfulness, and it names five plausible reasons for that divergence. The strengths are the real classroom deployment, the public repository, the clear worked example in Figure 1, and the transparent reporting of all counts in Table 1. However, the main inferential claim currently rests on an underspecified statistical test, the expert and qualitative coding lack reliability evidence, and the preliminary validation of the extended rubric is circular. These issues do not destroy the descriptive value of the mismatch counts, but they do limit the strength of the conclusion that rubric-based expert evaluation is systematically overly optimistic.

major comments (3)

[§4, Table 1 (RQ1)] The reported chi-square test is not the appropriate test for the claim being made and is under-specified. The data are paired, because the same hint receives both a student rating and an expert rating, so the null hypothesis relevant to 'experts are more optimistic' is that the two discordant counts (37 and 11) are equal. That hypothesis should be tested with McNemar's test or an exact binomial test on the discordant pairs, not with a chi-square test of independence on the 2×2 table, which tests association rather than asymmetry. In addition, the 139 rated hints are nested within 34 students, roughly four hints per student, so unadjusted tests may overstate significance; a cluster-aware paired test or a mixed-effects model is needed. The manuscript cites Cochran (1952) without reporting the test statistic, degrees of freedom, or any treatment of clustering, so the reader cannot verify that p≤0.001 refers to the asymmetry. Please re-run the analysis with a paired, cluster-adjusted test and report the statistic; if significance does not survive, the mismatch finding should be presented as descriptive rather than inferential.
[§5, Table 3a (preliminary validation)] The preliminary validation of the extended rubric is circular. The four new attributes in Table 3a were derived from the same 16 mismatch cases that are then re-rated with the extended rubric, so the finding that all 16 hints are rated low-quality is largely by construction. This does not establish that the extended rubric aligns with student perceptions on unseen cases. Please validate the extended rubric on held-out mismatch cases, on cases where the original rubric agreed with students, and ideally with independent coders who were not involved in designing the attributes.
[§3 'Rubric and expert ratings'; §4 RQ2] The expert side of the comparison is a joint consensus rating produced by two experts discussing each hint, and no inter-rater reliability is reported. Because the central comparison treats expert ratings as ground truth, the absence of an independent-rating agreement statistic leaves open the possibility that the expert ratings are idiosyncratic. Similarly, the RQ2 categories in Table 2 were assigned by the authors without a second coder or an agreement measure such as Cohen's kappa. Please report inter-rater reliability for the rubric and coding reliability for the taxonomy, or explicitly characterize these as consensus/exploratory judgments whose generalizability is not yet established.

minor comments (4)

[Figure 1 caption] The caption says the adjusted prompt incorporates student thoughts with 'changes highlighted in yellow,' but the yellow highlighting is not visible in the printed text; please state exactly which parts were added.
[§5] The sentence 'Since resolving pedagogical conflicts is not the main consideration of this work' excludes the largest mismatch category from the 16-case validation; please clarify the rationale, since the pedagogical-objectives category is 27.0% of the known-reason cases.
[Abstract and §1] The paper moves between '74 students' in the course and '34 students requested hints'; consider stating both numbers in the abstract to avoid ambiguity about the effective sample size.
[Table 2] The five mismatch categories are presented as mutually exclusive, but a single hint could plausibly fit more than one category; please state whether the categories were treated as exclusive and, if so, how ties were resolved.

Circularity Check

2 steps flagged · score 6.0 of 10

Core student-vs-expert mismatch finding is independent, but the Section 5 rubric-extension validation reduces to a circular fit: the new rubric attributes are built from the same 16 mismatch cases they are then used to re-rate.

fitted input called prediction [Section 5, 'Ideas for Improvement and Initial Findings', Table 3a]
"Based on the results above, we propose an extension to the current rubric... Preliminary results show that experts following this extended rubric rate all 16 hints in the four mismatch categories of interest as low-quality, suggesting a higher alignment with students in cases the current rubric was insufficient."

Table 3a's four new attributes—'Accounting student's concern', 'Informative given history', 'Tackling overarching bug', and 'Guiding'—are direct operationalizations of the four RQ2 categories: 'Student's concern is not addressed', 'Student's trajectory is ignored', 'Struggle in solving approach is ignored', and 'Student's progress is not acknowledged'. The 16 hints 'in the four mismatch categories of interest' are exactly the hints categorized under those four failure modes. Re-scoring those same 16 hints with a rubric whose new attributes are defined to detect those failure modes mechanically yields low quality for all of them; the 16/16 result is entailed by the way the rubric and the sample are defined, not discovered empirically.
self definitional [Section 5, 'Ideas for Improvement and Initial Findings']
"In our initial findings, these adjustments address the 16 mismatch cases above, generating hints meeting the extended rubric, suggesting to be more helpful to students"

The proposed prompt adjustments (Pre-hint Thoughts, Post-hint Thoughts, Trajectory, Overarching Bug, Solving Plan) are engineered to supply exactly the information that the extended rubric attributes check for, and the evaluation criterion is the same extended rubric constructed from the 16 target cases. Thus the claim that the adjustments 'address the 16 mismatch cases above, generating hints meeting the extended rubric' is a closed loop: the target cases are the ones used to define both the rubric attributes and the adjustments. The paper labels these as 'initial findings' and acknowledges limitations, but this particular validation is circular because the metric and the sample share the same origin.

full rationale

The paper's main RQ1 result—37 hints where experts said high-quality but students said unhelpful, versus 11 in the opposite direction—is not circular. It compares student ratings collected during a course deployment with expert ratings made under a pre-existing four-attribute rubric from prior literature; the data are independently collected, and the conclusion that rubric-based expert evaluation may be overly optimistic is a substantive empirical claim. RQ2's mismatch categorization is explicitly described as 'informed by the patterns observed in the collected data,' which is descriptive rather than circular. The circularity is confined to Section 5's preliminary validation. The four added rubric attributes are one-to-one restatements of the four named mismatch categories, so re-rating the same 16 cases with the extended rubric cannot fail to mark them low-quality; this is fit-by-construction rather than evidence of better alignment. Similarly, evaluating the proposed prompt adjustments against that same extended rubric on the same 16 cases creates a closed validation loop. The paper's own hedges ('preliminary results', 'suggesting', 'initial findings') and its limitation statement that 'the extended rubric might also render more hints to be recognized as low-quality' reduce the weight of the circularity, but they do not remove it. The statistical critique of the RQ1 chi-square test (paired data, clustering by student) is a correctness and robustness concern, not a circularity concern, and therefore does not contribute to the circularity score.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities. The analysis depends on measurement assumptions about student self-reports, joint expert ratings, and hand-coded categories; the extended rubric validation is in-sample and therefore weak evidence.

assumptions (3)

domain assumption Student binary Helpful/Unhelpful ratings are a valid measure of perceived hint helpfulness.
The central comparison treats the student self-report as ground truth; no reliability or validity evidence for the single binary item is reported.
domain assumption Two experts rating each hint jointly with the four-attribute rubric provide a stable expert ground truth.
Section 3 describes the joint rating procedure but no inter-rater reliability; the expert side could be idiosyncratic.
domain assumption The five mismatch categories identified by the authors are reliable and complete for the 37 mismatch cases.
Section 4 presents the categories with no second coder or agreement statistic, and 11 of 37 cases are left unexplained.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bridging Gaps Between Student and Expert Evaluations of AI-Generated Programming Hints." pith.science (2026). https://pith.science/paper/DYXNJDGI

@misc{pith2026250903269,
  author       = {Pith},
  title        = {Pith review of: Bridging Gaps Between Student and Expert Evaluations of AI-Generated Programming Hints},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DYXNJDGI}},
  note         = {Machine review of arXiv:2509.03269}
}

read the original abstract

Generative AI has the potential to enhance education by providing personalized feedback to students at scale. Recent work has proposed techniques to improve AI-generated programming hints and has evaluated their performance based on expert-designed rubrics or student ratings. However, it remains unclear how the rubrics used to design these techniques align with students' perceived helpfulness of hints. In this paper, we systematically study the mismatches in perceived hint quality from students' and experts' perspectives based on the deployment of AI-generated hints in a Python programming course. We analyze scenarios with discrepancies between student and expert evaluations, in particular, where experts rated a hint as high-quality while the student found it unhelpful. We identify key reasons for these discrepancies and classify them into categories, such as hints not accounting for the student's main concern or not considering previous help requests. Finally, we propose and discuss preliminary results on potential methods to bridge these gaps, first by extending the expert-designed quality rubric and then by adapting the hint generation process, e.g., incorporating the student's comments or history. These efforts contribute toward scalable, personalized, and pedagogically sound AI-assisted feedback systems, which are particularly important for high-enrollment educational settings.

Figures

Figures reproduced from arXiv: 2509.03269 by the authors.

**Figure 2.** Interaction between students and our system. It is [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Data analysis workflow. After experts rate the hints, [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 28 canonical work pages

[1]

Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. 2024. STaR-GATE: Teaching Language Models to Ask Clarifying Questions. CoRR abs/2403.19154 (2024)

arXiv 2024
[2]

Imen Azaiz, Natalie Kiesler, and Sven Strickroth. 2024. Feedback-Generation for Programming Exercises With GPT-4. In Proceedings of Innovation and Technology in Computer Science Education (ITiCSE)

work page 2024
[3]

JJ Beishuizen, Els Hof, CM Van Putten, Samantha Bouwmeester, and JJ Asscher

work page
[4]

Param Biyani, Yasharth Bajpai, Arjun Radhakrishna, Gustavo Soares, and Sumit Gulwani. 2024. RUBICON: Rubric-Based Evaluation of Domain-Specific Human AI Conversations. In Proceedings of the International Conference on AI-Powered Software (AIware)

work page 2024
[5]

David Boud and Elizabeth Molloy. 2012. Feedback in Higher and Professional Education. Routledge

work page 2012
[6]

Brown et al

Tom B. Brown et al. 2020. Language Models are Few-Shot Learners. InProceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS)

work page 2020
[7]

Bhavya Chopra, Yasharth Bajpai, Param Biyani, Gustavo Soares, Arjun Radhakr- ishna, Chris Parnin, and Sumit Gulwani. 2024. Exploring Interaction Patterns for Debugging: Enhancing Conversational Capabilities of AI-assistants. CoRR abs/2402.06229 (2024)

arXiv 2024
[8]

William G. Cochran. 1952. The 𝜒 2 Test of Goodness of Fit. The Annals of Mathematical Statistics 23, 3 (1952), 315–345

work page 1952

Show all 37 references

[9]

Peter Ferguson. 2011. Student Perceptions of Quality Feedback in Teacher Edu- cation. Assessment & Evaluation in Higher Education 36 (2011)

2011
[10]

Andreas Göldi and Roman Rietsche. 2023. Insert-Expansions for Tool-Enabled Conversational Agents. CoRR abs/2307.01644 (2023)

2023 arXiv
[11]

Xueyu Hu, Kun Kuang, Jiankai Sun, Hongxia Yang, and Fei Wu. 2024. Leveraging Print Debugging to Improve Code Generation in Large Language Models. CoRR abs/2401.05319 (2024)

2024 arXiv
[12]

Anders Jonsson and Gunilla Svingby. 2007. The Use of Scoring Rubrics: Reliability, Validity and Educational Consequences. Educational Research Review 2, 2 (2007)

2007
[13]

Bailey Kimmel and et al. 2024. Enhancing Programming Error Messages in Real Time with Generative AI. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI)

2024
[14]

Paul A Kirschner and Jeroen JG van Merriënboer. 2013. Do Learners Really Know Best? Urban Legends in Education. Educational Psychologist 48 (2013)

2013
[15]

Nachiket Kotalwar, Alkis Gotovos, and Adish Singla. 2024. Hints-in-browser: Benchmarking Language Models for Programming Feedback Generation. In Annual Conference on Neural Information Processing Systems (NeurIPS)

2024
[16]

Mareike Kunter and Jürgen Baumert. 2007. Who is the Expert? Construct and Criteria Validity of Student and Teacher Ratings of Instruction. Learning Envi- ronments Research (2007)

2007
[17]

Liffiton, Brad E

Mark H. Liffiton, Brad E. Sheese, Jaromír Savelka, and Paul Denny. 2023. Code- help: Using Large Language Models with Guardrails for Scalable Support in Programming Classes. In Proceedings of the Koli Calling International Conference on Computing Education Research (Koli Calling)

2023
[18]

Ying-Chun Lin and et al. 2024. Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models. CoRR abs/2403.12388 (2024)

2024 arXiv
[19]

Dominic Lohr, Hieke Keuning, and Natalie Kiesler. 2025. You’re (Not) My Type- Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks? Journal Of Computer Assisted Learning 41 (2025)

2025
[20]

OpenAI. 2023. GPT-4 Technical Report. CoRR abs/2303.08774 (2023)

2023 arXiv
[21]

Bunichi Otaki and Oskar Lindwall. 2024. Generative AI and the Human Touch: Investigating the Changing Landscape of Feedback in Higher Education. In Proceedings of the International Conference of the Learning Sciences (ICLS)

2024
[22]

Maciej Pankiewicz and Ryan Shaun Baker. 2023. Large Language Models (GPT) for Automating Feedback on Programming Assignments. CoRR abs/2307.00150 (2023)

2023 arXiv
[23]

Tung Phung, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023. Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models. In Proceedings of the International Conference on Educational Data Min...

2023
[24]

Tung Phung and et al. 2023. Generative AI for Programming Education: Bench- marking ChatGPT, GPT-4, and Human Tutors. In Proceedings of the Conference on International Computing Education Research (ICER) - Volume 2

2023
[25]

Tung Phung and et al. 2024. Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation. In Proceedings of the International Learning Analytics and Knowledge Conference (LAK)

2024
[26]

Y Malini Reddy and Heidi Andrade. 2010. A Review of Rubric Use in Higher Education. Assessment & Evaluation in Higher Education 35 (2010)

2010
[27]

Adel Remadi, Karim El Hage, Yasmina Hobeika, and Francesca Bugiotti. 2024. To Prompt or Not to Prompt: Navigating the Use of Large Language Models for Integrating and Modeling Heterogeneous Data. Data & Knowledge Engineering 152 (2024)

2024
[28]

Lianne Roest, Hieke Keuning, and Johan Jeuring. 2024. Next-Step Hint Generation for Introductory Programming Using Large Language Models. In Proceedings of the Australasian Computing Education Conference (ACE)

2024
[29]

Adish Singla. 2023. Evaluating ChatGPT and GPT-4 for Visual Programming. In Proceedings of the Conference on International Computing Education Research (ICER) - Volume 2

2023
[30]

Lisa Gueldenzoph Snyder and Mark J Snyder. 2008. Teaching Critical Thinking and Problem Solving Skills. The Journal of Research in Business Education 50 (2008)

2008
[31]

Susanne Voelkel, Tunde Varga-Atkins, and Luciane V Mello. 2020. Students Tell Us What Good Written Feedback Looks Like. FEBS Open Bio 10 (2020)

2020
[32]

Mitchell, and Chris Piech

Sierra Wang, John C. Mitchell, and Chris Piech. 2024. A Large Scale RCT on Effective Error Messages in CS1. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE)

2024
[33]

Chi, Quoc V

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompt- ing Elicits Reasoning in Large Language Models. In Proceedings of the Annual Conference on Neural Information Processing Systems...

2022
[34]

Juliette Woodrow, Ali Malik, and Chris Piech. 2024. AI Teaches the Art of Elegant Coding: Timely, Fair, and Helpful Style Feedback in a Global Course. In Proceedings of the Technical Symposium on Computer Science Education (SIGCSE)

2024
[35]

Zamfirescu-Pereira, Laryn Qi, Bjorn Hartmann, John DeNero, and Narges Norouzi

J.D. Zamfirescu-Pereira, Laryn Qi, Bjorn Hartmann, John DeNero, and Narges Norouzi. 2023. Conversational Programming with LLM-Powered Interactive Support in an Introductory Computer Science Course. NeurIPS’23 Workshop on Generative AI for Education (GAIED) (2023)

2023
[36]

J. D. Zamfirescu-Pereira, Laryn Qi, Björn Hartmann, John DeNero, and Narges Norouzi. 2024. 61A-Bot: AI Homework Assistance in CS1 is Fast and Cheap - But is It Helpful? CoRR abs/2406.05600 (2024)

2024 arXiv
[2001]

British Journal of Educational Psychology 71 (2001)

Students’ and Teachers’ Cognitions about Good Teachers. British Journal of Educational Psychology 71 (2001)

2001

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. 2024. STaR-GATE: Teaching Language Models to Ask Clarifying Questions. CoRR abs/2403.19154 (2024)

arXiv 2024

[2] [2]

Imen Azaiz, Natalie Kiesler, and Sven Strickroth. 2024. Feedback-Generation for Programming Exercises With GPT-4. In Proceedings of Innovation and Technology in Computer Science Education (ITiCSE)

work page 2024

[3] [3]

JJ Beishuizen, Els Hof, CM Van Putten, Samantha Bouwmeester, and JJ Asscher

work page

[4] [4]

Param Biyani, Yasharth Bajpai, Arjun Radhakrishna, Gustavo Soares, and Sumit Gulwani. 2024. RUBICON: Rubric-Based Evaluation of Domain-Specific Human AI Conversations. In Proceedings of the International Conference on AI-Powered Software (AIware)

work page 2024

[5] [5]

David Boud and Elizabeth Molloy. 2012. Feedback in Higher and Professional Education. Routledge

work page 2012

[6] [6]

Brown et al

Tom B. Brown et al. 2020. Language Models are Few-Shot Learners. InProceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS)

work page 2020

[7] [7]

Bhavya Chopra, Yasharth Bajpai, Param Biyani, Gustavo Soares, Arjun Radhakr- ishna, Chris Parnin, and Sumit Gulwani. 2024. Exploring Interaction Patterns for Debugging: Enhancing Conversational Capabilities of AI-assistants. CoRR abs/2402.06229 (2024)

arXiv 2024

[8] [8]

William G. Cochran. 1952. The 𝜒 2 Test of Goodness of Fit. The Annals of Mathematical Statistics 23, 3 (1952), 315–345

work page 1952