{"id":"b22faca1-0102-4312-b33e-c5168cd80a5d","arxiv_id":"2411.18151","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"With a carefully designed multi-stage prompt, GPT-4 generated next-step hints that Java educators ranked higher than hints written by experienced human educators.","lead":"This study asked 44 Java educators to judge hints for stuck beginner programmers, generated by four large language models with five different prompts and by five human experts. The best GPT-4 prompt produced hints rated better than expert human hints, with hint length and reading level the strongest predictors of quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline GPT-4-versus-human comparison lacks a significance test and may be driven by one snapshot or by hint length/readability; the claim needs a statistical reanalysis.","rationale":"The reader's weakest assumption focused on the representativeness of the four snapshots and on educator ranking as a proxy for student learning. I agree that the small number of snapshots is a real limitation, and the paper itself flags it. My stress-test concentrates on a more immediate and testable gap: the paper's flagship comparison of GPT-4 against human experts is never subjected to any inferential statistical test. The data are open, so this is directly checkable. If a reanalysis shows no significant difference or shows that the difference disappears when controlling for word count and reading level, the central claim as stated in the abstract and conclusion would not be supported. This does not make the study worthless: the methodology, the snapshot-selection process, the comparative-judgement instrument, and the hint-characteristic findings are valuable and reproducible. It does mean the conclusion should be framed conditionally, which is exactly the reader's current verdict. I therefore recommend no change to the CONDITIONAL verdict, while adding a precise statistical robustness check as the condition for upgrading to ACCEPT.","tokens_in":29399,"tokens_out":3869,"duration_ms":37829,"concrete_test":"Using the open OSF data, fit a mixed-effects model to RankScore with generator type (GPT-4 vs Human) as a fixed effect and Snapshot as a random effect, using the 20 GPT-4 and 20 human rank scores, and report the effect size with a bootstrap confidence interval. Then repeat the analysis leaving out each Snapshot in turn, and again with WordCount and Flesch-Kincaid reading level as covariates. If the GPT-4 advantage is not significant at p<0.05, disappears when any single Snapshot is removed, or is fully mediated by word count and reading level, the conclusion should be weakened to 'GPT-4 produced hints with characteristics that educators preferred in this sample.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GPT-4 outperformed human experts rests on the raw mean RankScore difference shown in Figure 8 and discussed in Section 4.10, but the paper never reports a significance test, confidence interval, or effect size for that difference. With only four snapshots and ranks that are zero-sum within each snapshot, a single snapshot can move the GPT-4 mean by several rank positions. The paper's own caution in Section 4.10 that per-Generator results 'may be interpreting noise' applies at least as strongly to the headline comparison, since the human benchmark consists of five author-generated hints per snapshot. Moreover, the random-forest analysis in Table 5 gives Model an importance of only 3.0, compared with 23.9 for WordCount and 17.9 for Flesch-Kincaid grade level, so the apparent GPT-4 advantage may be a proxy for producing hints in the 80-160-word, grade-9-or-below sweet spot rather than an intrinsic model superiority. Without an inferential test and a sensitivity analysis, 'GPT-4 outperformed human experts' is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates automatically generated next-step programming hints for novices by having experienced Java educators rank hints via adaptive comparative judgement. Hints were produced by four LLMs (Mixtral-8x7B, Gemini, GPT-3.5, GPT-4) using five prompts (two multi-stage), plus five human expert hints, for four real 'stuck' student code snapshots drawn from Blackbox. The authors report three main findings: (1) GPT-4 with a well-designed prompt outperformed the human expert hints; (2) prompt choice matters as much as model choice, with multi-stage prompt 3 best; and (3) hint quality is most strongly associated with word count (80-160 words ideal) and reading level (US grade 9 or below), while offering alternative approaches is rated negatively. The study is pre-registered, with open data and analysis scripts, and includes several checks on judging validity and inter-rater reliability.","tokens_in":29563,"tokens_out":3607,"duration_ms":35134,"significance":"If the central claim holds, the result is practically significant: a state-of-the-art LLM with an expert-designed prompt could produce next-step hints that experienced educators rank above hints written by experienced human educators, even without knowing the original task. The paper also offers a methodological contribution by demonstrating comparative judgement as a viable tool for computing-education research, and the open data and reproducible pipeline are exemplary. However, the headline GPT-4-versus-human comparison rests on only four rank observations per generator and lacks an inferential test, and the random-forest analysis is descriptive and sensitive to correlated predictors; these issues currently prevent the paper from fully establishing its strongest conclusion.","major_comments":[{"comment":"The central claim that GPT-4 outperformed human experts is based on the raw mean RankScore difference in Figure 8, but no significance test, confidence interval, or effect size is reported for that comparison. With only four data points per generator and ranks that are zero-sum within each snapshot, a single snapshot could move the GPT-4 mean by several rank positions. The paper's own caution in Section 4.10 that per-generator results 'may be interpreting noise' applies directly to this comparison. I recommend a reanalysis: a permutation test or a mixed-effects model with Snapshot as a random effect and Generator (or Model) as a fixed effect, plus a sensitivity analysis leaving out one snapshot at a time. Unless such an analysis is provided, the conclusion in Sections 6.2 and 9 should be weakened to a descriptive statement about this sample.","section":"§4.10, Fig. 8; §6.2; §9"},{"comment":"The random-forest importance values in Table 5 are used to conclude that word count and reading level are the most important hint characteristics, and that Model is relatively unimportant (importance 3.0 vs 23.9 and 17.9). Because WordCount and Flesch-Kincaid grade level are correlated with the model (Mixtral produced longer, less readable hints), the low Model importance may simply reflect that the model effect is mediated by these text properties; it does not establish that the GPT-4 advantage is fully explained by length and readability. The analysis would be more convincing with a correlation matrix, a model omitting WordCount and FleschKincaidGradeLevel to see how Model importance changes, and cross-validated or uncertainty-aware estimates of importance rather than point values from a single forest fit on 100 hints.","section":"§4.11, Table 5"},{"comment":"The human benchmark consists of hints written by the five researchers who also designed the prompts, selected the snapshots, and wrote the notes given to judges. This creates a potential conflict: the human hints may be at a disadvantage because the task is framed in the researchers' own terms, and the comparison is not against an independent sample of expert educators. The paper should explicitly acknowledge this limitation and ideally include some external human hints (or justify why the authors' hints are a fair benchmark). Without this, the claim of beating 'human experts' is narrower than stated.","section":"§3.3 and §4.3"},{"comment":"The inter-rater reliability for Snapshot 4 is reported as 0.60, below the 0.7 threshold the paper cites from the comparative-judgement literature. Since each generator contributes exactly one hint per snapshot, a low-reliability snapshot injects substantial noise into every generator's aggregate score. The paper should report the reliability per snapshot alongside the generator means, and discuss how the ranking for Snapshot 4 in particular might affect the GPT-4-versus-humans comparison.","section":"§4.5"}],"minor_comments":[{"comment":"The word 'Artifical' in the Introduction should be 'Artificial'.","section":"§1"},{"comment":"The phrase 'reminisicent' should be 'reminiscent'.","section":"§6.5"},{"comment":"The 'Model' attribute is a generation mechanism, not a hint characteristic; including it in the same table as WordCount and Sentiment conflates two different research questions. Consider presenting the model/prompt analysis separately from the hint-characteristic analysis, or clearly stating that Model is used only as a covariate.","section":"§4.7, Table 2"},{"comment":"The chi-squared test for left/right bias is reported as p=0.06; since the result is borderline, it would be helpful to report the effect size or the exact test statistic with degrees of freedom so that readers can judge the strength of the check.","section":"§4.5"},{"comment":"The row showing a single '0' with no check marks is confusing; a footnote or reformatting would clarify that it represents hints with none of the four feedback-literacy concepts present.","section":"Table 4"},{"comment":"The reference to 'San Verhavert and Maeyer' is incomplete in the author list; also reference [38] appears to cite Kolen and Brennan's book on test equating for the NoMoreMarking platform, which is likely an incorrect citation.","section":"§3.4 and References"},{"comment":"The violin plots with overlaid dots make individual data points difficult to distinguish when multiple hints share the same rank; consider adding horizontal jitter or a summary table of means and standard deviations.","section":"Figures 8-10"}],"recommendation":"major_revision","confidential_remarks":"The paper has several strengths: pre-registration, open data, thoughtful validity checks for the judging process, and a clear methodological contribution in applying comparative judgement to hint evaluation. The main concern is that the headline claim is not yet statistically supported; a reanalysis with appropriate inferential methods should be feasible within the manuscript's scope. I would also encourage the authors to consider whether the human benchmark being self-generated (by the same researchers who designed prompts and selected snapshots) should be more prominently caveated, since it directly bears on the 'beats human experts' framing. The paper is otherwise well within the scope of a computing-education venue and likely to be a useful contribution after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, it does something genuinely useful: it applies comparative judgement at a larger scale than prior hint-evaluation work, with pre-registration, open data, and multiple LLMs and prompts, and it gives concrete, actionable findings about hint length and reading level. Second, the headline claim—that GPT-4 with a well-designed prompt beats experienced human educators at writing next-step hints—rests on weaker evidence than the prose suggests. There is no significance test, only four snapshots, and the random-forest analysis gives Model an importance of 3.0 versus 23.9 for word count. The paper's own Section 4.10 warns that per-generator results 'may be interpreting noise'; that warning applies straight to the central comparison.\n\nWhat is new is the combination: multiple models, multiple prompts, a human-expert benchmark, and comparative judgement in one study. The finding that 80–160 words and reading grade 9 or below are the strongest predictors of educator preference is a concrete, potentially actionable contribution. The multi-stage prompt that first infers the task and then generates a hint is a useful technique worth copying. The paper is transparent: pre-registered, open data, full processing pipeline, and honest limitations sections, including the missing student perspective and the low completion rate.\n\nWhere it is soft: the GPT-4-vs-humans claim is the weakest link. Four snapshots is a tiny sample; a single snapshot can move the mean by several rank positions. The human benchmark is only five author-generated hints, so 'human experts' is a low bar. The random forest is fit on 100 hints with correlated predictors and is descriptive, not inferential. The fourth snapshot's reliability (0.60) is below the 0.7 threshold. None of this kills the paper, but it means the abstract's 'GPT-4 outperformed human experts' is not established; it is a suggestive finding that needs replication and a proper statistical treatment—at minimum a permutation test or mixed-effects model across snapshots.\n\nWho is this for: people building LLM-based hinting tools in programming education, and methods-oriented researchers interested in comparative judgement. It deserves a serious referee rather than a desk reject; a revise-and-resubmit with a request for a reanalysis of the main comparison, and either more snapshots or a softened claim, would be appropriate. I would cite the hint-characteristics findings, but I would not cite the GPT-4-beats-humans claim as established.","headline":"Useful, open, well-conducted study of hint quality, but the bold GPT-4-beats-humans claim needs inferential support and replication before it is taken as established.","tokens_in":30105,"tokens_out":3363,"would_cite":true,"duration_ms":28084,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-4 tops expert educators at writing next-step code hints","keywords":["LLMs","AI","Java","next-step hints","comparative judgement","programming education","GPT-4","prompt engineering"],"falsifier":"A larger comparative-judgement replication using more snapshots, or a randomised classroom study assigning stuck students to GPT-4 hints versus human expert hints and measuring subsequent progress and learning, would settle whether the rank advantage is real; if GPT-4's mean rank advantage disappears or student outcomes show no benefit, the paper's core conclusion collapses.","tokens_in":29199,"feed_emoji":"🤖","tokens_out":7849,"duration_ms":67326,"temperature":0.7,"pith_summary":"This paper asks whether a large language model can generate the next-step hint that a stuck novice programmer needs, and what makes such a hint good. The authors took real snapshots of broken Java code from the Blackbox dataset, gave four LLMs and five experienced educators only that code (no statement of the assignment), and collected 25 candidate hints per snapshot. A panel of 41 experienced Java educators ranked the hints in blind pairwise comparisons. The paper's central claim is that GPT-4, used with a carefully designed multi-stage prompt, produced hints ranked above those written by the human educators. The paper also reports that educators consistently preferred hints of 80–160 words at a low reading level, valued guidance beyond a direct fix, disliked alternative solution paths, and showed no preference over sentiment.","feed_headline":"GPT-4 tops expert educators at writing next-step code hints","feed_subtitle":"Experienced Java educators ranked GPT-4's hints above humans' even when both saw only student code.","key_machinery":"The machinery is comparative judgement, a ranking method in which judges repeatedly choose the better of two hints and an adaptive algorithm turns those pairwise choices into a full ordering, avoiding the inconsistency of absolute rating scales. The supporting machinery consists of the five carefully engineered prompts (especially prompt 3, a multi-stage prompt that first infers the student's task and then writes the hint), the four stuck-student code snapshots used as scenarios, and a random-forest analysis that identifies which hint attributes predict high rank. The random forest is what surfaces the two dominant factors, word count and reading grade level, and separates those from the weaker pedagogical and stylistic factors.","core_discovery":"The paper's central claim is that a state-of-the-art LLM with a well-designed prompt can outperform experienced human educators at the specific task of writing a one-shot next-step hint for a stuck novice programmer. The comparison was deliberately demanding: generators saw only the student's code, with no problem description, so the task had to be inferred. Judged by a separate panel of experienced Java educators using comparative judgement, GPT-4 obtained the highest mean rank, ahead of the five human hints, while GPT-3.5, Gemini, and Mixtral-8x7B did not beat the humans. The best prompt was a two-stage one that first asked the model to state what task the student appeared to be working on, then fed that inference back to generate the hint. The authors conclude that automatic hint generation is immediately viable, with the caveat that the right model and prompt are both required.","pith_inferences":["If the educator rankings reflect what actually helps novices, the main cost of high-quality hinting shifts from writing hints to maintaining a validated prompt layer as models change — an inference this study does not itself test.","The null effects for sentiment and for explaining general principles suggest that a stuck novice's immediate need is concrete, short direction; whether that holds for older learners or for non-Java languages is an untested extension.","A direct classroom comparison — students randomly assigned GPT-4 hints versus human hints, with progress and learning measured — would tell whether the ranking advantage is a real pedagogical gain; the paper does not run that experiment.","Because the authors found large variation across prompts and models but little variation across their five human hint-writers, a natural follow-up is to treat prompt design itself as the main engineering target for hint quality."],"forward_implications":["Tool-builders can treat expert-validated LLM hints as immediately usable in novice programming environments, as long as the prompt is supplied by the tool rather than composed by the student.","Hint generators should target 80–160 words and a reading level of US grade 9 or below; hints outside that band lose several rank positions.","A multi-stage prompt that first infers the task from the code and then produces the hint is the most reliable way to generate good hints with current LLMs.","Educators and hint authors should avoid offering alternative solution approaches in a next-step hint, since judges consistently penalised them.","The comparative-judgement protocol itself provides a reusable benchmark for evaluating future LLM versions and prompt designs without needing absolute grading scales."],"supporting_citations":[{"why":"Supplies the real stuck-student code snapshots from which all hints were generated.","marker":"[7]"},{"why":"Introduces adaptive comparative judgement, the ranking method that turns pairwise educator choices into a hint ordering.","marker":"[61]"},{"why":"Meta-analysis establishing the reliability of comparative judgement with expert judges, justifying the study's judge design.","marker":"[73]"},{"why":"Provides the online comparative-judgement platform used to collect the pairwise decisions.","marker":"[38]"},{"why":"Supplies the feedback-literacy categories (telling, guiding, developing understanding, opening up a perspective) used to characterise the hints.","marker":"[11]"},{"why":"Defines the reading grade-level formula used to measure hint readability.","marker":"[37]"},{"why":"Motivates the choice of hint length and readability as the key hint characteristics to measure.","marker":"[15]"}],"fun_headline_variants":["GPT-4 beats human teachers at coding hints for stuck novices","Study: GPT-4 writes better Java hints than expert educators","LLM with smart prompt outranks human experts in hint quality","Comparative judgement: GPT-4 wins hint-writing contest vs humans","Two-stage prompt lets GPT-4 out-teach human Java tutors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that four snapshots of stuck student code, ranked by experienced educators, accurately represent whether a hint helps a real novice learn; the paper itself warns that four data points per generator may be noise and that educator preference may not match what students find useful.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4 beats human teachers at coding hints for stuck novices","Study: GPT-4 writes better Java hints than expert educators","LLM with smart prompt outranks human experts in hint quality","Comparative judgement: GPT-4 wins hint-writing contest vs humans","Two-stage prompt lets GPT-4 out-teach human Java tutors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000143,"raw_usage":{"total_tokens":1228,"prompt_tokens":1055,"completion_tokens":173,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":84}},"tokens_in":671,"tokens_out":173,"duration_ms":2363,"temperature":1.0,"reasoning_tokens":84,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:27:15.906761+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A larger comparative-judgement replication using more snapshots, or a randomised classroom study assigning stuck students to GPT-4 hints versus human expert hints and measuring subsequent progress and learning, would settle whether the rank advantage is real; if GPT-4's mean rank advantage disappears or student outcomes show no benefit, the paper's core conclusion collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces adaptive comparative judgement, the ranking method that turns pairwise educator choices into a hint ordering."},{"cited_title":"Price, Samiha Marwan, Michael Winters, and Joseph Jay Williams","cited_arxiv_id":null,"evidence_quote":"Meta-analysis establishing the reliability of comparative judgement with expert judges, justifying the study's judge design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the online comparative-judgement platform used to collect the pairwise decisions."},{"cited_title":"Becker, Catherine Mooney, John Homer, Zachary C Albrecht, and Garrett B","cited_arxiv_id":null,"evidence_quote":"Motivates the choice of hint length and readability as the key hint characteristics to measure."}],"review_version":1}