REVIEW 3 major objections 5 minor 38 references
Narrowing the Gap: Supervised Fine-Tuning of Open-Source LLMs as a Viable Alternative to Proprietary Models for Pedagogical Tools
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Supervised fine-tuning on 40,000 GPT-4.1-generated C error explanations brings a 4-billion-parameter open-source model to within 0.10 of GPT-4.1 on every expert-judged rubric metric, and all fine-tuned models beat the deployed DCC Help…
desk verdict Solid empirical study of SFT for C error explanations, but the 'within 0.10 of GPT-4.1' claim rests on weak-rubric metrics and no uncertainty intervals; the core improvement claim is credible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is distillation via supervised fine-tuning combined with a structured prompting strategy and dual evaluation. A proprietary model (GPT-4.1) was prompted with a fixed three-part template — error-message clarification, potential causes, and guided hints — to produce 40,000 explanations from real student compile- and run-time errors, and these outputs served as training targets for Qwen3-4B, Llama-3.1-8B, and Qwen3-32B using QLoRA, a parameter-efficient fine-tuning method that trains small adapters on a quantized base model. Quality was assessed with an eight-criterion binary rubric (correctness, selectivity, completeness, clarity, novice-appropriateness, no-solution, no-overhelp, socratic) applied by four human experts and scaled by a strict-unanimity ensemble of three LLM judges (GPT-4.1, Gemini-2.5-Flash, Qwen3-32B), with the AC1 inter-rater reliability coefficient measuring agreement. This design lets the authors separate the effect of fine-tuning from model size and exposes where automated judging departs from human judgment.
What would settle it
Re-run the large-scale evaluation with a judge panel that excludes GPT-4.1 and Qwen-32B, or with human annotation on a fresh sample of a few hundred responses; if SFT-Qwen-4B's gap to GPT-4.1 on Completeness and Socratic widens beyond the claimed margins, or expert-LLM AC1 stays near zero, the large-N model comparisons for those criteria are unreliable.
Extended reading notes
Core claim
The paper's central claim is that SFT improves the capability of smaller open-source language models to generate C programming error explanations for CS1/2 students, narrowing the gap to proprietary frontier models. On a 100-example expert ranking, GPT-4.1 had the best mean rank (3.09), followed by SFT-Qwen-32B and base Qwen-32B (3.60 and 3.62), then SFT-Qwen-4B (4.12) and SFT-Llama-8B (4.27), with all fine-tuned models clearly preferred over their base counterparts and over the DCC Help baseline. On the expert-judged rubric, SFT-Qwen-4B stayed within 0.10 of GPT-4.1 on every criterion, while all three fine-tuned models surpassed the DCC Help baseline on almost all metrics, the exception being LLM-judged Completeness, which the authors attribute to weak expert-LLM agreement (AC1 = 0.05) rather than a real shortfall. The paper also reports that fine-tuning gave little or no benefit to Qwen-32B, whose base version already matched GPT-4.1 on several criteria, suggesting a ceiling for SFT on larger models.
Load-bearing premise
The large-scale comparisons rest on the assumption that the strict-unanimity three-judge LLM ensemble produces valid rubric labels; expert-LLM agreement is weak for Completeness (AC1 = 0.05) and Socratic (AC1 = 0.08), and GPT-4.1 and Qwen-32B appear on both sides as candidates and judges.
Editorial extensions
If this is right
- A 4B-parameter fine-tuned model can serve as a drop-in replacement for a frontier API in this task, with a mean-rank gap of 1.03 to GPT-4.1 and expert-rated quality within 0.10 on every rubric criterion.
- All three fine-tuned models beat the deployed DCC Help baseline, so the training and evaluation pipeline can be adopted directly to replace an existing LLM-based error-explanation service without sacrificing quality.
- The gains are largest for small models: Llama-3.1-8B and Qwen3-4B improved substantially after SFT, while Qwen3-32B did not, implying that most of the benefit comes from specialisation rather than scale.
- Because Qwen-32B (base or SFT) nearly matches GPT-4.1, institutions could generate their own training data with an open-source model, keeping student code on-premises throughout.
- The eight-criterion rubric and the dual expert/LLM evaluation provide a replicable protocol for comparing error-explanation quality across models, which later work can reuse on other languages and course levels.
Reading between the lines
- If style mimicry drives part of the SFT gains, then fine-tuning on outputs from a different teacher model, or a mix of teachers, would test whether the quality gains are pedagogical or stylistic; the paper's own self-bias caveat makes this a natural next experiment.
- The weak expert-LLM agreement on Completeness and Socratic suggests that the large-N rankings on those two criteria are not yet validated; a judge design that trains the ensemble on expert examples, or drops those criteria from automated scoring, would be a conservative alternative.
- The same distillation recipe could plausibly transfer to other programming languages, hint generation, or code explanation tasks, where a frontier model supplies training targets once and a small fine-tuned model then runs locally.
- For field deployment, the 4B model's performance raises the possibility of fully on-device error explanation inside a compiler or editor, eliminating both network round-trips and per-request API costs; the paper notes this as future work but does not test it.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether supervised fine-tuning (SFT) of open-source LLMs on a large dataset of C compiler error explanations can make them competitive with proprietary models for pedagogical error-message generation. The authors construct a 40,000-example training set by prompting GPT-4.1 with student errors from the DCC Help system, fine-tune Qwen3-4B, Llama-3.1-8B, and Qwen3-32B, and evaluate them against GPT-4.1, base models, and the deployed DCC Help baseline. Evaluation combines expert ratings and rankings on 100 examples with an LLM-as-judge ensemble applied to 8,000 examples (64,000 responses). The central claim is that SFT substantially improves small open-source models, with SFT-Qwen-4B reported as within 0.10 of GPT-4.1 on all expert-judged rubric metrics and all fine-tuned models surpassing the DCC Help baseline. The paper also contributes an eight-criterion rubric and a replicable data-generation and evaluation methodology.
Significance. If the claims hold, the work has clear practical significance for computing education: it would demonstrate that small, locally deployable models can provide compiler-error explanations of near-frontier quality, addressing privacy, cost, and operational concerns. The dataset of 40,000 real student errors and the dual expert/LLM evaluation protocol are valuable community resources, and the authors are commendably explicit about limitations, including self-bias in the LLM judge panel and low inter-rater reliability on several rubric criteria. The qualitative direction of the findings, especially the expert ranking evidence that SFT improves smaller models over their base counterparts, is credible and useful. However, the quantitative precision of the headline 'within 0.10' and 'surpasses baseline' claims is not yet supported by the reported statistics, and the large-scale automated evaluation rests on metrics with very weak expert-LLM agreement.
major comments (3)
- [§4.2 and §5, Table 3] The claim that SFT-Qwen-4B performs 'within a margin of 0.10 in all expert-judged metrics' is not statistically grounded. The expert rates in Table 3 are proportions based on roughly 100 examples per model, but no confidence intervals or significance tests are reported for any rubric rate. This matters most for the metrics with low expert inter-rater reliability: Clarity has Gwet's AC1 = 0.29 and Socratic has AC1 = -0.12, yet the paper highlights parity (0.65 vs 0.65 on Socratic and 0.76 vs 0.75 on Clarity) as evidence of competitive performance. The mean-rank confidence intervals in Table 2 also overlap substantially between GPT-4.1 (2.68-3.50) and SFT-Qwen-4B (3.69-4.55). The authors should either report uncertainty intervals and formal comparison tests for the rubric rates or soften the 'within 0.10 on all metrics' formulation.
- [§3.7 and §5.1, Table 3] The large-scale LLM-as-judge evaluation is not a valid basis for comparisons on Completeness and Socratic metrics, because expert-LLM agreement is very weak (AC1 = 0.05 and 0.08, respectively), and the paper itself acknowledges that self-bias persisted for GPT-4.1 and base Qwen-32B despite the strict-unanimity ensemble. Since GPT-4.1 generated the SFT training labels and also served as a judge, and Qwen-32B appeared as both candidate and judge, the LLM-judged columns in Table 3 cannot be treated as objective measurements for those two metrics. This directly affects the conclusion that 'all SFT models surpass DCC Help except LLM-judged Completeness,' because the exception is dismissed as a methodological artifact rather than supported by a validity check. The authors should either restrict automated comparisons to metrics with acceptable expert-LLM agreement, or demonstrate that the unanimity policy removes self-bias through a dedicated analysis (e.g., judge-substitution experiments).
- [§3.6, Tables 2-3] The expert evaluation design confounds model quality with annotator identity. Four experts each rated 20 unique examples plus a shared set of 20, so the aggregate expert rates for each model are pooled across raters who mostly saw disjoint examples; the paper states that for the shared subset only one random annotation per example was included. With n = 100 examples per model spread across four raters of potentially different severity, observed differences in rubric rates or mean ranks could be driven by rater effects rather than model performance. The manuscript should report per-rater rates, test for model-by-rater interactions, or use a rater-aware model (e.g., mixed-effects or adjustment for rater severity). Without this, the expert-judged comparisons in Table 3 and the 'within 0.10' claim remain vulnerable to an alternative explanation.
minor comments (5)
- [§3.7] The number of model responses in the LLM-judged evaluation is inconsistent: §3.4 reports 8,000 examples times eight models = 64,000 responses, and Table 3 says 'LLM-judged annotations (64,000 model responses)', but §3.7 says the expanded evaluation dataset comprises '32,000 model-generated responses.' Please correct the discrepancy or clarify which subset was judged.
- [§3.6] The annotation count is described confusingly: 'This brings the number of annotations per expert to 40 examples or 320 responses' while Table 3 refers to 'expert annotations (800 model responses).' Clarify whether 800 refers to the number of unique example-model pairs rated by at least one expert, and how the 320 figure relates to it.
- [§5.2] The Future Work section refers to 'Direct Preference Optimisation (DFO)'; the standard acronym is DPO (Direct Preference Optimization).
- [Table 3] The 'all' columns are not defined in the table caption or the rubric; the paper should state explicitly that these are the proportion of responses satisfying all eight criteria simultaneously, and note the low base rates that make such combined scores difficult to interpret.
- [References] Reference [26] has a formatting error in the author list ('Brett A 2024 Becker'), and several references use inconsistent capitalization and punctuation that should be normalized to the venue style.
Circularity Check
Automated LLM-judged comparison is partially circular because GPT-4.1 (the SFT teacher) and Qwen-32B (a candidate) served as judges; the central SFT-improvement claim still has independent expert support.
-
other
[Section 5.1 (Limitations), reflecting Section 3.2 dataset generation and Section 3.7 judge ensemble]
"Our SFT training dataset was generated using GPT-4.1, therefore our results are inherently bounded by the quality, style and biases of GPT-4.1. This also impacts the interpretability of LLM-judged results which used GPT-4.1 as part of the judge ensemble. The SFT process optimises smaller models to mimic the style and structure of GPT-4.1, consequently using GPT-4.1 as a judge may have favoured responses similar to its own output rather than rewarding pedagogical quality."
The SFT training labels were produced by gpt-4.1-2025-04-14 (Section 3.2), and GPT-4.1 is also one of the three judges in the automated evaluation (Section 3.7). Thus the large-scale LLM-judged quality scores are partly measuring how closely a candidate response matches the teacher model's own output, rather than measuring independent pedagogical quality. The paper explicitly acknowledges this self-bias, meaning the 64,000-response automated comparisons between GPT-4.1 and the SFT models are not fully independent evidence. The expert evaluation is separate but is limited to 100 examples, so it cannot fully repair the circularity of the automated scale-up.
-
other
[Section 5.1 (Limitations), with judge composition in Section 3.7]
"Similarly, as Qwen-32B and GPT-4.1 were represented in the judging ensemble and as candidate models, the LLM-judged evaluation is vulnerable to self-bias. While we sought to mitigate this by adding a third impartial model (Gemini-2.5-Flash) to the ensemble, and enforcing a unanimous-verdict policy, the markedly higher LLM-judged performance for GPT-4.1 and base Qwen-32B suggests that self-bias persisted."
Candidate models appear on the judging panel, and the strict-unanimity policy means a candidate's self-favorable rating cannot be overridden by the other judges. The paper's own observation that GPT-4.1 and base Qwen-32B receive markedly higher LLM-judged performance is consistent with the self-bias being load-bearing in the automated comparisons. Because the same models are both evaluated and evaluators, the automated rankings are partially self-referential by construction, although the paper does disclose this limitation.
full rationale
The paper's central claim, that SFT improves smaller open-source models for generating C error explanations, is substantially supported by independent expert data: win rates against base models, mean-rank scores, and expert rubric annotations are all human-generated and do not depend on GPT-4.1 as a judge. The automated LLM-as-judge evaluation, however, is partially circular because the SFT training labels were generated by GPT-4.1 and GPT-4.1 plus Qwen-32B, both candidate models, also served as judges. The authors acknowledge this explicitly in Section 5.1, including the observation that self-bias appears to have persisted. A separate reliability concern, not itself circularity, is that the expert evaluation uses only 100 examples and shows weak or negative inter-rater reliability on Socratic (AC1 = -0.12) and Clarity (AC1 = 0.20), which weakens the precision of the 'within 0.10 on all expert-judged metrics' formulation. No load-bearing uniqueness theorems or ansatz-smuggling via self-citation were found; citations to prior work support the prompting strategy and judge paradigm but do not force the empirical outcome. Overall, the derivation chain is only partially circular, leaving the main SFT-improvement conclusion with independent content, so a score of 4 is appropriate.
Assumptions & free parameters
free parameters (4)
- Learning rate and number of epochs =
2e-5, 1 epoch
- Per-week sampling caps for training data =
4,500 compile / 2,250 run-time per week; 40,000 total
- Per-week sampling caps for evaluation data =
3,000 compile / 1,500 run-time per week; 8,000 total
- Strict unanimity policy for LLM judge ensemble =
Unanimous verdict across 3 judges
assumptions (5)
- domain assumption The 8-criterion rubric is a valid measure of pedagogical quality for error explanations.
- domain assumption GPT-4.1-generated explanations are high-quality targets for distillation.
- domain assumption The LLM-as-judge ensemble labels the 8,000-example set reliably.
- domain assumption The DCC Help logs from one Australian university represent typical CS1/2 C errors.
- domain assumption Later teaching periods are a valid held-out set relative to training periods.
Cite this review
Pith. "Pith review of Narrowing the Gap: Supervised Fine-Tuning of Open-Source LLMs as a Viable Alternative to Proprietary Models for Pedagogical Tools." pith.science (2026). https://pith.science/paper/UC67XJGI
@misc{pith2026250705305,
author = {Pith},
title = {Pith review of: Narrowing the Gap: Supervised Fine-Tuning of Open-Source LLMs as a Viable Alternative to Proprietary Models for Pedagogical Tools},
year = {2026},
howpublished = {\url{https://pith.science/paper/UC67XJGI}},
note = {Machine review of arXiv:2507.05305}
}
read the original abstract
Frontier Large language models (LLMs) like ChatGPT and Gemini can decipher cryptic compiler errors for novice programmers, but their computational scale, cost, and tendency to over-assist make them problematic for widespread pedagogical adoption. This work demonstrates that smaller, specialised language models, enhanced via Supervised Fine-Tuning (SFT), present a more viable alternative for educational tools. We utilise a new dataset of 40,000 C compiler error explanations, derived from real introductory programming (CS1/2) student-generated programming errors, which we used to fine-tune three open-source models: Qwen3-4B, Llama-3.1-8B, and Qwen3-32B. We performed a dual evaluation, combining expert human reviews with a large-scale automated analysis of 8,000 responses using a validated LLM-as-judge ensemble. Our results show that SFT significantly boosts the pedagogical quality of smaller models, achieving performance comparable to much larger models. We analyse the trade-offs between model size and quality, confirming that fine-tuning compact, efficient models on high-quality, domain-specific data is a potent strategy for creating specialised models to drive educational tools. We provide a replicable methodology to foster broader access to generative AI capabilities in educational contexts.
Figures
Reference graph
Works this paper leans on
-
[1]
Nischal Ashok K. and Andrew Lan. 2024. Improving Socratic Question Generation using Data Augmentation and Preference Optimization. InProceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024). Association for Computational Linguistics, Mexico City, Mexico, 108–118
work page 2024
-
[2]
Paul Denny, James Prather, and Brett A. Becker. 2021. On designing programming error messages for novices: Readability and its constituent factors. In Conference on Human Factors in Computing Systems - Proceedings. Association for Computing Machinery, 1–15. doi:10.1145/3411764.3445696
arXiv 2021
-
[3]
Becker, James Finnie-Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N
Paul Denny, James Prather, Brett A. Becker, James Finnie-Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N. Reeves, Eddie Antonio Santos, and Sami Sarsa. 2024. Computing Education in the Era of Generative AI.Commun. ACM 67, 2 (1 2024), 56–67. doi:10.1145/3624720
doi:10.1145/3624720 2024
-
[4]
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLORA: Efficient Finetuning of Quantized LLMs. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, 441
work page 2023
-
[5]
Kilem Li Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. Brit. J. Math. Statist. Psych. 61, 1 (5 2008), 29–48. doi:10.1348/000711006X126600
-
[6]
Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva. 2023. Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests. In ICER 2023 - Proceedings of the 2023 Narrowing the Gap: Supervised Fine-Tuning of Open-Source LLMs as a Viable Alternative to Proprietary Models for Pedagogical Tools ar...
arXiv 2023
-
[7]
Yann Hicke, Anmol Agarwal, Qianou Ma, and Paul Denny. 2023. AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs. arXiv:2311.02775 [cs.LG]
arXiv 2023
-
[8]
Lan Huang. 2023. Ethics of Artificial Intelligence in Education: Student Privacy and Data Protection. Science Insights Education Frontiers 16, 2 (6 2023), 2577–2587. doi:10.15354/sief.23.re202
Show all 38 references
-
[10]
Nachiket Kotalwar, Alkis Gotovos, and Adish Singla. 2024. Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation. 38th Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks. (2024)
2024
-
[11]
Charles Koutcheme, Nicola Dainese, Sami Sarsa, Arto Hellas, Juho Leinonen, Syed Ashraf, and Paul Denny. 2025. Evaluating Language Models for Generating and Judging Programming Feedback. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 (Pitt...
2025
-
[12]
Charles Koutcheme, Nicola Dainese, Sami Sarsa, Arto Hellas, Juho Leinonen, and Paul Denny. 2024. Open Source Language Models Can Provide Feedback: Evalu- ating LLMs’ Ability to Help Students Using GPT-4-As-A-Judge. InProceedings of the 2024 Innovation and Technology in Compute...
2024
-
[13]
Juho Leinonen, Paul Denny, Stephen Macneil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. In Annual Conference on Innovation and Technology in Computer Science Education, I...
2023
-
[14]
Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker. 2023. Using Large Language Models to Enhance Program- ming Error Messages. In SIGCSE 2023 - Proceedings of the 54th ACM Technical Symposium on Computer Science Education , Vol...
2023
-
[15]
Mark Liffiton, Brad Sheese, Jaromir Savelka, and Paul Denny. 2023. CodeHelp: Us- ing Large Language Models with Guardrails for Scalable Support in Programming Classes. In 23rd Koli Calling International Conference on Computing Education Re- search. Association for Computing Ma...
2023
-
[16]
Rongxin Liu, Julianna Zhao, Benjamin Xu, Christopher Perez, Yuliia Zhukovets, and David J. Malan. 2025. Improving AI in CS50 Leveraging Human Feedback for Better Learning. In SIGCSE TS 2025 - Proceedings of the 56th ACM Technical Symposium on Computer Science Education , Vol. ...
2025
-
[17]
Evanfiya Logacheva, Arto Hellas, James Prather, Sami Sarsa, and Juho Leinonen
-
[18]
Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2022. Experiences from Using Code Expla- nations Generated by Large Language Models in a Web Software Development E-Book. In 54th ACM Technical Symposium on Comput...
2022 arXiv
-
[19]
Marie Hélěne Nienaltowski, Michela Pedroni, and Bertrand Meyer. 2008. Compiler error messages: What can help novices? SIGCSE’08 - Proceedings of the 39th ACM Technical Symposium on Computer Science Education (2008), 168–172. doi:10. 1145/1352135.1352192
2008
-
[20]
Maciej Pankiewicz and Ryan S Baker. 2023. Large Language Models (GPT) for automating feedback on programming assignments. arXiv:2307.00150 (2023)
2023 arXiv
-
[21]
Tung Phung, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023. Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models. (4 2023). http: //arxiv.org/abs/2302.04662
2023 arXiv
-
[22]
James Prather, Juho Leinonen, Natalie Kiesler, Jamie Gorson Benario, Sam Lau, Stephen MacNeil, Narges Norouzi, et al. 2025. Beyond the Hype: A Comprehen- sive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools. In 2024 Working Group Reports on In...
2025
-
[23]
Jake Renzella, Alexandra Vassar, Lorenzo Lee Solano, and Andrew Taylor. 2025. Compiler-Integrated, Conversational AI for Debugging CS1 Programs. In Pro- ceedings of the 56th ACM Technical Symposium on Computer Science Education V.1 (SIGCSETS 2025), Vol. 2022-March. Association...
2025
-
[24]
Emily Ross, Yuval Kansal, Jake Renzella, Alexandra Vassar, and Andrew Tay- lor. 2025. Supervised Fine-Tuning LLMs to Behave as Pedagogical Agents in Programming Education. (2 2025). https://arxiv.org/abs/2502.20527v1
2025 arXiv
-
[25]
Audrey Salmon, Katie Hammer, Eddie Antonio Santos, and Brett A. Becker
-
[26]
Eddie Antonio Santos, Brett A Becker, and Brett A 2024 Becker. 2024. Not the Silver Bullet: LLM-enhanced Programming Error Messages are Ineffective in Practice. Proceedings of the 2024 Conference on United Kingdom & Ireland Computing Education Research (11 2024). doi:10.1145/3689535
2024 doi
-
[27]
Eddie Antonio Santos, Prajish Prasad, and Brett A. Becker. 2023. Always Provide Context: The Effects of Code Context on Programming Error Message Enhance- ment. CompEd 2023 - Proceedings of the ACM Conference on Global Computing Education 1 (12 2023), 147–153. doi:10.1145/3576...
2023
-
[28]
Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic Gen- eration of Programming Exercises and Code Explanations Using Large Language Models. In ICER 2022 - Proceedings of the 2022 ACM Conference on International Computing Education Research, Vol. 1. Associa...
2022
-
[29]
Alexander Scarlatos, Digory Smith, Simon Woodhead, and Andrew Lan. 2024. Improving the Validity of Automatically Generated Feedback via Reinforcement Learning. Artificial Intelligence in Education: 25th International Conference (AIED
2024
-
[30]
Andrew Taylor, Alexandra Vassar, Jake Renzella, and Hammond Pearce. 2024. dcc - Help: Transforming the Role of the Compiler by Generating Context-Aware Error Explanations with Large Language Models. In SIGCSE 2024 - Proceedings of the 55th ACM Technical Symposium on Computer S...
2024
-
[31]
Llama team. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https: //arxiv.org/abs/2407.21783
2024 arXiv
- [32]
-
[33]
Smith, and Stephen MacNeil
Andrew Tran, Kenneth Angelikas, Egi Rama, Chiku Okechukwu, David H. Smith, and Stephen MacNeil. 2023. Generating Multiple Choice Questions for Comput- ing Courses Using Large Language Models. In 2023 IEEE Frontiers in Education Conference (FIE). 1–8. doi:10.1109/FIE58773.2023.10342898
2023
-
[34]
Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arXiv:2404.18796 [cs.CL] https://ar...
2024 arXiv
-
[35]
Qwen Team. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https: //arxiv.org/abs/2505.09388
2025 arXiv
-
[36]
Xing, Hao Zhang, Joseph E
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT- bench and Chatbot Arena. In Proceedings of the 37th Interna...
2023
-
[38]
Sierra Wang, John Mitchell, and Chris Piech. 2024. A Large Scale RCT on Effective Error Messages in CS1. In SIGCSE 2024 - Proceedings of the 55th ACM Technical Symposium on Computer Science Education , Vol. 1. Association for Computing Machinery, Inc, 1395–1401. doi:10.1145/36...
2024
-
[2024]
95-113)
Evaluating Contextually Personalized Programming Exercises Created with Generative AI (pp. 95-113).. In Proceedings of the 2024 ACM Conference on International Computing Education Research-Volume 1 (ICER, 2024) , Vol. 1. ACM, 95–113. doi:10.1145/3632620.3671103
2024
-
[2025]
doi:10.1145/arXiv.2501.05706
Debugging Without Error Messages: How LLM Prompting Strategy Affects Programming Error Explanation Effectiveness. doi:10.1145/arXiv.2501.05706
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.