pith. sign in

arxiv: 2604.11836 · v1 · submitted 2026-04-12 · 💻 cs.CY · cs.SE

Design and Deployment of a Course-Aware AI Tutor in an Introductory Programming Course

Iris Groher , Patrick Heissenberger , Michael Vierhauser This is my paper

Pith reviewed 2026-05-10 16:17 UTC · model grok-4.3

classification 💻 cs.CY cs.SE
keywords AI tutorintroductory programmingPythoncourse-aware guidanceSocratic questionsretrieval-augmentedstudent feedback
0
0 comments X

The pith

A course-specific Python tutor supplies hints and Socratic questions drawn from class materials instead of complete solutions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to build and test an AI tutor locked to the content of one introductory programming course. The tutor combines a browser-based coding workspace with a chat agent that retrieves relevant explanations and poses guiding questions drawn only from the course notes and assignments. A sympathetic reader would care because many students now turn to general large language models for ready-made answers, which can sidestep the practice of solving problems themselves. The authors collected feedback forms and usage logs to see whether this restricted, course-tied design changed how students asked for help.

Core claim

We designed and deployed a course-specific online Python tutor that provides retrieval-augmented, course-aligned guidance without generating complete solutions. The tutor integrates a web-based programming environment with a conversational agent that offers hints, Socratic questions, and explanations grounded in course materials. Students used the system during self-study to work on homework assignments, and the tutor also supported questions about the broader course material. We observed that students used the tutor primarily for conceptual understanding, implementation guidance, and debugging, and perceived it as a course-aligned, context-aware learning support that encourages engagement.

What carries the argument

Retrieval-augmented conversational agent that pulls only from the course's own materials to generate hints, Socratic questions, and partial explanations while refusing to emit full program solutions.

If this is right

  • Students receive targeted help on homework while still having to carry out the core coding steps themselves.
  • The same system can answer questions about lecture concepts and course policies in addition to assignment-specific issues.
  • Usage logs show the dominant requests are for explanations of ideas, step-by-step implementation pointers, and help locating bugs.
  • Students report the tutor feels tied to their class rather than a generic tool that simply hands out answers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same retrieval-plus-restriction pattern could be tried in other first-year technical courses where over-reliance on general AI tools is a concern.
  • Future deployments could add logging of whether students eventually solve the problem after receiving hints, to track changes in persistence.
  • If the tutor is made available across multiple semesters, usage patterns might reveal which course topics consistently need more scaffolding.

Load-bearing premise

That the collected student feedback forms and interaction logs are enough to show the tutor actually keeps novices from neglecting their own problem-solving practice.

What would settle it

A controlled comparison in which students who used the tutor are later asked to complete similar programming tasks without any AI help, and their independent solution rates are measured against a matched group that never had the tutor.

read the original abstract

Large Language Models (LLMs) have become part of how students solve programming tasks, offering immediate explanations and even full solutions. Previous work has highlighted that novice programmers often heavily rely on LLMs, thereby neglecting their own problem-solving skills. To address this challenge, we designed a course-specific online Python tutor that provides retrieval-augmented, course-aligned guidance without generating complete solutions. The tutor integrates a web-based programming environment with a conversational agent that offers hints, Socratic questions, and explanations grounded in course materials. Students used the system during self-study to work on homework assignments, and the tutor also supported questions about the broader course material. We collected structured student feedback and analyzed interaction logs to investigate how they engaged with the tutor's guidance. We observed that students used the tutor primarily for conceptual understanding, implementation guidance, and debugging, and perceived it as a course-aligned, context-aware learning support that encourages engagement rather than direct solution copying.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper describes the design and deployment of a course-aware AI tutor for an introductory Python programming course. The system integrates a web-based IDE with a conversational agent that uses retrieval-augmented generation over course materials to deliver hints, Socratic questions, and explanations without generating complete solutions. Based on structured student feedback and interaction logs collected during self-study on homework, the authors observe that students primarily sought conceptual understanding, implementation guidance, and debugging support, and perceived the tutor as course-aligned support that encourages engagement rather than direct solution copying.

Significance. If the observations hold, the work offers a concrete example of a controlled LLM integration in programming education that aims to provide immediate support while discouraging over-reliance. It contributes practical design insights for course-specific tutors and real-world deployment data from an introductory course, which could inform future systems seeking to balance AI assistance with skill development.

major comments (1)
  1. [§5] §5 (Results/Evaluation): The central claim that the tutor design addresses novices neglecting their own problem-solving skills rests on the observation that students used it for conceptual hints and perceived it as encouraging engagement. However, the interaction logs only categorize query types without proxies for independent effort (e.g., time-to-first-query or error sequences before tutor use), and no baseline comparison to unrestricted LLM access or pre/post measures of problem-solving performance on held-out tasks are reported. Structured feedback is self-reported and lacks controls for social-desirability bias, leaving the claim that the system mitigates skill neglect unsupported by the presented evidence.
minor comments (2)
  1. [Abstract] Abstract and §4 (Data Collection): Sample size, exact number of interactions, distribution of query categories, and details of the qualitative/quantitative analysis methods are not reported, reducing the ability to assess the robustness of the observations.
  2. [§3] §3 (System Design): The precise prompting strategy and retrieval mechanism used to enforce 'no complete solutions' should be described in more detail, including any failure cases where full code was still generated.

Simulated Author's Rebuttal

1 responses · 1 unresolved

We thank the referee for their constructive and detailed feedback on our manuscript. We address the single major comment below, indicating where revisions will be made to clarify claims and limitations.

read point-by-point responses

  1. Referee: [§5] §5 (Results/Evaluation): The central claim that the tutor design addresses novices neglecting their own problem-solving skills rests on the observation that students used it for conceptual hints and perceived it as encouraging engagement. However, the interaction logs only categorize query types without proxies for independent effort (e.g., time-to-first-query or error sequences before tutor use), and no baseline comparison to unrestricted LLM access or pre/post measures of problem-solving performance on held-out tasks are reported. Structured feedback is self-reported and lacks controls for social-desirability bias, leaving the claim that the system mitigates skill neglect unsupported by the presented evidence.

    Authors: We appreciate the referee highlighting the distinction between design intent and empirical demonstration of skill preservation. The manuscript frames the work as an observational deployment study of a course-specific tutor that avoids generating complete solutions. The reported observations are that interaction logs show predominant use for conceptual understanding, implementation guidance, and debugging, while structured feedback indicates students perceived the tutor as aligned with course goals and encouraging engagement. We do not claim to have measured mitigation of skill neglect via direct proxies (time-to-first-query, error sequences) or controlled comparisons, as these data were not collected. Self-reported perceptions are indeed subject to bias. We will revise §5 and the discussion to explicitly distinguish the design goals and observed usage patterns from any assertion of proven skill development outcomes. The limitations section will be expanded to note the absence of baseline or pre/post measures and to recommend controlled studies for future work. revision: partial

standing simulated objections not resolved
  • We cannot add proxies for independent effort, baseline comparisons to unrestricted LLMs, or pre/post performance measures on held-out tasks, as the study was a single-group observational deployment without these elements in the data collection protocol.

Circularity Check

0 steps flagged

No circularity: purely descriptive design and observation study

full rationale

The paper describes the design and deployment of a course-specific AI tutor, reports interaction logs and structured student feedback, and presents observational findings on usage patterns. No equations, fitted parameters, predictions, uniqueness theorems, or derivation chains appear in the work. Claims about student engagement and perceptions are drawn directly from the collected data without any reduction to self-defined inputs or self-citation load-bearing steps. This is a standard non-circular descriptive systems paper.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical model, free parameters, axioms, or new entities are introduced; this is an applied design and evaluation study in educational technology.

pith-pipeline@v0.9.0 · 5462 in / 1212 out tokens · 74477 ms · 2026-05-10T16:17:43.848029+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages

  1. [1]

    F. I. Anaroua, Q. Li, Y. Tang, and H. P. Liu. Ai-driven formative assessment and adap- tive learning in data-science education: Evaluating an llm-powered virtual teaching assistant.arXiv:2509.20369, 2025

    work page arXiv 2025

  2. [2]

    P. L. S. Barbosa, R. A. F. d. Carmo, J. P. Gomes, and W. Viana. Adaptive learning in computer science education: A scoping review.Education and Information Technologies, 29(8):9139–9188, 2024

    work page 2024

  3. [3]

    Bassner, E

    P. Bassner, E. Frankford, and S. Krusche. Iris: An ai-driven virtual tutor for computer science education. InProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1, page 394–400. ACM, 2024

    work page 2024

  4. [4]

    B. A. Becker, P. Denny, J. Finnie-Ansley, A. Luxton-Reilly, J. Prather, and E. A. San- tos. Programming is hard - or at least it used to be: Educational opportunities and challenges of ai code generation. InProc. of the 54th ACM Techn. Symp. on Computer Science Education V. 1, page 500–506. ACM, 2023

    work page 2023

  5. [5]

    T. Crow, A. Luxton-Reilly, and B. Wuensche. Intelligent tutoring systems for pro- gramming education: a systematic review. InProc. of the 20th Australasian Computing Education Conf., page 53–62. ACM, 2018

    work page 2018

  6. [6]

    Denny, S

    P. Denny, S. MacNeil, J. Savelka, L. Porter, and A. Luxton-Reilly. Desirable charac- teristics for ai teaching assistants in programming education. InProc. of the 2024 on Innovation and Technology in Computer Science Education V. 1, page 408–414. ACM, 2024

    work page 2024

  7. [7]

    C. Dong, Y. Yuan, K. Chen, S. Cheng, and C. Wen. How to build an adaptive ai tutor for any course using knowledge graph-enhanced retrieval-augmented generation (kg-rag). InProc. of the 14th Int’l Conf. on Educational and Information Technology, pages 152–157. IEEE, 2025. 7 PREPRINT –18th International Conference on Computer Supported Education2026

    work page 2025

  8. [8]

    Farhood, M

    H. Farhood, M. Nyden, A. Beheshti, and S. Muller. Artificial intelligence-based per- sonalised learning in education: a systematic literature review.Discover Artificial Intelligence, 5(1):331, 2025

    work page 2025

  9. [9]

    Frankford, T

    E. Frankford, T. Antensteiner, M. Vierhauser, C. Sauerwein, V. Wallner, I. Groher, R. Plösch, and R. Breu. A survey on feedback types in automated programming assessment systems.ACM Trans. Comput. Educ., 26(1), Dec. 2025

    work page 2025

  10. [10]

    Groher, M

    I. Groher, M. Vierhauser, B. Sabitzer, L. Kuka, A. Hofer, and D. Muster. Exploring diversity in introductory programming classes: An experience report. InProc. of the 44th IEEE/ACM Int’l Conf/ on Software Engineering: Software Engineering Education and Training, pages 102–112. IEEE/ACM, 2022

    work page 2022

  11. [11]

    Groothuijsen, A

    S. Groothuijsen, A. van den Beemt, J. C. Remmers, and L. W. van Meeuwen. Ai chatbots in programming education: Students’ use in a scientific computing course and consequences for learning.Computers and Education: Artificial Intelligence, 7:100290, 2024

    work page 2024

  12. [12]

    Hellas, J

    A. Hellas, J. Leinonen, S. Sarsa, C. Koutcheme, L. Kujanpää, and J. Sorva. Exploring the responses of large language models to beginner programmers’ help requests. In Proc. of the ACM Conf. on Int’l Computing Education Research - Volume 1, page 93–105. ACM, 2023

    work page 2023

  13. [13]

    Kazemitabaar, J

    M. Kazemitabaar, J. Chow, C. K. T. Ma, B. J. Ericson, D. Weintrop, and T. Grossman. Studying the effect of ai code generators on supporting novice learners in introductory programming. InProc. of the 2023 CHI Conf. on Human Factors in Computing Systems. ACM, 2023

    work page 2023

  14. [14]

    Kazemitabaar, R

    M. Kazemitabaar, R. Ye, X. Wang, A. Z. Henley, P. Denny, M. Craig, and T. Grossman. Codeaid: Evaluating a classroom deployment of an llm-based programming assistant that balances student and educator needs. InProc. of the 2024 CHI Conf. on Human Factors in Computing Systems. ACM, 2024

    work page 2024

  15. [15]

    Krusche and A

    S. Krusche and A. Seitz. Artemis: An automatic assessment management system for interactive learning. InProc. of the 49th ACM Techn. Symp. on Computer Science Education, pages 284–289, 2018

    work page 2018

  16. [16]

    B., & Sting, F

    M. Lehmann, P. B. Cornelius, and F. J. Sting. AI meets the classroom: When does chatgpt harm learning?CoRR, abs/2409.09047, 2024

    work page arXiv 2024

  17. [17]

    Z. Li, J. Wang, W. Gu, V. Yazdanpanah, L. Shi, A. I. Cristea, S. Kiden, and S. Stein. Tu- torllm: Customizing learning recommendations with knowledge tracing and retrieval- augmented generation. InProc. of the 20th IFIP TC 13 Int’l Conf. on Human-Computer Interaction, page 137–146. Springer, 2025

    work page 2025

  18. [18]

    Liffiton, B

    M. Liffiton, B. E. Sheese, J. Savelka, and P. Denny. Codehelp: Using large language models with guardrails for scalable support in programming classes. InProc. of the 23rd Koli Calling Int’l Conf. on Computing Education Research, Koli Calling ’23. ACM, 2024

    work page 2024

  19. [19]

    J. J. Y. Lim, D. Zhang-Li, J. Yu, X. Cong, Y. He, Z. Liu, H. Liu, L. Hou, J. Li, and B. Xu. Learning in context: Personalizing educational content with large language models to enhance student learning.arXiv:2509.15068, 2025

    work page arXiv 2025

  20. [20]

    Lister, E

    R. Lister, E. S. Adams, S. Fitzgerald, W. Fone, J. Hamer, M. Lindholm, R. McCartney, J. E. Moström, K. Sanders, O. Seppälä, B. Simon, and L. Thomas. A multi-national study of reading and tracing skills in novice programmers.SIGCSE Bull., 36(4):119–150, June 2004

    work page 2004

  21. [21]

    R. Liu, C. Zenke, C. Liu, A. Holmes, P. Thornton, and D. J. Malan. Teaching cs50 with ai: Leveraging generative artificial intelligence in computer science education. InProc. of the 55th ACM Techn. Symp. on Computer Science Education V. 2, SIGCSE 2024, page

    work page 2024

  22. [22]

    W. Lyu, Y. Wang, T. R. Chung, Y. Sun, and Y. Zhang. Evaluating the effectiveness of llms in introductory computer science education: A semester-long field study. InProc. of the 1st ACM Conf. on Learning@Scale, page 63–74. ACM, 2024

    work page 2024

  23. [23]

    R. P. Medeiros, G. L. Ramalho, and T. P. Falcão. A systematic literature review on teach- ing and learning introductory programming in higher education.IEEE Transactions on Education, 62(2):77–90, 2018

    work page 2018

  24. [24]

    Roest, H

    L. Roest, H. Keuning, and J. Jeuring. Next-step hint generation for introductory programming using large language models. InProc. of the 26th Australasian Computing Education Conf., page 144–153. ACM, 2024

    work page 2024

  25. [25]

    Sheese, M

    B. Sheese, M. Liffiton, J. Savelka, and P. Denny. Patterns of student help-seeking when using a large language model-powered programming assistant. InProc. of the 26th Australasian Computing Education Conf., page 49–57. ACM, 2024

    work page 2024

  26. [26]

    L. C. Shum, Y. Rosunally, S. Scarle, and K. Munir. Personalised learning through context-based adaptation in the serious games with gating mechanism.Education and Information Technologies, 28(10):13077–13108, 2023

    work page 2023

  27. [27]

    Storey, R

    M.-A. Storey, R. Hoda, A. M. P. Milani, and M. T. Baldassarre. Guiding principles for using mixed methods research in software engineering.arXiv:2404.06011, 2024

    work page arXiv 2024

  28. [28]

    D. Teng, X. Wang, Y. Xia, Y. Zhang, L. Tang, Q. Chen, R. Zhang, S. Xie, and W. Yu. Investigating the utilization and impact of large language model-based intelligent teaching assistants in flipped classrooms.Educ. Inf. Technol., 30(8):10777–10810, 2025

    work page 2025

  29. [29]

    Weizenbaum

    J. Weizenbaum. Eliza—a computer program for the study of natural language commu- nication between man and machine.Commun. ACM, 9(1):36–45, Jan. 1966. 8

    work page 1966