Pith. sign in

REVIEW 4 major objections 5 minor 20 references

SmartCourse: A Contextual AI-Powered Course Advising System for Undergraduates

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that an LLM course advisor needs both the student's transcript and degree plan to give useful suggestions, and supports the claim with a four-way ablation on 25 queries.

desk verdict A working prototype with honest limitations, but the evaluation is circular: the degree plan is both input and ground truth, so the central claim is unsupported. read the letter →

arxiv 2507.22946 v1 pith:DET5YYP2 submitted 2025-07-26 cs.CY cs.AI

Yixuan Mi , Yiduo Yu , Yiyi Zhao This is my paper
classification cs.CYcs.AI
keywords academicadvisingcourserecommendationlargelanguagemodelsdegreeplanningtranscriptcontextablationOllamaSmart
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a locally hosted LLM can give substantially more relevant course recommendations when the prompt includes the student's transcript and four-year degree plan, and that omitting either input degrades recommendation quality sharply. To test this, SmartCourse fuses transcript, plan, and question into a structured prompt and evaluates 25 advising queries under four conditions: full context, no transcript, no plan, and question-only. The core finding is that full context yields the best personalized scores (PersonalScore 0.78, Lift 0.25), while question-only mode produces near-zero relevant recommendations and omitting the plan drops PlanScore to 0.03. The authors interpret this as confirming that transcript and plan information are necessary for personalized academic advising.

What carries the argument

The load-bearing mechanism is the contextual prompt construction: SmartCourse takes the student's question, their current transcript, and the four-year degree plan, concatenates them into a structured prompt, and sends it to a locally hosted llama3.1:8b model through Ollama, then post-filters the model's output against the course catalog. The evaluation machinery consists of four custom metrics defined over the recommended set R, the outstanding plan set P, and the low-grade set L: PlanScore (|R∩P|/|R|), PersonalScore (|R∩(P∪L)|/|R|), Lift (PersonalScore − PlanScore), and Recall (|R∩P|/|P|). These metrics carry the argument by quantifying how much of the recommendation quality is attributable to plan alignment and how much to transcript-driven personalization.

What would settle it

Have a panel of experienced human advisors independently rate the same 25 recommendations from all four context conditions without knowing which condition produced them; if their ratings do not put full-context recommendations clearly above plan-only and question-only ones, the central claim would collapse. A cheaper check: compare full-context recommendations against a held-out set of plan courses excluded from the prompt, and see whether the model still recovers them.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that context matters decisively for LLM-based course advising: feeding the model both the student's transcript and the four-year plan produces recommendations that align with unmet degree requirements (mean PlanScore 0.53) and with unfinished or low-grade courses (mean PersonalScore 0.78), whereas removing the plan collapses PlanScore to 0.03 and removing both contexts yields essentially no useful recommendations. The paper also reports that providing only the plan (no transcript) keeps PlanScore high (0.60) but sacrifices personalization, with Lift dropping from 0.25 to 0.09, because the model cannot account for courses already completed or low grades. The authors conclude that full context enables the system to suggest both required courses and appropriate retakes, and that structured student records are necessary for meaningful AI-driven advising.

Load-bearing premise

The degree plan is treated as the ground truth for what counts as a good recommendation, even though the same plan is also fed to the model in the conditions being scored.

Editorial extensions

If this is right

  • If a deployed advisor omits the degree plan, it will mostly miss required courses: PlanScore drops from 0.53 in full context to 0.03 in no-plan mode.
  • Question-only advising from a bare LLM is effectively unusable for academic queries: PlanScore and PersonalScore both fall to 0.04.
  • Providing only the degree plan keeps curriculum coverage (PlanScore 0.60) but loses the personalization that comes from knowing completed and low-grade courses, so transcript context is the main driver of Lift.
  • Post-filtering LLM suggestions against the official course catalog is a practical defense against hallucinations, catching invented course codes and titles.
  • At roughly 48 seconds per response in full-context mode, local LLM advising needs optimization before it can support real-time interactive use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test would be to run the same four-condition ablation with a human advisor panel grading the recommendations, which would test whether the degree-plan-as-ground-truth assumption matches expert judgment.
  • The single student profile and 25 handmade queries make the quantitative gaps suggestive rather than conclusive; extending to multiple majors, multiple transcripts, and more varied queries would show whether the pattern generalizes.
  • The metrics could be enriched with a precision measure against the course catalog and a penalty for recommending courses outside the major, which would directly address the paper's acknowledged blind spot for useful non-plan electives.
  • The authors' own limitation note that Recall becomes ill-defined when a student has nearly finished all plan courses points to a straightforward fix: compute Recall only over the set of students with a nontrivial number of outstanding requirements.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents SmartCourse, a course-management and advising system that combines a CLI and Gradio GUI with a locally hosted LLM (llama3.1:8b via Ollama). The system fuses the student's transcript, four-year degree plan, and question into a prompt to generate course recommendations. To justify the design, the authors run an ablation over 25 hypothetical queries under four context conditions (full context, no transcript, no plan, question-only) and report custom metrics (PlanScore, PersonalScore, Lift, Recall) plus latency. The paper's central claim is that full context yields substantially more relevant recommendations than context-omitted modes, thereby confirming the necessity of transcript and plan information for personalized academic advising.

Significance. The architecture is concrete, the source code is publicly available, and the context-ablation setup is a sensible way to study input sensitivity. If the empirical results were valid, they would provide useful evidence that structured academic records improve LLM-based course advising. However, the empirical core is not currently valid: the reference set used by PlanScore and Recall is the degree plan, which is also a prompt input in the conditions that claim to perform best. With a single student profile, no external ground truth, and no significance tests, the paper cannot support its central claim. The honest limitations section does not remove this problem.

major comments (4)
  1. [V-C and III-B] PlanScore and Recall are circular. P is the set of outstanding degree-plan courses (Section V-C), and Section III-B says the prompt fuses the question, transcript, and four-year plan. Thus in the full-context and noTranscript conditions the model is given P directly, and PlanScore = |R∩P|/|R| and Recall = |R∩P|/|P| become measures of whether the model echoes plan entries, not of recommendation quality. The noPlan condition, which withholds the plan, yields near-zero scores (0.03 and 0.01); this contrast is essentially a sanity check that the answer key was present in the prompt. A trivial baseline that returns any six outstanding plan courses would achieve PlanScore = 1.0 and Recall = 0.33, exceeding the reported full-context values (0.53 and 0.15). An external reference set (e.g., human-advisor judgments or a held-out portion of a plan not included in the prompt) is required.
  2. [V-A, V-B, VI, and Table I] The evaluation rests on one CPS student profile and 25 hypothetical queries. There are no significance tests, no human raters, no comparison against an advising professional, and no variation in majors, institutions, or transcripts. The 10,000 bootstrap iterations only resample the same 25 responses; they cannot correct the confound that the full and noTranscript conditions include the reference set in the prompt. The abstract's phrase 'substantially more relevant' is therefore not supported by the evidence; at most the results show that the model outputs more plan-derived course codes when the plan is in the prompt.
  3. [V-C and Table I] PersonalScore and Lift are also affected by input leakage and by the imbalance in the number of recommendations. L (low-grade courses from the transcript) is available to the model in the full and noPlan conditions but not in the noTranscript condition. The high Lift in full context (0.25) and the high PersonalScore (0.78) may reflect that the model was told which courses are low-grade, rather than an emergent advising capability. Moreover, PlanScore and PersonalScore are computed over different numbers of recommendations (6.56 vs 2.24 vs 6.20 vs 0.04), so raw mean comparisons mix precision and output-length effects.
  4. [VII-B and V-B] The limitation paragraph 'Incomplete Evaluation Metrics' admits that the metrics assume the degree plan is ground truth, but it does not acknowledge the more serious problem that the degree plan is one of the model's inputs. The paper states 'Because no external ground-truth answers exist, we treat the degree plan as the reference set' (Section V-B); this conflation of input and ground truth is what makes the central claim circular. The limitation should be restated as a fundamental validity threat, not a minor caveat.
minor comments (5)
  1. [Table I caption] The caption contains a long run of '/uni00000049/...' tokens; this appears to be a text-encoding corruption and should be removed or replaced with the actual figure.
  2. [Fig. 6] The text refers to Fig. 6 but the supplied text contains only corrupted tokens; ensure the figure is properly embedded and readable in the final version.
  3. [V-A] The 25 queries are not listed; for reproducibility, include the full query set in an appendix along with the model, sampling parameters, and the parsing/filtering rules used to extract course codes from the LLM output.
  4. [V-C] The paper uses 'B−' as the low-grade threshold for L but does not state how this is encoded in the transcript data; please define the grade scale and threshold explicitly.
  5. [Abstract] The phrase 'confirming the necessity' should be weakened to 'suggesting' until the evaluation is redone with a non-circular protocol and broader data.

Circularity Check

2 steps flagged · score 6.0 of 10

PlanScore, Recall, and PersonalScore are computed against the degree plan and transcript that are also prompt inputs, so the context-ablation claim largely measures whether the answer key was included.

  1. self definitional [Section V-B and V-C; prompt construction in Section III-B]
    "Because no external ground-truth answers exist, we treat the degree plan as the reference set ... PlanScore = |R ∩ P|/|R| ... Recall = |R ∩ P|/|P|."

    The reference set P is defined as outstanding degree-plan courses, but Section III-B states that the prompt fuses the question, transcript, and four-year plan. In Full Context and No Transcript modes, P is directly recoverable from the prompt. PlanScore and Recall therefore measure how much the LLM repeats plan entries supplied to it, not independent advising relevance. The Full-vs-NoPlan contrast mostly checks whether the plan was withheld; a trivial baseline returning the next six outstanding plan courses would score PlanScore ≈ 1.0 and Recall ≈ 0.33, exceeding the reported full-context 0.53/0.15. The Section VII-B limitation acknowledges the plan is assumed ground truth but does not address its simultaneous role as a prompt input.

  2. self definitional [Section V-C with Section III-B prompt content]
    "PersonalScore = |R ∩(P ∪ L)|/|R| ... L the set of courses the student has taken but with a low grade ... prompt that fuses the question text, current transcript, and four-year plan context."

    PersonalScore's reward set is P ∪ L, where P comes from the degree plan and L is derived from the transcript. Both are prompt inputs in Full Context, so the metric rewards the model for returning exactly the information it was given. The reported PersonalScore (0.78) and Lift (0.25) thus quantify consistency with the supplied record rather than external advising quality. The 'personalization' result is partially tautological: the model is evaluated against the same transcript and plan that define its context.

full rationale

The paper's central claim is that full context yields substantially more relevant recommendations. That claim is supported by PlanScore, PersonalScore, Lift, and Recall, whose reference sets are the degree plan and low-grade transcript courses. Those same artifacts are inserted into the LLM prompt in the Full Context and No Transcript conditions (Section III-B). PlanScore and Recall therefore measure overlap between the model's output and a prompt input, not alignment with an external gold standard. The paper states 'Because no external ground-truth answers exist, we treat the degree plan as the reference set,' but this does not remove the input/reference conflation; the Section VII-B limitation ('Our relevance metrics assume the degree plan is ground truth') names the assumption but leaves the dual role unresolved. The noPlan ablation still shows input-dependence, but the stronger conclusion that context yields 'substantially more relevant' recommendations is partly forced by construction. Score 6 reflects partial circularity in the central evaluation, not a complete derivation collapse; the system implements a real pipeline and the reported numbers are internally consistent.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two unvalidated assumptions: the degree plan is the correct answer set, and the custom metrics measure relevance. The only hand-chosen number is the low-grade threshold (B-) used to define retake suggestions. No new entities are introduced.

free parameters (1)
  • Low-grade threshold = B- (grades below B-)
    Used in PersonalScore and Lift (Section V-C) to define L. Chosen by the authors without justification; affects reported Lift.
assumptions (3)
  • domain assumption The four-year degree plan is treated as ground truth for relevant courses.
    Section V-B: 'Because no external ground-truth answers exist, we treat the degree plan as the reference set.' This is load-bearing because PlanScore and Recall measure against it.
  • domain assumption The 25 hand-written queries are representative of real advising questions.
    The queries are hypothetical and authored by the researchers; there is no evidence of representativeness. Section V-B.
  • ad hoc to paper The metrics PlanScore, PersonalScore, Lift, Recall capture recommendation quality.
    These metrics are introduced by the authors and not validated against user satisfaction or learning outcomes. Section V-C.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SmartCourse: A Contextual AI-Powered Course Advising System for Undergraduates." pith.science (2026). https://pith.science/paper/DET5YYP2

@misc{pith2026250722946,
  author       = {Pith},
  title        = {Pith review of: SmartCourse: A Contextual AI-Powered Course Advising System for Undergraduates},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DET5YYP2}},
  note         = {Machine review of arXiv:2507.22946}
}
read the original abstract

We present SmartCourse, an integrated course management and AI-driven advising system for undergraduate students (specifically tailored to the Computer Science (CPS) major). SmartCourse addresses the limitations of traditional advising tools by integrating transcript and plan information for student-specific context. The system combines a command-line interface (CLI) and a Gradio web GUI for instructors and students, manages user accounts, course enrollment, grading, and four-year degree plans, and integrates a locally hosted large language model (via Ollama) for personalized course recommendations. It leverages transcript and major plan to offer contextual advice (e.g., prioritizing requirements or retakes). We evaluated the system on 25 representative advising queries and introduced custom metrics: PlanScore, PersonalScore, Lift, and Recall to assess recommendation quality across different context conditions. Experiments show that using full context yields substantially more relevant recommendations than context-omitted modes, confirming the necessity of transcript and plan information for personalized academic advising. SmartCourse thus demonstrates how transcript-aware AI can enhance academic planning.

Figures

Figures reproduced from arXiv: 2507.22946 by the authors.

Figure 1
Figure 1. SmartCourse system architecture. Users interact via CLI (administra [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 5
Figure 5. Student interface for registering and dropping courses, viewing [PITH_FULL_IMAGE:figures/full_fig_p003_5.png] view at source ↗
Figure 3
Figure 3. Administrator interface for managing accounts, courses, and AI [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Instructor interface for assigning grades and viewing student course [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 6
Figure 6. Figure 6: Comparison of average evaluation metrics (PlanScore, PersonalScore, [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 9 canonical work pages

  1. [1]

    The role of academic advising in student retention and persistence,

    J. K. Drake, “The role of academic advising in student retention and persistence,” About Campus , vol. 16, no. 3, pp. 8–12, 2011. doi: 10.1002/abc.20062

    work page doi:10.1002/abc.20062 2011

  2. [2]

    Academic Analytics: A New Tool for a New Era,

    J. P. Campbell, P. B. DeBlois, and D. G. Oblinger, “Academic Analytics: A New Tool for a New Era,” EDUCAUSE Review, vol. 42, no. 4, pp. 40–57, Jul./Aug 2007. Available: https://er.educause.edu/articles/2007/7/academic-analytics-a-new- tool-for-a-new-era

    work page 2007

  3. [3]

    Learning analytics: The emergence of a discipline,

    G. Siemens, “Learning analytics: The emergence of a discipline,” American Behavioral Scientist , vol. 57, no. 10, pp. 1380–1400, 2013. doi: 10.1177/0002764213498851

    doi:10.1177/0002764213498851 2013

  4. [4]

    Advisely: AI-Powered Academic Advising Using Large Language Models (LLMs),

    S. Abdelhamid, J. Bangura, and S. Shah, “Advisely: AI-Powered Academic Advising Using Large Language Models (LLMs),” in Pro- ceedings of the 14th International Conference on New Perspectives in Science Education (NPSE’14) , Florence, Italy, Mar. 21, 2025. doi: 10.18608/npse14.7058

    work page doi:10.18608/npse14.7058 2025

  5. [5]

    Jill Watson: A virtual teaching assis- tant for online education,

    A. K. Goel and L. Polepeddi, “Jill Watson: A virtual teaching assis- tant for online education,” in C. Dede, J. Richards, and B. Saxberg, Eds., Learning Engineering for Online Education: Theoretical Contexts and Design-Based Examples , 1st ed., Routledge, 2018, ch. 7. doi: 10.4324/9781351186193-7

    work page doi:10.4324/9781351186193-7 2018

  6. [6]

    Domain-aware grade prediction and top- n course recommendation,

    A. Elbadrawy and G. Karypis, “Domain-aware grade prediction and top- n course recommendation,” in Proceedings of the 10th ACM Conference on Recommender Systems (RecSys ’16) , Boston, MA, USA, 2016, pp. 183–190. doi: 10.1145/2959100.2959133

    arXiv 2016

  7. [7]

    Personalized course sequence recommendations,

    J. Xu, T. Xing, and M. van der Schaar, “Personalized course sequence recommendations,” IEEE Transactions on Signal Processing , vol. 64, no. 20, pp. 5340–5352, Oct. 2016. doi: 10.1109/TSP.2016.2595495

    arXiv 2016

  8. [8]

    Helping university students to choose elective courses by using a hybrid multi-criteria recommendation system with genetic optimization,

    A. Esteban, A. Zafra, and C. Romero, “Helping university students to choose elective courses by using a hybrid multi-criteria recommendation system with genetic optimization,” Knowledge-Based Systems, vol. 194, p. 105385, Apr. 2020. doi: 10.1016/j.knosys.2019.105385

    arXiv 2020

Show all 20 references
  1. [9]

    Degree planning with PLAN-BERT: Multi-semester recommendation using future courses of interest,

    E. Shao, S. Guo, and Z. Pardos, “Degree planning with PLAN-BERT: Multi-semester recommendation using future courses of interest,” Pro- ceedings of the AAAI Conference on Artificial Intelligence , vol. 35, no. 17, pp. 14920–14929, May 2021. doi: 10.1609/aaai.v35i17.17751

    2021 doi

  2. [10]

    From interests to in- sights: An LLM approach to course recommendations using natural language queries,

    H. Van Deventer, M. Mills, and A. Evrard, “From interests to in- sights: An LLM approach to course recommendations using natural language queries,” arXiv preprint arXiv:2412.19312, 2024. Available: https://arxiv.org/abs/2412.19312

    2024 arXiv

  3. [11]

    ChatGPT for good? On opportunities and challenges of large language models for education,

    E. Kasneci, K. Sessler, S. K ¨uchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. G ¨unnemann, E. H ¨ullermeier, S. Kr- usche, G. Kutyniok, T. Michaeli, C. Nerdel, J. Pfeffer, O. Poquet, M. Sailer, A. Schmidt, T. Seidel, M. Stadler, J. Weller, J. Kuhn, and ...

    2023

  4. [12]

    Exploring LLM-based student simulation for metacog- nitive cultivation,

    H. Li, J. Yu, X. Cong, Y . Dang, D. Zhang-li, Y . Zhan, H. Liu, and Z. Liu, “Exploring LLM-based student simulation for metacog- nitive cultivation,” arXiv preprint arXiv:2502.11678, 2025. Available: https://arxiv.org/abs/2502.11678

    2025

  5. [13]

    AI-Augmented Advising: A Comparative Study of GPT-4 and Advisor-based Major Recommendations,

    K. Lekan and Z. A. Pardos, “AI-Augmented Advising: A Comparative Study of GPT-4 and Advisor-based Major Recommendations,” Journal of Learning Analytics , vol. 12, no. 1, pp. 110–128, Mar. 2025. doi: 10.18608/jla.2025.8593

    2025

  6. [14]

    Large language model in higher education: Leveraging Llama2 for effective academic advising,

    A. Aguila, N. Nhan, N. D. Nguyen, K. T. Huynh, A. Mai, L. Tan, N. Tan, and T. Nguyen, “Large language model in higher education: Leveraging Llama2 for effective academic advising,” in Proceedings of the 2024 International Conference on Advanced Technologies for Communications ...

    2024

  7. [15]

    Evalu- ating collaborative filtering recommender systems,

    J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl, “Evalu- ating collaborative filtering recommender systems,” ACM Transactions on Information Systems , vol. 22, no. 1, pp. 5–53, Jan. 2004. doi: 10.1145/963770.963772

    2004

  8. [16]

    Evaluating recommendation systems,

    G. Shani and A. Gunawardana, “Evaluating recommendation systems,” in F. Ricci, L. Rokach, B. Shapira, and P. B. Kantor, Eds., Recommender Systems Handbook, Boston, MA: Springer US, 2011, pp. 257–297. doi: 10.1007/978-0-387-85820-3 8

    2011 doi

  9. [17]

    Survey of hallucination in natural language generation,

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y . Xu, E. Ishii, Y . J. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,” ACM Computing Surveys, vol. 55, no. 12, art. 248, pp. 1–38, Mar. 2023. doi: 10.1145/3571730

    2023 doi

  10. [18]

    Protecting student privacy,

    U.S. Department of Education, “Protecting student privacy,” Student Privacy Policy Office, Privacy Technical Assistance Center , [Online]. Available: https://studentprivacy.ed.gov/ferpa

  11. [19]

    What is GDPR, the EU’s new data protection law?,

    B. Wolford, “What is GDPR, the EU’s new data protection law?,” GDPR.eu, [Online]. Available: https://gdpr.eu/what-is-gdpr/

  12. [20]

    The rise and potential of large language model based agents: A survey,

    Z. Xi et al. , “The rise and potential of large language model based agents: A survey,” arXiv preprint arXiv:2309.07864, 2023. Available: https://arxiv.org/abs/2309.07864

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.