{"id":"025e285c-7a07-4fa1-8c02-baa43d87a655","arxiv_id":"2507.22946","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Giving an LLM a transcript and degree plan makes its course suggestions align more with that plan, but the evaluation partly assumes that plan is the correct answer.","lead":"SmartCourse is a prototype system that combines course management with a locally hosted AI model to suggest classes for undergraduates. The paper reports that giving the AI a student's transcript and four-year plan improves course suggestions, but the evaluation uses that same plan as the measure of success.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PlanScore/Recall are circular: the degree plan is both a prompt input in full-context and no-transcript modes and the ground-truth reference set, so the ablation mainly measures whether the answer key was included in the prompt.","rationale":"The paper is a plausible prototype description, and the authors are transparent about prototype limitations. The system uses real components (CLI, Gradio, local Ollama-backed LLM) and the code is public. However, the experimental design supporting the central scientific claim is structurally circular: the degree plan is the reference set for PlanScore/Recall and is simultaneously included in the prompt in the two conditions being compared. The score differences therefore reflect information access, not recommendation quality. The reader's weakest assumption—that the degree plan is a valid ground truth—captures this issue, and my reading agrees with it. The proposed baseline check would be decisive: a rule-based plan-listing baseline should trivially match or exceed the system's PlanScore and Recall, which would refute the claim that the LLM's contextual reasoning produces the reported improvement. No change to the reader's REJECT verdict is needed.","tokens_in":8410,"tokens_out":4406,"duration_ms":52303,"concrete_test":"Run a deterministic baseline that, for each of the 25 queries, returns the first six outstanding courses from the degree plan (sorted by plan year/semester), using no LLM. Compute PlanScore, PersonalScore, and Recall and compare with the reported full-context values (0.53, 0.78, 0.15). If the baseline scores at or above the full-context model on PlanScore and Recall, the ablation result is an artifact of placing the answer key in the prompt rather than evidence of advising quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that full context yields substantially more relevant recommendations—rests on the metrics in Section V-C: PlanScore = |R∩P|/|R| and Recall = |R∩P|/|P|, where P is the set of outstanding degree-plan courses. Section III-B states that the prompt fuses the question, transcript, and four-year plan, so P is derivable directly from the input in the full-context and noTranscript conditions. Section V-B justifies P as \"the reference set\" because \"no external ground-truth answers exist,\" but making the answer key part of the prompt turns PlanScore and Recall into measures of whether the model repeats plan entries, not of advising quality. The full-vs-noPlan contrast mostly checks whether the plan was withheld. Internal numbers confirm the concern: a trivial baseline that returns the next six outstanding plan courses would score PlanScore ≈ 1.0 and Recall ≈ 0.33, above the reported full-context values of 0.53 and 0.15, while the noPlan condition (0.03, 0.01) has the key withheld. The limitation in Section VII-B acknowledges the plan is assumed ground truth but does not address its simultaneous role as a prompt input, so the circularity stands.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SmartCourse, a course-management and advising system that combines a CLI and Gradio GUI with a locally hosted LLM (llama3.1:8b via Ollama). The system fuses the student's transcript, four-year degree plan, and question into a prompt to generate course recommendations. To justify the design, the authors run an ablation over 25 hypothetical queries under four context conditions (full context, no transcript, no plan, question-only) and report custom metrics (PlanScore, PersonalScore, Lift, Recall) plus latency. The paper's central claim is that full context yields substantially more relevant recommendations than context-omitted modes, thereby confirming the necessity of transcript and plan information for personalized academic advising.","tokens_in":8679,"tokens_out":6547,"duration_ms":78174,"significance":"The architecture is concrete, the source code is publicly available, and the context-ablation setup is a sensible way to study input sensitivity. If the empirical results were valid, they would provide useful evidence that structured academic records improve LLM-based course advising. However, the empirical core is not currently valid: the reference set used by PlanScore and Recall is the degree plan, which is also a prompt input in the conditions that claim to perform best. With a single student profile, no external ground truth, and no significance tests, the paper cannot support its central claim. The honest limitations section does not remove this problem.","major_comments":[{"comment":"PlanScore and Recall are circular. P is the set of outstanding degree-plan courses (Section V-C), and Section III-B says the prompt fuses the question, transcript, and four-year plan. Thus in the full-context and noTranscript conditions the model is given P directly, and PlanScore = |R∩P|/|R| and Recall = |R∩P|/|P| become measures of whether the model echoes plan entries, not of recommendation quality. The noPlan condition, which withholds the plan, yields near-zero scores (0.03 and 0.01); this contrast is essentially a sanity check that the answer key was present in the prompt. A trivial baseline that returns any six outstanding plan courses would achieve PlanScore = 1.0 and Recall = 0.33, exceeding the reported full-context values (0.53 and 0.15). An external reference set (e.g., human-advisor judgments or a held-out portion of a plan not included in the prompt) is required.","section":"V-C and III-B"},{"comment":"The evaluation rests on one CPS student profile and 25 hypothetical queries. There are no significance tests, no human raters, no comparison against an advising professional, and no variation in majors, institutions, or transcripts. The 10,000 bootstrap iterations only resample the same 25 responses; they cannot correct the confound that the full and noTranscript conditions include the reference set in the prompt. The abstract's phrase 'substantially more relevant' is therefore not supported by the evidence; at most the results show that the model outputs more plan-derived course codes when the plan is in the prompt.","section":"V-A, V-B, VI, and Table I"},{"comment":"PersonalScore and Lift are also affected by input leakage and by the imbalance in the number of recommendations. L (low-grade courses from the transcript) is available to the model in the full and noPlan conditions but not in the noTranscript condition. The high Lift in full context (0.25) and the high PersonalScore (0.78) may reflect that the model was told which courses are low-grade, rather than an emergent advising capability. Moreover, PlanScore and PersonalScore are computed over different numbers of recommendations (6.56 vs 2.24 vs 6.20 vs 0.04), so raw mean comparisons mix precision and output-length effects.","section":"V-C and Table I"},{"comment":"The limitation paragraph 'Incomplete Evaluation Metrics' admits that the metrics assume the degree plan is ground truth, but it does not acknowledge the more serious problem that the degree plan is one of the model's inputs. The paper states 'Because no external ground-truth answers exist, we treat the degree plan as the reference set' (Section V-B); this conflation of input and ground truth is what makes the central claim circular. The limitation should be restated as a fundamental validity threat, not a minor caveat.","section":"VII-B and V-B"}],"minor_comments":[{"comment":"The caption contains a long run of '/uni00000049/...' tokens; this appears to be a text-encoding corruption and should be removed or replaced with the actual figure.","section":"Table I caption"},{"comment":"The text refers to Fig. 6 but the supplied text contains only corrupted tokens; ensure the figure is properly embedded and readable in the final version.","section":"Fig. 6"},{"comment":"The 25 queries are not listed; for reproducibility, include the full query set in an appendix along with the model, sampling parameters, and the parsing/filtering rules used to extract course codes from the LLM output.","section":"V-A"},{"comment":"The paper uses 'B−' as the low-grade threshold for L but does not state how this is encoded in the transcript data; please define the grade scale and threshold explicitly.","section":"V-C"},{"comment":"The phrase 'confirming the necessity' should be weakened to 'suggesting' until the evaluation is redone with a non-circular protocol and broader data.","section":"Abstract"}],"recommendation":"reject","confidential_remarks":"The paper's main contribution is a system prototype with a public code repository; the empirical evaluation is unfortunately not salvageable from the current data because the reference metric is derived from a prompt input. If the authors rework the evaluation with human-expert judgments across multiple student profiles and remove the answer-key leakage, a future version may be reconsidered. The /uni artifacts in Table I also suggest a rendering problem that should be fixed before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the system is real and the code is public, but the evaluation does not support the paper's central claim. The degree plan is both a prompt input and the ground-truth reference set, so PlanScore and Recall mostly measure whether the model repeats the plan it was given. The stress-test note is right about that.\n\nWhat is actually new: SmartCourse integrates a course manager, degree plan repository, and local LLM into one CLI/GUI system. The integration is modest but genuine, and the source code is available. The paper runs a clean context ablation (full, no transcript, no plan, question-only) and reports bootstrap CIs. The limitations section is unusually candid: single student profile, hypothetical queries, no external ground truth, privacy not addressed, and the admission that the metrics assume the plan as ground truth.\n\nWhere it goes soft: the ground-truth assumption is not just fragile, it is circular. In full-context and no-transcript modes the plan is in the prompt, so high PlanScore means the model echoes its input. The no-transcript mode actually scores slightly higher PlanScore (0.60 vs 0.53) than full context; the authors attribute that to the model defaulting to plan suggestions, but that is exactly why the metric cannot distinguish good advice from regurgitation. Add one student profile, 25 queries, no baseline beyond the ablation, and no human evaluation, and the abstract's 'substantially more relevant' is overreach. The authors flag the ground-truth issue in Section VII-B but do not address the prompt-input overlap, so the circularity stands.\n\nWho this is for: people building LLM advising prototypes. It is a useful minimal example, and the honest failure modes are instructive. But as evidence that transcript and plan context is necessary, it fails.\n\nRecommendation: this is a workshop-level systems description, not a research paper in its current form. I would desk-reject at a selective venue and tell the authors to rework the evaluation: use held-out human advisor judgments or track-based ground truth not fed to the model, multiple students, and a nontrivial baseline. With that fixed, the system description is publishable as a short paper.","headline":"A working prototype with honest limitations, but the evaluation is circular: the degree plan is both input and ground truth, so the central claim is unsupported.","tokens_in":665,"tokens_out":999,"would_cite":false,"duration_ms":36632,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an LLM course advisor needs both the student's transcript and degree plan to give useful suggestions, and supports the claim with a four-way ablation on 25 queries.","keywords":["academic advising","course recommendation","large language models","degree planning","transcript context","context ablation","Ollama","SmartCourse"],"falsifier":"Have a panel of experienced human advisors independently rate the same 25 recommendations from all four context conditions without knowing which condition produced them; if their ratings do not put full-context recommendations clearly above plan-only and question-only ones, the central claim would collapse. A cheaper check: compare full-context recommendations against a held-out set of plan courses excluded from the prompt, and see whether the model still recovers them.","tokens_in":8237,"feed_emoji":"🎓","tokens_out":3591,"duration_ms":40926,"temperature":0.7,"pith_summary":"The paper argues that a locally hosted LLM can give substantially more relevant course recommendations when the prompt includes the student's transcript and four-year degree plan, and that omitting either input degrades recommendation quality sharply. To test this, SmartCourse fuses transcript, plan, and question into a structured prompt and evaluates 25 advising queries under four conditions: full context, no transcript, no plan, and question-only. The core finding is that full context yields the best personalized scores (PersonalScore 0.78, Lift 0.25), while question-only mode produces near-zero relevant recommendations and omitting the plan drops PlanScore to 0.03. The authors interpret this as confirming that transcript and plan information are necessary for personalized academic advising.","feed_headline":"Context-free AI course advice scores near zero on relevance","feed_subtitle":"A 25-query ablation shows transcript and degree plan boost personalization, lifting PlanScore from 0.03 to 0.53.","key_machinery":"The load-bearing mechanism is the contextual prompt construction: SmartCourse takes the student's question, their current transcript, and the four-year degree plan, concatenates them into a structured prompt, and sends it to a locally hosted llama3.1:8b model through Ollama, then post-filters the model's output against the course catalog. The evaluation machinery consists of four custom metrics defined over the recommended set R, the outstanding plan set P, and the low-grade set L: PlanScore (|R∩P|/|R|), PersonalScore (|R∩(P∪L)|/|R|), Lift (PersonalScore − PlanScore), and Recall (|R∩P|/|P|). These metrics carry the argument by quantifying how much of the recommendation quality is attributable to plan alignment and how much to transcript-driven personalization.","core_discovery":"On the paper's own terms, the central discovery is that context matters decisively for LLM-based course advising: feeding the model both the student's transcript and the four-year plan produces recommendations that align with unmet degree requirements (mean PlanScore 0.53) and with unfinished or low-grade courses (mean PersonalScore 0.78), whereas removing the plan collapses PlanScore to 0.03 and removing both contexts yields essentially no useful recommendations. The paper also reports that providing only the plan (no transcript) keeps PlanScore high (0.60) but sacrifices personalization, with Lift dropping from 0.25 to 0.09, because the model cannot account for courses already completed or low grades. The authors conclude that full context enables the system to suggest both required courses and appropriate retakes, and that structured student records are necessary for meaningful AI-driven advising.","pith_inferences":["A natural next test would be to run the same four-condition ablation with a human advisor panel grading the recommendations, which would test whether the degree-plan-as-ground-truth assumption matches expert judgment.","The single student profile and 25 handmade queries make the quantitative gaps suggestive rather than conclusive; extending to multiple majors, multiple transcripts, and more varied queries would show whether the pattern generalizes.","The metrics could be enriched with a precision measure against the course catalog and a penalty for recommending courses outside the major, which would directly address the paper's acknowledged blind spot for useful non-plan electives.","The authors' own limitation note that Recall becomes ill-defined when a student has nearly finished all plan courses points to a straightforward fix: compute Recall only over the set of students with a nontrivial number of outstanding requirements."],"forward_implications":["If a deployed advisor omits the degree plan, it will mostly miss required courses: PlanScore drops from 0.53 in full context to 0.03 in no-plan mode.","Question-only advising from a bare LLM is effectively unusable for academic queries: PlanScore and PersonalScore both fall to 0.04.","Providing only the degree plan keeps curriculum coverage (PlanScore 0.60) but loses the personalization that comes from knowing completed and low-grade courses, so transcript context is the main driver of Lift.","Post-filtering LLM suggestions against the official course catalog is a practical defense against hallucinations, catching invented course codes and titles.","At roughly 48 seconds per response in full-context mode, local LLM advising needs optimization before it can support real-time interactive use."],"supporting_citations":[{"why":"Supplies the recommender-evaluation principle that accuracy alone is insufficient, motivating the inclusion of coverage and user-need alignment in PlanScore, PersonalScore, Lift, and Recall.","marker":"[15]"},{"why":"Provides the evaluation-framework context cited for measuring coverage and user-need alignment, grounding the custom metric design.","marker":"[16]"},{"why":"Offers a comparative study of GPT-4 versus advisor-based major recommendations, which this work builds on to justify integrating LLMs into the advising workflow.","marker":"[13]"},{"why":"Supplies prior work on degree-plan-aware recommendation with PLAN-BERT, establishing that structured curricular requirements are a recognized input for course recommendation.","marker":"[9]"},{"why":"Documents the hallucination phenomenon in natural language generation, which the paper cites to explain and justify its post-filtering of LLM output against the course catalog.","marker":"[17]"}],"fun_headline_variants":["Transcript and degree plan lift AI advising relevance from 0.03 to 0.53","LLM course advice needs transcript and plan: PlanScore 0.03→0.53","AI advising relevance 17x higher with transcript and degree plan","Context-free AI advisors fail: PlanScore 0.03 vs 0.53"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The degree plan is treated as the ground truth for what counts as a good recommendation, even though the same plan is also fed to the model in the conditions being scored.","fun_headline_variants_meta":{"raw":{"variants":["Transcript and degree plan lift AI advising relevance from 0.03 to 0.53","LLM course advice needs transcript and plan: PlanScore 0.03→0.53","AI advising relevance 17x higher with transcript and degree plan","Context-free AI advisors fail: PlanScore 0.03 vs 0.53"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000687,"raw_usage":{"total_tokens":3094,"prompt_tokens":901,"completion_tokens":2193,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":2104}},"tokens_in":517,"tokens_out":2193,"duration_ms":19255,"temperature":1.0,"reasoning_tokens":2104,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:50:12.006135+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a panel of experienced human advisors independently rate the same 25 recommendations from all four context conditions without knowing which condition produced them; if their ratings do not put full-context recommendations clearly above plan-only and question-only ones, the central claim would collapse. A cheaper check: compare full-context recommendations against a held-out set of plan courses excluded from the prompt, and see whether the model still recovers them.","supporting_citations":[{"cited_title":"AI-Augmented Advising: A Comparative Study of GPT-4 and Advisor-based Major Recommendations,","cited_arxiv_id":null,"evidence_quote":"Offers a comparative study of GPT-4 versus advisor-based major recommendations, which this work builds on to justify integrating LLMs into the advising workflow."},{"cited_title":"Degree planning with PLAN-BERT: Multi-semester recommendation using future courses of interest,","cited_arxiv_id":null,"evidence_quote":"Supplies prior work on degree-plan-aware recommendation with PLAN-BERT, establishing that structured curricular requirements are a recognized input for course recommendation."}],"review_version":1}