{"id":"877323d4-4da3-4351-b06f-a3d9f67bb09c","arxiv_id":"2509.08862","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A deployed LLM course assistant served 589 students across three CS courses; logs show heavy evening use and homework questions, while only about 11% of responses included AI follow-ups that students mostly ignored.","lead":"This paper reports on a large deployment of an LLM-powered course assistant across three computer science courses, analyzing student usage and response quality. It finds evening and night use, homework-mode dominance, and that students often ignore AI follow-up questions, raising questions about inquiry-based learning with LLMs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Bloom's taxonomy and response-quality claims rest on single-rater annotation with no reliability statistics, so the core 'LLM limitations' finding is not yet supported.","rationale":"The paper provides a useful descriptive account of a real LLM course-assistant deployment, and the usage-pattern findings (evening/night usage, introductory-course differences) are supported by log data and are likely robust. However, the abstract's broader pedagogical claims—especially the Bloom's-based limitation of LLM capabilities—are only as strong as the annotation. Section 4.4.1 states that conversations were annotated but gives no reliability statistics, no number of annotators, and no coding manual; two authors have an employment conflict of interest with the system vendor. This is exactly the reader's weakest assumption, and I agree. Without a reliability check, the percentages could reflect single-rater subjectivity. My proposed re-coding test directly addresses this: if independent coders cannot reproduce the Bloom's distributions or the correct/helpful rates, the central cognitive-level claims should be downgraded. The external-validity concern—that one system's prompt may not represent 'current LLM capabilities'—is real but secondary; even if re-coding confirms the levels, a prompt-ablation or multi-model study would be needed for the broad generalization. Thus the reader's CONDITIONAL verdict remains appropriate, with no change needed.","tokens_in":19337,"tokens_out":4411,"duration_ms":55430,"concrete_test":"Recruit two independent annotators (e.g., graduate students in CS education) who are blind to the paper's claims and affiliations. Provide them with a written coding manual based on the paper's Bloom's taxonomy categories and have them independently re-code a stratified random sample of at least 100 of the 600 conversations (or, if the data cannot be released, a representative sample of the same dialogue types). Compute Cohen's kappa for Bloom's level and for the correct/helpful judgment. If kappa is below 0.6, or if the proportion of higher-order (Analyze/Evaluate/Create) LLM-generated questions shifts by more than 10 percentage points relative to the reported values, the paper's key cognitive-level conclusion fails internal validity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline conclusion that 'current LLM capabilities are limited in generating higher-order cognitive questions' (Abstract; §4.4.2) depends entirely on manual annotation of Bloom's taxonomy levels of LLM-generated questions in 200 sampled conversations per course (§4.4.1). The paper reports no rubric, no double-coding, and no inter-rater reliability (e.g., Cohen's kappa), and the annotators are the system's own developers/authors (two authors are HiTA AI employees). This is a direct threat to the internal validity of every cognitive-level percentage in Figures 15–18, including the claim that LLM-generated questions are typically at the Apply level and do not deepen cognition. The same annotation pipeline also underlies the '92.33% correct/helpful' quality figure; if 'correct' was coded loosely, the effectiveness claims are similarly vulnerable. Additionally, the broad claim about 'current LLM capabilities' is an external inference from one GPT-4 system with one prompt configuration that explicitly instructs a Socratic dialogue style; an equally plausible interpretation is that the prompt shapes the cognitive level, not the LLM's intrinsic capability. Because the paper offers no reliability evidence, a reader cannot distinguish data-driven distributions from coder expectations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports on a deployed LLM-powered course assistant built on GPT-4 with retrieval-augmented generation, used by roughly 2,000 students across six courses at three institutions, and analyzes 12,060 conversations from three computer science courses at one institution in Spring 2024. The authors characterize usage patterns (temporal distribution, conversation mode, duration, and dialogue rounds), manually annotate sampled conversations for response correctness/helpfulness and Bloom's taxonomy levels, and analyze LLM-generated follow-up questions. They conclude that the assistant addresses temporal and novice support gaps, that homework mode is heavily used, that response quality is generally high with occasional computational/contextual errors, and that current LLM capabilities are limited in generating higher-order cognitive questions.","tokens_in":19672,"tokens_out":4323,"duration_ms":56266,"significance":"If fully supported, the paper would be a valuable empirical contribution to human-LLM interaction in education: it draws on a large real-world corpus, spans courses of different levels, and documents concrete failure cases (Figures 12-13). The descriptive usage results—temporal patterns, mode distribution, conversation lengths—are likely sound and constitute a useful baseline for future deployments. However, the headline cognitive-level and response-quality findings rest on manual annotation with no reported reliability evidence, and the generalization from one system/prompt configuration to 'current LLM capabilities' exceeds the evidence. The authors disclose the commercial affiliation and IRB review, which is commendable.","major_comments":[{"comment":"The Bloom's taxonomy classification, correctness/helpfulness coding, and follow-up response annotation are described as manual annotation of sampled conversations, but no rubric, coder count, training procedure, or inter-rater reliability (e.g., Cohen's kappa) is reported. Two authors are HiTA AI employees, so the coders are not independent of the system's developers. The 92.33% correct-and-helpful figure and the claim that LLM-generated questions are 'typically at the Apply level' therefore cannot be distinguished from coder expectation. This is load-bearing for RQ3 and for the Abstract's conclusion that current LLM capabilities are limited in generating higher-order cognitive questions.","section":"§4.4, Figures 15-18"},{"comment":"The paper generalizes from this specific system to 'current LLM capabilities.' The system uses GPT-4 with a prompt that explicitly instructs a Socratic dialogue style, and a particular RAG configuration (top-2 chunks, 4,000-character chunk size, §3.3). The observed low cognitive level of LLM-generated questions may be an artifact of this prompt and retrieval setup rather than an intrinsic property of current LLMs. The claim should be reframed as applying to this system/configuration, or supported by comparative evidence across prompts/models.","section":"Abstract; §4.4.2; §6(C)"},{"comment":"Developer/test conversations are not excluded from the usage analysis. Footnote 1 states that 4.13% of the 12,060 conversations come from developer users, and that 44.89% of developer conversations contain no questions. Since all reported usage statistics (temporal distribution, mode distribution, dialogue-round distribution) appear to be computed over the full corpus, the reported percentages include non-student traffic. The paper should report the analyses with developer conversations removed, or show that the results are robust to their inclusion.","section":"§4.1, §4.2, footnote 1"},{"comment":"The claim that 'usage remains strong in the evenings and nights ... indicating that our system helps address temporal support gaps and novice learner needs' goes beyond the data. The study shows that students queried the system at night and in introductory courses, but it does not measure whether these interactions resolved students' problems, improved learning, or effectively substituted for unavailable human support. Since RQ1 asks whether the assistant 'addresses gaps,' this needs direct evidence (e.g., resolution rates, follow-up success, student-reported outcomes) or a more cautious wording such as 'is used during hours when conventional support is unavailable.'","section":"§4.2.2; Abstract"}],"minor_comments":[{"comment":"The linguistic feature annotations (grammar, politeness, relevance, copied content) also lack a detailed coding rubric and reliability statistics. If these annotations were double-coded, please report agreement; otherwise, present them as exploratory.","section":"§4.3.1"},{"comment":"The figures report counts and relative frequencies without confidence intervals. Given the sampled nature of the annotation (200 conversations per course), confidence intervals or error bars would aid interpretation and prevent over-reading small differences.","section":"Figures 15-18"},{"comment":"The sentence describing student recruitment says students were 'invited and given access,' but does not clarify whether participation was opt-in or whether informed consent was obtained as part of the IRB protocol. Please clarify the consent procedure.","section":"§4.1"},{"comment":"Minor typographical and stylistic issues: the arXiv title has inconsistent capitalization ('LARGELANGUAGEMODEL' run-on), the subsection label '(A) Asking while Learning' is awkward, and reference [5] is a bare URL lacking an access date.","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The paper's main contribution is an empirical case study of a single system. The lack of inter-rater reliability evidence is the most serious issue; if the authors can provide reliability statistics or substantially soften the LLM-capability generalization, the paper becomes publishable as a deployment report. I saw no problematic citation pattern; the self-citations are to relevant prior work by the same group."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper is genuinely useful for its descriptive usage data: 12,060 conversations across three CS courses, showing evening and night usage, homework-mode dominance, short conversations, and low follow-up engagement. These are log-derived and don't depend on judgment, so they're likely solid.\n\nWhat's new: most prior deployments (CS50 AI, CodeAid, Jill Watson) were smaller or single-course. This gives a multi-course picture with temporal and mode breakdowns. The finding that ~21% of conversations end before a question is posed, and that students ignore most LLM-generated follow-up questions in advanced courses, is interesting and worth knowing.\n\nSoft spots, in order of importance. First, the Bloom's taxonomy and correctness/helpfulness claims rest on manual annotation with no rubric details, no double-coding, and no inter-rater reliability. The annotators are the system's own developers, which is a real conflict for quality claims. That doesn't make the numbers false, but it means the paper cannot support the strong general claim that 'current LLM capabilities are limited in generating higher-order cognitive questions.' The data show this system with this prompt produced mostly Apply-level questions. That's a system/prompt finding, not an LLM-capability verdict.\n\nSecond, the abstract overstates the deployment: 'six courses at three institutions' becomes three courses at one institution in the analysis. Not fatal, but the abstract and title should be clearer.\n\nThird, the error examples are useful but selected; the 92.33% 'correct and helpful' figure is self-coded and no breakdown by course or question type is given.\n\nFourth, the discussion sometimes drifts into speculation (e.g., about linguistic degradation and social harm) beyond what the data support. That's minor.\n\nCitation pattern is fine; self-citations are to prior system papers and are relevant.\n\nWho this is for: researchers working on LLM course assistants and AI-in-education deployment studies. It deserves a serious referee because real deployment data with thousands of conversations is scarce and valuable, even if the quality claims need hardening. My recommendation: send it to review, but ask for reliability stats and a tempered conclusion. I'd cite the descriptive findings, not the Bloom's claim.","headline":"Real deployment data on an LLM course assistant, but the headline cognitive-level conclusion rests on unreliability-checked self-annotation.","tokens_in":20069,"tokens_out":2835,"would_cite":true,"duration_ms":29232,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM course assistant usage peaks at night and in introductory courses, filling gaps in traditional academic support.","keywords":["LLM course assistants","student-LLM interaction","retrieval-augmented generation","Bloom's taxonomy","inquiry-based learning","homework mode","temporal support gaps","educational technology"],"falsifier":"An independent, blinded re-annotation of the same 600 sampled conversations by two or more coders who were not involved in building the system, with reported inter-rater agreement, would settle whether the correctness and Bloom's-level findings are stable; low agreement or a substantially higher error rate would overturn the central quality claims.","tokens_in":19313,"feed_emoji":"🌙","tokens_out":8334,"duration_ms":88041,"temperature":0.7,"pith_summary":"The paper reports on a deployed LLM-powered course assistant—a retrieval-augmented chatbot with educator-curated course content and separate general, homework, and practice modes—used by about 2,000 students across six courses at three universities by Spring 2024. Analyzing 12,060 conversations from three computer science courses, the authors aim to show the assistant fills temporal and individual support gaps: usage stays high from 6 PM to 1 AM, peaks before exams, and is proportionally stronger in introductory courses. They also aim to characterize its pedagogical limits: 53.79% of conversations used homework mode, and manual annotation of 200 conversations per course found 92.33% of responses correct and helpful, 2.17% erroneous, 5.5% unhelpful, and only 4% containing worked examples. When the system posed Socratic follow-up questions, they were often ignored, and the questions rarely reached higher cognitive levels. If these patterns hold, LLM assistants can extend academic support beyond campus hours, but realizing deeper inquiry will require more educator involvement in prompts, content, and policies.","feed_headline":"LLM course assistants draw heaviest use at night and in intro courses","feed_subtitle":"Why it matters: the assistant works after hours, but its push for deeper inquiry largely falls flat.","key_machinery":"The load-bearing mechanism is the deployed course assistant: a retrieval-augmented generation (RAG) pipeline on a GPT-4 backbone that retrieves chunks from an educator-curated knowledge database, applies course-specific instructions, and routes each question into general, homework, or practice mode. A homework auto-detection feature directs homework-like queries to a hints-instead-of-answers response. The prompt template also instructs the assistant to end some responses with Socratic follow-up questions; this feature is the probe for testing inquiry-based learning, and the logged conversations are the dataset behind every reported pattern.","core_discovery":"The central discovery is a data-driven characterization of how students use an LLM-powered course assistant in real courses. Usage stayed consistently high from 6 PM to 1 AM, ramped up before midterms and finals, and was proportionally stronger in introductory computer science than in advanced courses—evidence, the authors argue, that the assistant reaches students when human support is unavailable and is especially important to novices. In 600 manually annotated conversations, 92.33% of responses were judged correct and helpful, 2.17% contained errors (about half computational, half conceptual), and 5.5% were unhelpful or off-target; only 4% included worked examples. The system's Socratic f","pith_inferences":["Editorial inference: the nighttime usage result could generalize to other institutions with similar course pacing, but the absence of learning-outcome measures means the paper does not establish that off-hours use improves grades or retention.","Editorial inference: the low follow-up response rate may partly reflect interface design—follow-up questions appearing at the end of long text blocks can be easy to miss—rather than a fixed student disinterest in Socratic interaction.","Editorial inference: the Bloom's-level ceiling is likely configuration-dependent; explicitly prompting for 'Evaluate' or 'Create' level questions or using a secondary classifier could push the distribution higher than observed here.","Editorial inference: a testable extension would be to compare courses with and without educator-authored follow-up questions to see whether content-based questions receive higher response rates than LLM-generated ones."],"forward_implications":["If the usage patterns hold, LLM course assistants can demonstrably extend academic support into the evening and night hours when human help is unavailable.","If introductory-course students are the heaviest users, institutions adopting such assistants should expect the largest impact—and the largest responsibility—in early CS courses.","The prevalence of homework-mode use and the 21% of sessions with no question suggest that homework support with hints rather than answers is a key function, and that many students may need more scaffolding to ask productive questions.","If LLM-generated follow-up questions remain at lower Bloom's levels and are mostly ignored, prompt-level changes alone are unlikely to create deep inquiry; educator-designed question sets and policies will be needed.","The observed computational and conceptual errors, though a small share of responses, imply that monitoring and educator oversight of correctness remain necessary in deployed systems."],"supporting_citations":[{"why":"Supplies the retrieval-augmented generation method that grounds assistant answers in course materials.","marker":"[23]"},{"why":"Defines the GPT-4 backbone the assistant runs on.","marker":"[50]"},{"why":"Provides the Bloom's taxonomy framework used for all cognitive-level classifications.","marker":"[54, 55]"},{"why":"Supplies the inquiry-based learning theory that motivates the follow-up-question feature.","marker":"[17, 18]"},{"why":"Describes an earlier RAG-based LLM course assistant that this deployment extends.","marker":"[26]"},{"why":"Presents an LLM tutoring system with scaffolds and hints, informing the homework-mode design.","marker":"[31]"},{"why":"Reports on a deployed LLM programming assistant whose feedback features frame the design and evaluation.","marker":"[32]"},{"why":"Describes the RAG-based educational platform and educator-in-the-loop workflow this paper's system is built on.","marker":"[36]"}],"fun_headline_variants":["LLM course assistants busiest at night and in intro courses","Evening and intro students lean hardest on AI course help","LLM tutors help after hours, but Socratic prompts fall flat","Night and intro courses dominate AI assistant usage","AI course assistant: peak at night, weak on higher-order questions"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper's quality and cognitive-level results rest on the authors' own manual annotation of sampled conversations, with no reported inter-rater reliability or independent coder check.","fun_headline_variants_meta":{"raw":{"variants":["LLM course assistants busiest at night and in intro courses","Evening and intro students lean hardest on AI course help","LLM tutors help after hours, but Socratic prompts fall flat","Night and intro courses dominate AI assistant usage","AI course assistant: peak at night, weak on higher-order questions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001128,"raw_usage":{"total_tokens":4536,"prompt_tokens":767,"completion_tokens":3769,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":3686}},"tokens_in":511,"tokens_out":3769,"duration_ms":34261,"temperature":1.0,"reasoning_tokens":3686,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T20:57:23.628909+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent, blinded re-annotation of the same 600 sampled conversations by two or more coders who were not involved in building the system, with reported inter-rater agreement, would settle whether the correctness and Bloom's-level findings are stable; low agreement or a substantially higher error rate would overturn the central quality claims.","supporting_citations":[],"review_version":1}