{"id":"83638ad9-eb8c-44de-a99a-d94beb980455","arxiv_id":"2505.10074","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A Graph RAG pipeline that uses educational and personal knowledge graphs generates personalized MOOC questions, but its answer accuracy in a 100-item instructor evaluation was only 45 percent.","lead":"This paper builds a system for MOOCs that suggests personalized questions based on concepts learners mark as unclear, then answers those questions using Wikipedia content linked through a knowledge graph. A small instructor evaluation found the questions useful, but only 45 percent of the answers were judged correct, so the system is at an early stage.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No baseline or ablation: the high question-generation scores could be produced by the LLM alone, so the graph-specific contribution claimed by the paper is not established.","rationale":"The reader's verdict is CONDITIONAL, and this stress-test does not move that verdict. The paper is an early-stage system paper with a hedged central claim, and the authors honestly report the 45% answer accuracy. The main positive evidence is the question-generation evaluation, but that evidence is missing the single most important control: a comparison with a non-graph RAG or plain LLM pipeline. The reader's stated weakest assumption focuses on Wikipedia coverage sufficiency in the EduKG; that is a related but different concern. Our concern is that even when the knowledge base does contain the needed information, the evaluation does not establish that the graph components are what make retrieval or generation work. This is a concrete, addressable methodological gap rather than a fatal flaw. The recommended action remains conditional acceptance with a required revision: add a baseline or ablation, release data and code, and report inter-rater agreement and variance. Therefore UNCHANGED is appropriate.","tokens_in":7407,"tokens_out":4849,"duration_ms":52759,"concrete_test":"Run a pre-registered, blinded head-to-head study on the same 100 question-answer pairs with the same three instructor raters: condition A is the proposed PKG/EduKG Graph RAG pipeline; condition B is a non-graph baseline that feeds the same LLM the same slide text and DNU concept for question generation and uses the same embedding model for top-k paragraph retrieval over the same Wikipedia corpus, but without PKG state, graph traversal, or graph-based re-ranking. Raters see questions and answers in randomized order without knowing the condition. If condition B scores within annotation noise of condition A (e.g., no significant difference, or a difference below 0.2 on the 1-3 scale), the graph-specific contribution is not supported and the manuscript should explicitly say so.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the proposed Graph RAG pipeline, specifically PKG-based Question Generation and EduKG-based Question Answering, empowers learners. For that claim to be supported, the evaluation must show that the graph and knowledge-graph components contribute beyond a plain LLM pipeline. Section 4.1 reports strong question-generation scores (weighted average 2.862 out of 3) from three instructors, but the generation prompt already includes the slide text, the slide concepts, and the DNU concept; the PKG and the graph-based re-ranking are not ablated, and there is no comparison against a non-graph baseline such as prompting GPT-3.5 with the same slide text alone. The high scores could therefore be entirely due to the underlying LLM and the carefully written prompt, not to the graph structure. Similarly, Section 4.2 reports only 45% answer accuracy, but this is not compared with plain Wikipedia retrieval or a no-RAG LLM; the 'Emergency Exit' disambiguation failure shows the EduKG did not resolve a basic ambiguity. Thus the paper's graph-specific contribution is not demonstrated, even though the authors honestly report limitations and do not overclaim QA effectiveness. The concern is about evaluation validity, not internal inconsistency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Graph RAG pipeline for the MOOC platform CourseMapper, consisting of two components: PKG-based Question Generation, which uses a learner's personal knowledge graph to prompt GPT-3.5-turbo to generate questions about concepts marked as \"Did Not Understand\" and then re-ranks them by semantic similarity to the slide text, and EduKG-based Question Answering, which retrieves Wikipedia paragraphs of the slide's main concepts (and, if needed, full Wikipedia articles of related concepts selected by an LLM) and extracts answers verbatim with citations. The system was evaluated by three expert instructors across three MOOCs. Question generation received a weighted average of 2.862 out of 3 on fluency, clarity, conciseness, and two relevance dimensions; answer accuracy was only 45% on average. The authors conclude that the pipeline shows \"potential\" for supporting learners, while acknowledging that question answering needs further improvement.","tokens_in":7633,"tokens_out":3624,"duration_ms":37688,"significance":"If substantiated, this work would address a real gap in RAG-based educational tools: actively guiding learners toward useful questions while grounding answers in structured knowledge. The paper has concrete strengths: it describes a full implementation in a real platform, evaluates with domain-expert instructors rather than crowdsourced workers, reports the limitations of the QA component honestly (e.g., the 'Emergency Exit' disambiguation failure), and is appropriately cautious in its conclusions. However, the evaluation as designed does not isolate the contribution of the graph components. Because there is no comparison against a plain LLM or standard RAG baseline, and no ablation of the graph-based re-ranking or graph-guided retrieval, the high question-generation scores and the 45% accuracy cannot be attributed to the Graph RAG pipeline specifically. The significance of the claimed contribution therefore remains to be established, though the paper is a reasonable proof-of-concept description.","major_comments":[{"comment":"The central claim that PKG-based Question Generation is effective is not supported by the evaluation design because there is no baseline or ablation. The QG prompt in Figure 3 (P1) already contains the slide text, the DNU concept, and the slide concepts, so the high scores (weighted average 2.862/3) could plausibly be produced by the same LLM prompted without the PKG or EduKG structures. To establish the graph-specific contribution, the authors should compare against (a) prompting GPT-3.5-turbo with only the slide text and the DNU concept, (b) the same prompt without the graph-based re-ranking step, and (c) the full pipeline, reporting scores for each condition with per-evaluator and per-DNU-concept breakdowns.","section":"Section 4.1, Table 1"},{"comment":"The QA evaluation reports only internal accuracy (45% weighted average) without any comparison to alternatives such as standard RAG over Wikipedia paragraphs, a no-RAG LLM, or a non-graph dense retriever. Without such a comparison, the paper's claim that EduKG-based Question Answering has \"potential\" is not empirically grounded. The 'Emergency Exit' example in Section 4.2 actually demonstrates that the EduKG did not resolve a basic ambiguity, so the graph-guided retrieval step appears to be a source of error rather than an improvement; an ablation or comparative study is needed to determine whether the EduKG relationships help or hurt retrieval.","section":"Section 4.2, Table 2"},{"comment":"The human evaluation lacks the statistical and methodological detail needed to support the strong question-generation results. Only three instructors evaluated the system, no inter-rater reliability metric is reported, no variance or standard deviations are given for the Table 1 scores, and the 1-to-3 scale is coarse. The authors should report per-item or per-DNU-concept distributions, compute agreement among evaluators, and either recruit more evaluators or explicitly frame the results as an exploratory pilot rather than evidence of effectiveness.","section":"Sections 4.1 and 4.2"},{"comment":"The main graph-specific mechanism in QA is the selection of related concepts (RCs) by an LLM prompt (P3) when the main-concept Wikipedia paragraphs do not contain an answer. No evidence is provided that this graph traversal improves retrieval over simply searching all Wikipedia articles of the RCs or all paragraphs in the EduKG. Since the paper's contribution is specifically Graph RAG, the RC-selection step should be ablated or compared with a non-graph baseline to show that the graph structure, rather than the LLM's general knowledge, is responsible for any retrieval benefit.","section":"Section 3.3, Figure 3"}],"minor_comments":[{"comment":"There is a typo in the extractive QA prompt template: \"Extracive QA prompt template\" should be \"Extractive QA prompt template\", and the question generation prompt contains \"insuffiecient\" instead of \"insufficient\".","section":"Figure 3"},{"comment":"The text says \"30 Question-Answer pairs\" and \"40 Question-Answer pairs\" were evaluated for question generation, but the evaluation dimensions apply to questions, not answers. Please clarify whether the evaluators also saw answers, or rename these counts as \"generated questions\".","section":"Section 4.1"},{"comment":"The \"Avg.\" row is a weighted average, but the weights (30, 40, 30 questions) are not stated in the table caption. Reporting standard deviations or per-DNU ranges would also help readers assess the stability of the 2.862 average.","section":"Table 1"},{"comment":"The term \"graph-based re-ranking\" is used for ranking questions by cosine similarity to the slide text. Since this ranking does not use graph structure, a more neutral name such as \"embedding-based re-ranking\" would avoid overstating the graph contribution.","section":"Section 3.2"},{"comment":"The EduKG construction relies entirely on Wikipedia articles and keyphrase extraction, but the paper does not discuss coverage or accuracy of this construction step. A sentence acknowledging that Wikipedia coverage may be incomplete for domain-specific MOOC concepts would strengthen the reader's understanding of the later QA accuracy results.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, clearly-written systems paper from the Duisburg group, and the authors do not oversell it. The new bit is the integration: using a learner PKG to generate personalized questions for a concept marked 'did not understand', re-ranking those questions by similarity to slide text, then answering with an EduKG over Wikipedia paragraphs and articles, with citations and highlighting. Individually the pieces exist, but this particular loop — PKG-driven QG plus EduKG-based QA — is not in the cited literature. That counts as a real, if incremental, contribution.\n\nWhat it does well: the user scenario is concrete, the prompt design is transparent, and the evaluation is honestly reported. The 45 percent answer accuracy is a bad number for the system but a good sign for the authors' credibility: they give plausible causes (abstract extractive answers, no disambiguation) instead of hiding it. The 'Emergency Exit' UX example is useful failure analysis. Self-citations to their own EduKG/PKG construction papers are appropriate; those are system components, not evidence for the new claim.\n\nWhere it falls short: the stress-test concern lands. There is no baseline. GPT-3.5 with the same slide text and a well-written prompt might produce questions that instructors score just as highly; the graph-guided retrieval and the re-ranking are never ablated. So the paper supports 'an LLM with a good prompt can generate relevant questions' but not 'the graph structure is what makes it work'. Same for QA: 45% accuracy is not compared against plain Wikipedia retrieval or a no-RAG LLM, so we do not know whether the EduKG helps or hurts. Also, three non-blinded instructors, no inter-rater reliability, no variance, and no learner participants — the QG scores of ~2.86/3 are promising but could shift a lot with more raters. The claim is hedged ('potential'), so there is no internal contradiction; the gap is between the data and the graph-specific attribution.\n\nWho it is for: researchers building RAG systems for education, especially those working with knowledge-graph-enhanced course materials. It deserves a serious referee. My recommendation: send it out, with the expectation of major revision — add at least one non-graph baseline, ablate the re-ranking, report agreement and variance, and ideally include learners. Release code and data if possible. If the authors do that, the paper would be a useful reference point for the field.","headline":"An honest early-stage Graph RAG system paper; the graph-specific benefit is plausible but not yet demonstrated because there is no baseline or ablation.","tokens_in":8185,"tokens_out":1790,"would_cite":true,"duration_ms":18262,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Graph RAG can guide MOOC learners to ask the right questions, but its answers are judged correct only 45 percent of the time.","keywords":["Educational Knowledge Graphs","Personal Knowledge Graphs","Graph Retrieval-Augmented Generation","Large Language Models","Question Generation","Question Answering","MOOC","Knowledge Concepts"],"falsifier":"Build a test set in which the answer to each question appears in the course slides but is deliberately absent from the linked Wikipedia articles; if the system answers from unrelated Wikipedia text or refuses, the EduKG-as-source assumption fails. A sharper test: take a set of ambiguous 'did not understand' concepts with known intended senses and count how often the retriever lands on the correct Wikipedia article; the 'Emergency Exit' failure suggests that rate will be far below 100 percent.","tokens_in":7190,"feed_emoji":"🎓","tokens_out":9527,"duration_ms":82958,"temperature":0.7,"pith_summary":"Massive open online courses leave learners without an instructor to ask, and generic LLM chatbots can hallucinate. The paper tries to establish that a retrieval-augmented system organized around knowledge graphs can fill this gap: a personalized graph records which concepts a learner marked as not understood, and that graph drives both the generation of practice questions and the retrieval of Wikipedia-based answers. In a study with three instructors over three courses, the generated questions scored 2.86 out of 3 on linguistic quality and relevance to the slide and concept, while the answers were judged correct only 45 percent of the time. The contribution is asymmetric: the evidence supports graph-guided question generation as an effective way to keep learners on-topic, while the paper's own numbers show graph-guided answering needs more work before it can be trusted.","feed_headline":"Knowledge graphs guide MOOC learners to the right questions","feed_subtitle":"Question generation scores 2.86 out of 3, but answers are right only 45 percent of the time.","key_machinery":"The load-bearing object is the knowledge graph itself, in two forms. The EduKG is built automatically from each learning material: slides are linked to main concepts extracted by a keyphrase algorithm, each main concept is tagged with a Wikipedia article, and related concepts are drawn from other articles referenced inside that article; all of this is stored in a Neo4j graph database. The PKG is the learner-specific slice of that graph, recording which main concepts the learner marked as 'did not understand.' Question generation queries the PKG for the DNU concept, the slide text, and the slide's other main concepts, feeds them to a zero-shot prompt for GPT-3.5-turbo, and re-ranks the generated questions by cosine similarity between question embeddings and the slide-text embedding. Question answering indexes Wikipedia paragraphs of the main concepts in a graph vector store, retrieves the paragraphs most similar to the selected question, and routes them through an extractive QA prompt; if no answer emerges, an LLM retriever traverses the graph to pick related concepts whose full Wikipedia articles are then used. This graph structure is what converts unstructured slide material into a retrievable, personalized learning context.","core_discovery":"The central claim is that a Graph RAG pipeline built on an educational knowledge graph—whose nodes are course slides, main concepts, related concepts, and learners—can give MOOC learners a structured way to understand new concepts. When a learner marks a concept as 'did not understand' in CourseMapper, the system generates a set of personalized questions using the slide text and neighboring concepts, re-ranks them by similarity to the slide, and presents them to the learner. When the learner picks a question, the system retrieves the most similar Wikipedia paragraphs of the slide's main concepts and, as a fallback, full Wikipedia articles of related concepts selected by an LLM, then extracts an answer verbatim with a citation. The authors conclude that this reveals the potential of Graph RAG for personalized learning: question generation was perceived as effective at guiding learners to ask the right questions in context, while answer accuracy (45 percent weighted across 100 question-answer pairs) still requires further enhancement.","pith_inferences":["The gap between question quality (2.86 out of 3) and answer accuracy (45 percent) suggests the bottleneck is retrieval and disambiguation, not generation; a version that answers from a curated course corpus or from the slide text itself might close much of the gap.","At 45 percent accuracy, a deployed system would need an instructor or human moderator in the loop; otherwise learners could receive confidently worded, cited-but-wrong answers.","The graph-based re-ranking step compares question embeddings only to the slide-text embedding, so it may systematically prefer questions that paraphrase the slide and under-weight questions that reach beyond it; this is a testable prediction for future work."],"forward_implications":["Learners who mark a concept as not understood will receive questions tied to that concept and to the slide they are reading, rather than generic questions, because the PKG supplies that context.","The evaluation indicates the question generation component produces fluent, clear, and concise questions that instructors rated as relevant to both the slide (2.853) and the target concept (2.668) on a 1 to 3 scale.","Answers come with citations and source highlighting, so learners have a path to read beyond the generated response.","The weighted answer accuracy of 45 percent defines the current ceiling for trusting the answering component in a real MOOC, and the authors state that it requires enhancement.","The authors propose two future directions: enrich the EduKG with sources beyond Wikipedia, and allow the LLM to reason over evidence rather than only extract from it."],"supporting_citations":[{"why":"Supplies the idea that an EduKG can link course concepts to external learning resources usable as a RAG data source.","marker":"[1]"},{"why":"Provides the word-embedding-based method for automatic construction of the EduKG entities used in the pipeline.","marker":"[2]"},{"why":"Describes the CourseMapper platform in which the EduKG, PKG, and the proposed pipeline are implemented.","marker":"[3]"},{"why":"Introduces learner modeling with personal knowledge graphs, the basis for marking concepts as not understood.","marker":"[4]"},{"why":"Supplies the seven-dimensional question generation evaluation framework from which the five evaluation dimensions are drawn.","marker":"[6]"},{"why":"Defines accuracy as correct or incorrect for a RAG-based educational platform, the measure the paper aligns with.","marker":"[9]"},{"why":"Another RAG-in-education system whose accuracy definition anchors the evaluation.","marker":"[10]"},{"why":"Defines the three-step Graph RAG paradigm (graph-based indexing, graph-guided retrieval, graph-enhanced generation) that the pipeline adopts.","marker":"[12]"}],"fun_headline_variants":["Graph RAG asks the right questions for MOOC learners","Personalized graph RAG questions for MOOC learners","Graph RAG generates questions, but answers lag in MOOCs","Graph RAG: right questions, shaky answers","Graph RAG for MOOCs: right questions, weak answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach depends on the Wikipedia articles attached to course concepts containing the text needed to answer learners' questions, and on the system automatically picking the right meaning when a concept name has several meanings, such as 'Emergency Exit' in user experience rather than in building safety. If either part fails, retrieval supplies the wrong context and the extractive answer can be confidently wrong.","fun_headline_variants_meta":{"raw":{"variants":["Graph RAG asks the right questions for MOOC learners","Personalized graph RAG questions for MOOC learners","Graph RAG generates questions, but answers lag in MOOCs","Graph RAG: right questions, shaky answers","Graph RAG for MOOCs: right questions, weak answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000867,"raw_usage":{"total_tokens":3775,"prompt_tokens":978,"completion_tokens":2797,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":2721}},"tokens_in":594,"tokens_out":2797,"duration_ms":19297,"temperature":1.0,"reasoning_tokens":2721,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:16:51.935100+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test set in which the answer to each question appears in the course slides but is deliberately absent from the linked Wikipedia articles; if the system answers from unrelated Wikipedia text or refuses, the EduKG-as-source assumption fails. A sharper test: take a set of ambiguous 'did not understand' concepts with known intended senses and count how often the retriever lands on the correct Wikipedia article; the 'Emergency Exit' failure suggests that rate will be far below 100 percent.","supporting_citations":[{"cited_title":"LLM-Assisted Knowledge Graph Completion for Curriculum and Domain Modelling in Personalized Higher Education Recommendations","cited_arxiv_id":"2501.12300","evidence_quote":"Supplies the idea that an EduKG can link course concepts to external learning resources usable as a RAG data source."},{"cited_title":"Information 14(10), 526 (2023)","cited_arxiv_id":null,"evidence_quote":"Provides the word-embedding-based method for automatic construction of the EduKG entities used in the pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces learner modeling with personal knowledge graphs, the basis for marking concepts as not understood."},{"cited_title":"In: International Conference on Artificial Intelligence in Education","cited_arxiv_id":null,"evidence_quote":"Defines accuracy as correct or incorrect for a RAG-based educational platform, the measure the paper aligns with."},{"cited_title":"In: Proceedings of the 55th ACM Technical Symposium on Computer Science Ed- ucation V","cited_arxiv_id":null,"evidence_quote":"Another RAG-in-education system whose accuracy definition anchors the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the three-step Graph RAG paradigm (graph-based indexing, graph-guided retrieval, graph-enhanced generation) that the pipeline adopts."}],"review_version":1}