{"id":"a156b927-fd70-4f72-91e6-658833586d5b","arxiv_id":"2501.12300","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An LLM-assisted, teacher-validated pipeline builds a curriculum, domain, and user knowledge graph from two embedded-systems modules, with modest evaluation evidence.","lead":"This paper builds a knowledge graph for university courses by using large language models to extract and classify topics from lecture materials, with teachers checking the results. Its goal is to let universities connect similar course content across different programs and eventually recommend personalized learning paths for students.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that the pipeline 'enhances the ability to connect related courses' is not supported by any evaluation of the correctness of the semantic cross-module edges; ADC and modularity are structural by-products of adding edges, not evidence of semantic quality.","rationale":"The paper is an applied systems paper; its strongest claim is that LLM-assisted graph completion enhances the ability to connect related courses across disciplines. For that to hold, the semantic edges between courses must be (a) correct, (b) noticeably better than a cheap baseline, and (c) useful for personalization. The pipeline and human-in-the-loop design are clear, and the extraction precision/recall numbers in Table 1 are valuable evidence that topic and sub-topic classification is workable. However, these numbers concern node extraction and classification, not the cross-module edges that are the actual vehicle for the claimed contribution. The only quantitative evidence about edges is ADC and modularity, which are structural properties: any added edge improves ADC, and cross-module edges normally lower modularity, so the observed changes are partly definitional. The paper also mentions a random-sample teacher validation of relations but omits its results. Without a correctness measure for the semantic relations or a baseline comparison, the central claim is under-supported, even though the system may work well in practice. The visual-content assumption flagged by the reader is real, but the more load-bearing gap is the lack of validation of the semantic relations themselves; even perfect text extraction would not establish the cross-course connection claim if the similarity edges were spurious. I therefore agree partly with the reader's diagnosis, and the existing CONDITIONAL verdict remains appropriate: the paper should be accepted only with the addition of edge-level validation and an independent or baseline comparison.","tokens_in":7381,"tokens_out":3935,"duration_ms":40990,"concrete_test":"Extract the complete set of cross-module semantic Topic/Sub-Topic edges added between the Embedded Systems and FPGA modules, and have two independent domain experts (not involved in the pipeline design) rate each edge as 'related,' 'unrelated,' or 'uncertain' using the original lecture materials. Compute inter-annotator agreement and edge precision; then compare with a baseline where edges are generated by a simple lexical overlap or embedding similarity without LLM-assisted extraction. If edge precision is not substantially above baseline or chance, the structural ADC/modularity changes cannot support the cross-discipline connection claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The KG-completion claim rests on the semantic Topic/Sub-Topic relations added between the two modules, but Section 4.2 evaluates only graph-level structural metrics: average degree centrality rises from 0.90 to 1.03 and modularity falls from 0.769 to 0.767. These metrics are expected consequences of inserting a small number of cross-module similarity edges; they do not measure whether those edges represent genuinely related course content. The paper mentions semi-supervised validation of a random sample of relations in Section 3.4, yet reports no results from that validation, no precision/recall for the semantic links, and no comparison to a non-LLM or lexical baseline. Thus the quantitative evidence cannot distinguish a truly useful semantic bridge from randomly or spuriously added edges. The qualitative expert comments are suggestive, but they come from the lecturers of the same two modules (Section 4.1), so they do not independently establish that cross-disciplinary connection quality is high. Unless the semantic relations themselves are correct and useful, the structural improvement is largely definitional and the abstract's claim about enhancing the ability to connect related courses is not yet demonstrated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an LLM-assisted knowledge graph completion pipeline for higher education, built on an ontology with curriculum, domain, and user models. The pipeline transcribes lecture videos with Whisper, extracts topics and sub-topics with GPT-4o, and has teachers validate the extractions. The implementation covers two embedded-systems modules, and the evaluation combines expert precision/recall/F1 on 1197 extraction samples, graph-structural metrics (average degree centrality and modularity), and qualitative lecturer feedback. The central claim is that the approach enhances the ability to connect related courses across disciplines for personalized learning, but the quantitative support for that specific claim is limited to structural changes in the constructed graph.","tokens_in":7678,"tokens_out":4024,"duration_ms":39198,"significance":"If validated, the contribution is relevant and useful: it addresses teacher workload in curriculum modelling, integrates domain and user models, and provides an ontology-based, human-in-the-loop approach for KG construction. The manuscript's strengths include a concrete pipeline description, a clear statement of the visual-content assumption, a non-trivial expert-labeled sample for extraction quality, and a reproducible-looking experimental design. However, the paper's headline result about cross-module semantic linking is not yet supported because the semantic relations themselves are not evaluated.","major_comments":[{"comment":"The quantitative evidence for the central claim is the ADC increase from 0.90 to 1.03 and the modularity decrease from 0.769 to 0.767. These measures are expected consequences of adding a small number of edges: degree centrality rises mechanically, and modularity falls when cross-community edges are inserted. They do not show that the added edges are semantically correct or useful. The manuscript mentions semi-supervised validation of semantic similarities in Section 3.4 but reports no results for that validation, no precision/recall for the semantic relations, no baseline comparison, and no random-edge control. Without such evidence, the abstract's claim that the approach enhances the ability to connect related courses across disciplines is not demonstrated.","section":"Section 4.2"},{"comment":"The evaluators are the module lecturers from the same institute as several co-authors. Because they are also the content providers and participants in the collaborative design, their qualitative acceptance does not independently validate the ontology, the pipeline, or the cross-module connections. The quantitative extraction results also rely on their judgments, so the evaluation mixes self-assessment with independent validation. The paper should include at least some external evaluators who are not involved in developing the modules, or otherwise should clearly separate self-assessment from external validation.","section":"Section 4.1"},{"comment":"The assumption that figures, images, and equations in lecture slides are annotated or described textually is central to the transcription and extraction pipeline; if engineering content is largely visual, the extracted KG will be incomplete despite high text-based scores. The paper does not test this assumption or report the proportion of visual content in the evaluated modules. Moreover, the two evaluation modules are both in embedded systems, so the claim of across-discipline connection is not covered by the empirical scope. The authors should either report visual-content coverage and broaden the evaluation, or soften the claim accordingly.","section":"Section 3.2"}],"minor_comments":[{"comment":"The high similarity thresholds mentioned in Section 4.2 are not quantified; reporting the exact thresholds and the prompt templates used for semantic relation extraction would improve reproducibility.","section":"Section 3.4"},{"comment":"The decimal separator is inconsistent (0,99 vs. 0.97) and the sample sizes per class are not shown in the table; the text gives totals but not class-wise counts.","section":"Table 1"},{"comment":"The corresponding-author email in the footnote appears corrupted ('/envel⌢pe-⌢penhasan...') and should be corrected.","section":"Footnote"},{"comment":"The terms knowledge graph completion and knowledge graph construction are used interchangeably; the relationship between the baseline graph and the completed graph should be made explicit.","section":"Abstract and Section 3.4"},{"comment":"Figure 3 would be clearer with an explicit statement of node and edge counts before and after completion, beyond the ADC and modularity values.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The main concern is the mismatch between the abstract's claim and the evidence in Section 4.2. A revision that evaluates the semantic edges directly (e.g., precision/recall on a teacher-validated sample of relations, with a non-LLM baseline) and adds external evaluators would materially strengthen the paper. The extraction quality results are good and worth publishing once the connection claim is supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful applied paper, not a conceptual breakthrough. The genuinely new part is the integrated pipeline—Whisper transcription, GPT-4o extraction/classification, teacher validation, and semantic linking—applied to two real modules in the same research program as EduCOR. The authors are transparent about the pipeline, the assumption that visual content in slides is textually annotated, and the small scale. The extraction precision/recall numbers (P/R/F1 mostly in the mid-0.9s) are plausible and, importantly, were produced with teacher validation on 1,197 samples. That is real work and it supports the narrower claim that LLMs give teachers a good starting point for structuring lecture content.\n\nThe soft spots are real but not fatal. First, the central claim about 'enhancing the ability to connect related courses' is not directly evaluated. Section 4.2 reports average degree centrality rising from 0.90 to 1.03 and modularity falling from 0.769 to 0.767. Those are structural by-products of adding a handful of cross-module edges; they don't tell you whether the added edges correspond to genuinely related content. Second, Section 3.4 promises semi-supervised validation of a random sample of predicted relations, but the paper reports no results from that validation. That is exactly the evidence needed to support the connection claim, and it's missing. Third, the expert evaluators are the lecturers of the same two modules from the same institute; their qualitative feedback is useful but not independent. No baseline comparison (e.g., lexical similarity or a non-LLM extractor) is given, so it's hard to quantify what the LLM adds beyond existing NLP pipelines. Fourth, the small scale (two modules, nine sessions each) is acknowledged but limits generality.\n\nI don't think these issues sink the paper. The extraction and classification part is reasonably demonstrated, and the authors are honest about limitations, even mentioning the video-recording style problem and prompt engineering fixes. As a systems description for the education technology community, it deserves serious referee time. But the title-level claim of KG completion and cross-course linking needs direct edge-level evaluation, baselines, and ideally a released artifact before it can be taken as demonstrated.","headline":"A solid, clearly described human-in-the-loop pipeline for LLM-based curriculum KG building, but the headline claim about cross-course linking rests on metrics that don't directly test whether the semantic edges are correct.","tokens_in":8171,"tokens_out":1864,"would_cite":true,"duration_ms":18307,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an LLM-assisted, teacher-validated pipeline can extract fine-grained topics from lecture materials and link related courses across disciplines in a knowledge graph, supporting personalized higher-education learning…","keywords":["Higher education","Knowledge graph","Large language models","Curriculum model","Domain model","User model","Learning personalization","Human-AI collaboration"],"falsifier":"Feed the pipeline a course whose slides convey core concepts mainly through diagrams and equations with little textual annotation, then have the same teachers judge the extracted topics; if central figure-only concepts are missing from the validated graph, the assumption fails.","tokens_in":7204,"feed_emoji":"🎓","tokens_out":5541,"duration_ms":52938,"temperature":0.7,"pith_summary":"This paper claims that large language models and human teachers can collaborate to build a knowledge graph of university course content that links topics across modules and disciplines. The authors propose an ontology with curriculum, domain, and user models, use an LLM to extract fine-grained topics and sub-topics from lecture slides, manuscripts, and video transcripts, and have teachers validate the results. Applied to two embedded-systems courses, the pipeline achieved high precision, recall, and F1 scores on topic and sub-topic extraction and added semantic cross-course links to the graph. The authors argue this supports personalized learning-path recommendations by connecting related content across courses.","feed_headline":"LLM pipeline maps two engineering courses into one knowledge graph","feed_subtitle":"Teacher-validated topic extraction scored high precision and found cross-course links for personalized learning paths.","key_machinery":"The carrying mechanism is the three-model ontology—curriculum model, domain model, user model—with Topic and Sub-Topic as the core taxonomic classes. The pipeline feeds lecture slides, manuscripts, and video transcripts (transcribed by a speech recognizer) to an LLM prompted with ontology definitions so it can extract and classify concepts; teachers then validate or correct the extraction, and the validated titles and descriptions feed a semantic-similarity step that creates cross-course relations. The ontology keeps the extracted content interoperable across faculties and institutions, and the KG in turn becomes context for later LLM extraction.","core_discovery":"On its own terms, the paper establishes that a semi-automated, teacher-validated pipeline can produce a curriculum knowledge graph that domain experts accept. On two embedded-systems modules the LLM extraction achieved topic precision of 0.99 and 0.97, sub-topic precision of 1.00 and 0.89, and recall and F1 values in the same high range across classes; the automatically generated sub-topic descriptions also scored highly on precision. Graph-level structural measures moved in the intended direction after semantic links joined the two modules: average degree centrality rose from 0.9 to 1.03 and modularity fell slightly from 0.769 to 0.767. The lecturers reported that the extracted topics and cross-course similarities helped them restructure content and would support personalized learning paths for students with different backgrounds.","pith_inferences":["Beyond the paper, the same pipeline could be pointed at assessment data, using the user-model parameters to recommend topic-level remediation rather than whole courses.","The reported graph-metric change is small; a stronger test of the linking idea would be a larger multi-course graph where semantic relations connect more than two modules, or a lower similarity threshold.","The explicit assumption that figures and equations are textually described suggests a natural extension to multimodal LLMs that caption diagrams directly; that extension is testable on the same two modules.","Whether the approach saves teacher time is the practical crux; a before-and-after measurement of validation hours would quantify the collaboration benefit that the qualitative feedback supports."],"forward_implications":["Teachers can use the cross-course links to spot overlapping content and restructure their modules, reducing duplication for students who take both.","Students with prior knowledge in one domain can receive learning paths that skip already-covered topics and connect to related modules from other faculties.","The semi-supervised validation scheme, where teachers check a random sample of predicted relations, lets the approach scale as the graph grows.","The KG doubles as context for later LLM extraction, which should improve accuracy when new lectures resemble content already in the graph.","Prompt engineering rules that distinguish topics from sub-topics and ignore side discussions are necessary for clean extraction from live recordings."],"supporting_citations":[{"why":"Motivates the problem by noting teachers lack time to analyze course content for personalization.","marker":"[1]"},{"why":"Supplies the NLP semantic-similarity pipeline used to find related topics across lectures.","marker":"[5]"},{"why":"Supports using LLMs for ontology and knowledge graph construction rather than fully manual expert work.","marker":"[7]"},{"why":"Provides prior evidence that LLMs can create educational knowledge graphs from course materials, which this paper extends to lecture-level content.","marker":"[8]"},{"why":"Shows that knowledge graphs can serve as context sources for LLM-based learning recommendations, motivating the KG's role.","marker":"[9]"},{"why":"Supplies the adopted ontological classes for the user model, including academic parameters.","marker":"[12]"},{"why":"Provides the speech-recognition model used to transcribe lecture videos for topic extraction.","marker":"[13]"}],"fun_headline_variants":["LLM pipeline hits 0.99 topic precision in course knowledge graph","Teachers validate LLM-extracted topics at 0.99 precision for learning paths","LLM-expert graph completion links courses with 0.99 precision","Semi-automated curriculum graph: 0.99 precision, expert-approved"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that figures, images, and equations in lecture slides are annotated or described in text, so the transcription and LLM extraction can capture the real teaching content.","fun_headline_variants_meta":{"raw":{"variants":["LLM pipeline hits 0.99 topic precision in course knowledge graph","Teachers validate LLM-extracted topics at 0.99 precision for learning paths","LLM-expert graph completion links courses with 0.99 precision","Semi-automated curriculum graph: 0.99 precision, expert-approved"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1524,"prompt_tokens":949,"completion_tokens":575,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":492}},"tokens_in":565,"tokens_out":575,"duration_ms":6168,"temperature":1.0,"reasoning_tokens":492,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:18:28.895155+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed the pipeline a course whose slides convey core concepts mainly through diagrams and equations with little textual annotation, then have the same teachers judge the extracted topics; if central figure-only concepts are missing from the validated graph, the assumption fails.","supporting_citations":[{"cited_title":"Meissner, L","cited_arxiv_id":null,"evidence_quote":"Motivates the problem by noting teachers lack time to analyze course content for personalization."},{"cited_title":"Abu-Rasheed, M","cited_arxiv_id":null,"evidence_quote":"Supplies the NLP semantic-similarity pipeline used to find related topics across lectures."},{"cited_title":"Jhajj, X","cited_arxiv_id":null,"evidence_quote":"Provides prior evidence that LLMs can create educational knowledge graphs from course materials, which this paper extends to lecture-level content."},{"cited_title":"Ilkou, H","cited_arxiv_id":null,"evidence_quote":"Supplies the adopted ontological classes for the user model, including academic parameters."}],"review_version":1}