REVIEW 3 major objections 5 minor 13 references
LLM-Assisted Knowledge Graph Completion for Curriculum and Domain Modelling in Personalized Higher Education Recommendations
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper claims that an LLM-assisted, teacher-validated pipeline can extract fine-grained topics from lecture materials and link related courses across disciplines in a knowledge graph, supporting personalized higher-education learning…
desk verdict A solid, clearly described human-in-the-loop pipeline for LLM-based curriculum KG building, but the headline claim about cross-course linking rests on metrics that don't directly test whether the semantic edges are correct. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the three-model ontology—curriculum model, domain model, user model—with Topic and Sub-Topic as the core taxonomic classes. The pipeline feeds lecture slides, manuscripts, and video transcripts (transcribed by a speech recognizer) to an LLM prompted with ontology definitions so it can extract and classify concepts; teachers then validate or correct the extraction, and the validated titles and descriptions feed a semantic-similarity step that creates cross-course relations. The ontology keeps the extracted content interoperable across faculties and institutions, and the KG in turn becomes context for later LLM extraction.
What would settle it
Feed the pipeline a course whose slides convey core concepts mainly through diagrams and equations with little textual annotation, then have the same teachers judge the extracted topics; if central figure-only concepts are missing from the validated graph, the assumption fails.
Extended reading notes
Core claim
On its own terms, the paper establishes that a semi-automated, teacher-validated pipeline can produce a curriculum knowledge graph that domain experts accept. On two embedded-systems modules the LLM extraction achieved topic precision of 0.99 and 0.97, sub-topic precision of 1.00 and 0.89, and recall and F1 values in the same high range across classes; the automatically generated sub-topic descriptions also scored highly on precision. Graph-level structural measures moved in the intended direction after semantic links joined the two modules: average degree centrality rose from 0.9 to 1.03 and modularity fell slightly from 0.769 to 0.767. The lecturers reported that the extracted topics and cross-course similarities helped them restructure content and would support personalized learning paths for students with different backgrounds.
Load-bearing premise
The pipeline assumes that figures, images, and equations in lecture slides are annotated or described in text, so the transcription and LLM extraction can capture the real teaching content.
Editorial extensions
If this is right
- Teachers can use the cross-course links to spot overlapping content and restructure their modules, reducing duplication for students who take both.
- Students with prior knowledge in one domain can receive learning paths that skip already-covered topics and connect to related modules from other faculties.
- The semi-supervised validation scheme, where teachers check a random sample of predicted relations, lets the approach scale as the graph grows.
- The KG doubles as context for later LLM extraction, which should improve accuracy when new lectures resemble content already in the graph.
- Prompt engineering rules that distinguish topics from sub-topics and ignore side discussions are necessary for clean extraction from live recordings.
Reading between the lines
- Beyond the paper, the same pipeline could be pointed at assessment data, using the user-model parameters to recommend topic-level remediation rather than whole courses.
- The reported graph-metric change is small; a stronger test of the linking idea would be a larger multi-course graph where semantic relations connect more than two modules, or a lower similarity threshold.
- The explicit assumption that figures and equations are textually described suggests a natural extension to multimodal LLMs that caption diagrams directly; that extension is testable on the same two modules.
- Whether the approach saves teacher time is the practical crux; a before-and-after measurement of validation hours would quantify the collaboration benefit that the qualitative feedback supports.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents an LLM-assisted knowledge graph completion pipeline for higher education, built on an ontology with curriculum, domain, and user models. The pipeline transcribes lecture videos with Whisper, extracts topics and sub-topics with GPT-4o, and has teachers validate the extractions. The implementation covers two embedded-systems modules, and the evaluation combines expert precision/recall/F1 on 1197 extraction samples, graph-structural metrics (average degree centrality and modularity), and qualitative lecturer feedback. The central claim is that the approach enhances the ability to connect related courses across disciplines for personalized learning, but the quantitative support for that specific claim is limited to structural changes in the constructed graph.
Significance. If validated, the contribution is relevant and useful: it addresses teacher workload in curriculum modelling, integrates domain and user models, and provides an ontology-based, human-in-the-loop approach for KG construction. The manuscript's strengths include a concrete pipeline description, a clear statement of the visual-content assumption, a non-trivial expert-labeled sample for extraction quality, and a reproducible-looking experimental design. However, the paper's headline result about cross-module semantic linking is not yet supported because the semantic relations themselves are not evaluated.
major comments (3)
- [Section 4.2] The quantitative evidence for the central claim is the ADC increase from 0.90 to 1.03 and the modularity decrease from 0.769 to 0.767. These measures are expected consequences of adding a small number of edges: degree centrality rises mechanically, and modularity falls when cross-community edges are inserted. They do not show that the added edges are semantically correct or useful. The manuscript mentions semi-supervised validation of semantic similarities in Section 3.4 but reports no results for that validation, no precision/recall for the semantic relations, no baseline comparison, and no random-edge control. Without such evidence, the abstract's claim that the approach enhances the ability to connect related courses across disciplines is not demonstrated.
- [Section 4.1] The evaluators are the module lecturers from the same institute as several co-authors. Because they are also the content providers and participants in the collaborative design, their qualitative acceptance does not independently validate the ontology, the pipeline, or the cross-module connections. The quantitative extraction results also rely on their judgments, so the evaluation mixes self-assessment with independent validation. The paper should include at least some external evaluators who are not involved in developing the modules, or otherwise should clearly separate self-assessment from external validation.
- [Section 3.2] The assumption that figures, images, and equations in lecture slides are annotated or described textually is central to the transcription and extraction pipeline; if engineering content is largely visual, the extracted KG will be incomplete despite high text-based scores. The paper does not test this assumption or report the proportion of visual content in the evaluated modules. Moreover, the two evaluation modules are both in embedded systems, so the claim of across-discipline connection is not covered by the empirical scope. The authors should either report visual-content coverage and broaden the evaluation, or soften the claim accordingly.
minor comments (5)
- [Section 3.4] The high similarity thresholds mentioned in Section 4.2 are not quantified; reporting the exact thresholds and the prompt templates used for semantic relation extraction would improve reproducibility.
- [Table 1] The decimal separator is inconsistent (0,99 vs. 0.97) and the sample sizes per class are not shown in the table; the text gives totals but not class-wise counts.
- [Footnote] The corresponding-author email in the footnote appears corrupted ('/envel⌢pe-⌢penhasan...') and should be corrected.
- [Abstract and Section 3.4] The terms knowledge graph completion and knowledge graph construction are used interchangeably; the relationship between the baseline graph and the completed graph should be made explicit.
- [Figure 3] Figure 3 would be clearer with an explicit statement of node and edge counts before and after completion, beyond the ADC and modularity values.
Circularity Check
Headline graph metric is a by-product of edge addition; extraction evaluation is non-circular, but semantic-edge quality is not independently measured.
-
self definitional
[Section 4.2, Figure 3]
"In the resulting KG, see Figure 3, we calculate an increase of the ADC value from 0.9 to 1.03, as well as a slight modularity decrease from 0.769 to 0.767. Figure 3: The structure of the KG before (left) and after (right) connecting both evaluation modules through semantic Topic and Sub-Topic relations."
ADC is average degree centrality, i.e., the mean number of relations per node. The 'after' graph is constructed by inserting semantic Topic/Sub-Topic relations between the two modules (Figure 3 caption); adding edges increases average degree by definition. The reported ADC rise from 0.90 to 1.03 therefore measures the edge-addition operation itself, not whether the inserted edges represent genuine relatedness or improve learning-path recommendations. The modularity drop of 0.002 is likewise a direct, small consequence of adding a few cross-module edges. The qualitative lecturer feedback provides some independent evidence, but the quantitative 'graph quality' validation does not independently confirm the cross-disciplinary connection claim.
full rationale
The paper's topic/sub-topic extraction results (Table 1) are compared against expert labels and are not circular. The central 'graph completion enhances connection' claim, however, is partly supported by an ADC increase that is definitionally produced by the edge-addition step of graph construction; this is a self-fulfilling metric. The modularity decrease is also a mechanical consequence of adding edges. Qualitative expert feedback (Section 4.1) offers non-circular, though small, corroboration; the semi-supervised validation of semantic relations mentioned in Section 3.4 is not reported, so the semantic quality of the added edges is not independently established. Self-citations [5] and [12] are used as method/background references rather than as authority for the central result, so they are not load-bearing circularity. Overall, the majority of the pipeline is self-contained, but the headline quantitative evidence for the cross-disciplinary-connection claim is partly definitional; hence a score of 4.
Assumptions & free parameters
free parameters (2)
- Semantic similarity threshold(s) for Topic and Sub-Topic relations =
Not reported in the paper.
- LLM prompt rules and ontology definitions for GPT-4o extraction =
Not reported in the paper.
assumptions (4)
- domain assumption Figures, images, and equations in lecture slides are annotated or described textually.
- ad hoc to paper The Topic/Sub-Topic hierarchy is a sufficient and unambiguous representation of curriculum content across diverse courses and institutions.
- domain assumption Whisper and GPT-4o perform extraction and classification with the accuracy and stability implied by the evaluation.
- domain assumption The three expert evaluators provide accurate, unbiased judgments of extraction quality.
invented entities (2)
-
Higher-education ontology classes (Topic, Sub-Topic, Session, Lecture, Domain, Sub-Domain, User model)
-
Automatically generated sub-topic descriptions
Cite this review
Pith. "Pith review of LLM-Assisted Knowledge Graph Completion for Curriculum and Domain Modelling in Personalized Higher Education Recommendations." pith.science (2026). https://pith.science/paper/EETUHOZL
@misc{pith2026250112300,
author = {Pith},
title = {Pith review of: LLM-Assisted Knowledge Graph Completion for Curriculum and Domain Modelling in Personalized Higher Education Recommendations},
year = {2026},
howpublished = {\url{https://pith.science/paper/EETUHOZL}},
note = {Machine review of arXiv:2501.12300}
}
read the original abstract
While learning personalization offers great potential for learners, modern practices in higher education require a deeper consideration of domain models and learning contexts, to develop effective personalization algorithms. This paper introduces an innovative approach to higher education curriculum modelling that utilizes large language models (LLMs) for knowledge graph (KG) completion, with the goal of creating personalized learning-path recommendations. Our research focuses on modelling university subjects and linking their topics to corresponding domain models, enabling the integration of learning modules from different faculties and institutions in the student's learning path. Central to our approach is a collaborative process, where LLMs assist human experts in extracting high-quality, fine-grained topics from lecture materials. We develop a domain, curriculum, and user models for university modules and stakeholders. We implement this model to create the KG from two study modules: Embedded Systems and Development of Embedded Systems Using FPGA. The resulting KG structures the curriculum and links it to the domain models. We evaluate our approach through qualitative expert feedback and quantitative graph quality metrics. Domain experts validated the relevance and accuracy of the model, while the graph quality metrics measured the structural properties of our KG. Our results show that the LLM-assisted graph completion approach enhances the ability to connect related courses across disciplines to personalize the learning experience. Expert feedback also showed high acceptance of the proposed collaborative approach for concept extraction and classification.
Figures
Reference graph
Works this paper leans on
-
[1]
R. Meissner, L. Köbis, Annotated Knowledge Graphs for Teaching in Higher Education: Supporting Mentors and Mentees by Digital Systems, in: M. Bielikova, T. Mikkonen, C. Pautasso (Eds.), Web Engineering, volume 12128, Springer International Publishing, Cham, 2020, pp. 551–555. URL: http: //link.springer.com/10.1007/978-3-030-50578-3_43. doi: 10.1007/978-3-...
-
[2]
N. Ibrahim, S. Aboulela, A. Ibrahim, R. Kashef, A survey on augmenting knowledge graphs (KGs) with large language models (LLMs): models, evaluation metrics, benchmarks, and challenges, Dis- cover Artificial Intelligence 4 (2024) 76. URL: https://link.springer.com/10.1007/s44163-024-00175-8. doi:10.1007/s44163-024-00175-8
-
[3]
H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, A. Mian, A Comprehensive Overview of Large Language Models, 2023. URL: https://arxiv.org/abs/2307.06435. doi:10.48550/ARXIV.2307.06435, version Number: 10
-
[4]
S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, X. Wu, Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023). URL: https://arxiv.org/abs/2306.08302. doi:10.48550/ARXIV.2306. 08302, publisher: arXiv Version Number: 3
-
[5]
H. Abu-Rasheed, M. Dornhöfer, C. Weber, G. Kismihók, U. Buchmann, M. Fathi, Building Con- textual Knowledge Graphs for Personalized Learning Recommendations Using Text Mining and Semantic Graph Completion, in: 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), IEEE, Orem, UT, USA, 2023, pp. 36–40. URL: https://ieeexplore.ieee.or...
-
[6]
Kejriwal, Knowledge Graphs: A Practical Review of the Research Landscape, Information 13 (2022) 161
M. Kejriwal, Knowledge Graphs: A Practical Review of the Research Landscape, Information 13 (2022) 161. URL: https://www.mdpi.com/2078-2489/13/4/161. doi:10.3390/info13040161
-
[7]
V. K. Kommineni, B. König-Ries, S. Samuel, From human experts to machines: An LLM supported approach to ontology and knowledge graph construction, 2024. URL: https://arxiv.org/abs/2403. 08345. doi:10.48550/ARXIV.2403.08345, version Number: 1
-
[8]
G. Jhajj, X. Zhang, J. R. Gustafson, F. Lin, M. P.-C. Lin, Educational Knowledge Graph Creation and Augmentation via LLMs, in: A. Sifaleras, F. Lin (Eds.), Generative Intelligence and Intelligent Tutoring Systems, volume 14799, Springer Nature Switzerland, Cham, 2024, pp. 292–304. URL: https://link.springer.com/10.1007/978-3-031-63031-6_25. doi: 10.1007/9...
Show all 13 references
-
[9]
Abu-Rasheed, C
H. Abu-Rasheed, C. Weber, M. Fathi, Knowledge Graphs as Context Sources for LLM-Based Expla- nations of Learning Recommendations, in: 2024 IEEE Global Engineering Education Conference (EDUCON), IEEE, Kos Island, Greece, 2024, pp. 1–5. URL: https://ieeexplore.ieee.org/document/...
2024
-
[10]
Stancin, P
K. Stancin, P. Poscic, D. Jaksic, Ontologies in education – state of the art, Education and Information Technologies 25 (2020) 5301–5320. URL: https://link.springer.com/10.1007/s10639-020-10226-z. doi:10.1007/s10639-020-10226-z
2020 doi
-
[11]
Katis, H
E. Katis, H. Kondylakis, G. Agathangelos, K. Vassilakis, Developing an Ontology for Curriculum and Syllabus, in: A. Gangemi, A. L. Gentile, A. G. Nuzzolese, S. Rudolph, M. Maleshkova, H. Paul- heim, J. Z. Pan, M. Alam (Eds.), The Semantic Web: ESWC 2018 Satellite Events, volum...
2018 doi
-
[12]
Ilkou, H
E. Ilkou, H. Abu-Rasheed, M. Tavakoli, S. Hakimov, G. Kismihók, S. Auer, W. Nejdl, EduCOR: An Educational and Career-Oriented Recommendation Ontology, in: A. Hotho, E. Blomqvist, S. Dietze, A. Fokoue, Y. Ding, P. Barnaghi, A. Haller, M. Dragoni, H. Alani (Eds.), The Semantic W...
2021 doi
- [13]
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.