Pith. sign in

REVIEW 3 major objections 5 minor 13 references

LLM-Assisted Knowledge Graph Completion for Curriculum and Domain Modelling in Personalized Higher Education Recommendations

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read This paper claims that an LLM-assisted, teacher-validated pipeline can extract fine-grained topics from lecture materials and link related courses across disciplines in a knowledge graph, supporting personalized higher-education learning…

desk verdict A solid, clearly described human-in-the-loop pipeline for LLM-based curriculum KG building, but the headline claim about cross-course linking rests on metrics that don't directly test whether the semantic edges are correct. read the letter →

arxiv 2501.12300 v1 pith:EETUHOZL submitted 2025-01-21 cs.HC cs.AI

classification cs.HCcs.AI
keywords HighereducationKnowledgegraphLargelanguagemodelsCurriculummodelDomainUserLearningpersonalizationHuman-AIcollaboration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that large language models and human teachers can collaborate to build a knowledge graph of university course content that links topics across modules and disciplines. The authors propose an ontology with curriculum, domain, and user models, use an LLM to extract fine-grained topics and sub-topics from lecture slides, manuscripts, and video transcripts, and have teachers validate the results. Applied to two embedded-systems courses, the pipeline achieved high precision, recall, and F1 scores on topic and sub-topic extraction and added semantic cross-course links to the graph. The authors argue this supports personalized learning-path recommendations by connecting related content across courses.

What carries the argument

The carrying mechanism is the three-model ontology—curriculum model, domain model, user model—with Topic and Sub-Topic as the core taxonomic classes. The pipeline feeds lecture slides, manuscripts, and video transcripts (transcribed by a speech recognizer) to an LLM prompted with ontology definitions so it can extract and classify concepts; teachers then validate or correct the extraction, and the validated titles and descriptions feed a semantic-similarity step that creates cross-course relations. The ontology keeps the extracted content interoperable across faculties and institutions, and the KG in turn becomes context for later LLM extraction.

What would settle it

Feed the pipeline a course whose slides convey core concepts mainly through diagrams and equations with little textual annotation, then have the same teachers judge the extracted topics; if central figure-only concepts are missing from the validated graph, the assumption fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that a semi-automated, teacher-validated pipeline can produce a curriculum knowledge graph that domain experts accept. On two embedded-systems modules the LLM extraction achieved topic precision of 0.99 and 0.97, sub-topic precision of 1.00 and 0.89, and recall and F1 values in the same high range across classes; the automatically generated sub-topic descriptions also scored highly on precision. Graph-level structural measures moved in the intended direction after semantic links joined the two modules: average degree centrality rose from 0.9 to 1.03 and modularity fell slightly from 0.769 to 0.767. The lecturers reported that the extracted topics and cross-course similarities helped them restructure content and would support personalized learning paths for students with different backgrounds.

Load-bearing premise

The pipeline assumes that figures, images, and equations in lecture slides are annotated or described in text, so the transcription and LLM extraction can capture the real teaching content.

Editorial extensions

If this is right

  • Teachers can use the cross-course links to spot overlapping content and restructure their modules, reducing duplication for students who take both.
  • Students with prior knowledge in one domain can receive learning paths that skip already-covered topics and connect to related modules from other faculties.
  • The semi-supervised validation scheme, where teachers check a random sample of predicted relations, lets the approach scale as the graph grows.
  • The KG doubles as context for later LLM extraction, which should improve accuracy when new lectures resemble content already in the graph.
  • Prompt engineering rules that distinguish topics from sub-topics and ignore side discussions are necessary for clean extraction from live recordings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same pipeline could be pointed at assessment data, using the user-model parameters to recommend topic-level remediation rather than whole courses.
  • The reported graph-metric change is small; a stronger test of the linking idea would be a larger multi-course graph where semantic relations connect more than two modules, or a lower similarity threshold.
  • The explicit assumption that figures and equations are textually described suggests a natural extension to multimodal LLMs that caption diagrams directly; that extension is testable on the same two modules.
  • Whether the approach saves teacher time is the practical crux; a before-and-after measurement of validation hours would quantify the collaboration benefit that the qualitative feedback supports.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents an LLM-assisted knowledge graph completion pipeline for higher education, built on an ontology with curriculum, domain, and user models. The pipeline transcribes lecture videos with Whisper, extracts topics and sub-topics with GPT-4o, and has teachers validate the extractions. The implementation covers two embedded-systems modules, and the evaluation combines expert precision/recall/F1 on 1197 extraction samples, graph-structural metrics (average degree centrality and modularity), and qualitative lecturer feedback. The central claim is that the approach enhances the ability to connect related courses across disciplines for personalized learning, but the quantitative support for that specific claim is limited to structural changes in the constructed graph.

Significance. If validated, the contribution is relevant and useful: it addresses teacher workload in curriculum modelling, integrates domain and user models, and provides an ontology-based, human-in-the-loop approach for KG construction. The manuscript's strengths include a concrete pipeline description, a clear statement of the visual-content assumption, a non-trivial expert-labeled sample for extraction quality, and a reproducible-looking experimental design. However, the paper's headline result about cross-module semantic linking is not yet supported because the semantic relations themselves are not evaluated.

major comments (3)
  1. [Section 4.2] The quantitative evidence for the central claim is the ADC increase from 0.90 to 1.03 and the modularity decrease from 0.769 to 0.767. These measures are expected consequences of adding a small number of edges: degree centrality rises mechanically, and modularity falls when cross-community edges are inserted. They do not show that the added edges are semantically correct or useful. The manuscript mentions semi-supervised validation of semantic similarities in Section 3.4 but reports no results for that validation, no precision/recall for the semantic relations, no baseline comparison, and no random-edge control. Without such evidence, the abstract's claim that the approach enhances the ability to connect related courses across disciplines is not demonstrated.
  2. [Section 4.1] The evaluators are the module lecturers from the same institute as several co-authors. Because they are also the content providers and participants in the collaborative design, their qualitative acceptance does not independently validate the ontology, the pipeline, or the cross-module connections. The quantitative extraction results also rely on their judgments, so the evaluation mixes self-assessment with independent validation. The paper should include at least some external evaluators who are not involved in developing the modules, or otherwise should clearly separate self-assessment from external validation.
  3. [Section 3.2] The assumption that figures, images, and equations in lecture slides are annotated or described textually is central to the transcription and extraction pipeline; if engineering content is largely visual, the extracted KG will be incomplete despite high text-based scores. The paper does not test this assumption or report the proportion of visual content in the evaluated modules. Moreover, the two evaluation modules are both in embedded systems, so the claim of across-discipline connection is not covered by the empirical scope. The authors should either report visual-content coverage and broaden the evaluation, or soften the claim accordingly.
minor comments (5)
  1. [Section 3.4] The high similarity thresholds mentioned in Section 4.2 are not quantified; reporting the exact thresholds and the prompt templates used for semantic relation extraction would improve reproducibility.
  2. [Table 1] The decimal separator is inconsistent (0,99 vs. 0.97) and the sample sizes per class are not shown in the table; the text gives totals but not class-wise counts.
  3. [Footnote] The corresponding-author email in the footnote appears corrupted ('/envel⌢pe-⌢penhasan...') and should be corrected.
  4. [Abstract and Section 3.4] The terms knowledge graph completion and knowledge graph construction are used interchangeably; the relationship between the baseline graph and the completed graph should be made explicit.
  5. [Figure 3] Figure 3 would be clearer with an explicit statement of node and edge counts before and after completion, beyond the ADC and modularity values.

Circularity Check

1 steps flagged · score 4.0 of 10

Headline graph metric is a by-product of edge addition; extraction evaluation is non-circular, but semantic-edge quality is not independently measured.

  1. self definitional [Section 4.2, Figure 3]
    "In the resulting KG, see Figure 3, we calculate an increase of the ADC value from 0.9 to 1.03, as well as a slight modularity decrease from 0.769 to 0.767. Figure 3: The structure of the KG before (left) and after (right) connecting both evaluation modules through semantic Topic and Sub-Topic relations."

    ADC is average degree centrality, i.e., the mean number of relations per node. The 'after' graph is constructed by inserting semantic Topic/Sub-Topic relations between the two modules (Figure 3 caption); adding edges increases average degree by definition. The reported ADC rise from 0.90 to 1.03 therefore measures the edge-addition operation itself, not whether the inserted edges represent genuine relatedness or improve learning-path recommendations. The modularity drop of 0.002 is likewise a direct, small consequence of adding a few cross-module edges. The qualitative lecturer feedback provides some independent evidence, but the quantitative 'graph quality' validation does not independently confirm the cross-disciplinary connection claim.

full rationale

The paper's topic/sub-topic extraction results (Table 1) are compared against expert labels and are not circular. The central 'graph completion enhances connection' claim, however, is partly supported by an ADC increase that is definitionally produced by the edge-addition step of graph construction; this is a self-fulfilling metric. The modularity decrease is also a mechanical consequence of adding edges. Qualitative expert feedback (Section 4.1) offers non-circular, though small, corroboration; the semi-supervised validation of semantic relations mentioned in Section 3.4 is not reported, so the semantic quality of the added edges is not independently established. Self-citations [5] and [12] are used as method/background references rather than as authority for the central result, so they are not load-bearing circularity. Overall, the majority of the pipeline is self-contained, but the headline quantitative evidence for the cross-disciplinary-connection claim is partly definitional; hence a score of 4.

Assumptions & free parameters 2 free parameters · 4 assumptions · 2 invented entities

The central claim rests on the textual-content assumption for lecture materials, the sufficiency of the Topic/Sub-Topic hierarchy, external LLM behavior, and the validity of the in-house expert evaluation. The only hand-set numerical parameter disclosed is the semantic similarity threshold, though its value is not reported.

free parameters (2)
  • Semantic similarity threshold(s) for Topic and Sub-Topic relations = Not reported in the paper.
    The paper states that high thresholds were set to ensure relation quality, and these thresholds directly control how many edges are added and therefore the measured ADC and modularity changes.
  • LLM prompt rules and ontology definitions for GPT-4o extraction = Not reported in the paper.
    The authors adjusted the prompt after observing extraction failures caused by side discussions, so the final prompts are hand-tuned artifacts that influence the reported precision and recall.
assumptions (4)
  • domain assumption Figures, images, and equations in lecture slides are annotated or described textually.
    Explicitly stated in Section 3.2. If visual content is not represented in text, transcription and LLM extraction will miss important teaching content.
  • ad hoc to paper The Topic/Sub-Topic hierarchy is a sufficient and unambiguous representation of curriculum content across diverse courses and institutions.
    Defined in Section 3.1 and used in prompts, but tested only on two embedded-systems modules at one university.
  • domain assumption Whisper and GPT-4o perform extraction and classification with the accuracy and stability implied by the evaluation.
    The pipeline depends on external model behavior without version pinning or reproducible configuration, as described in Section 3.2.
  • domain assumption The three expert evaluators provide accurate, unbiased judgments of extraction quality.
    Evaluators are drawn from the same Embedded Systems Institute as several authors, and no independence or blinding is described in Section 4.1.
invented entities (2)
  • Higher-education ontology classes (Topic, Sub-Topic, Session, Lecture, Domain, Sub-Domain, User model)
    purpose: Provide a shared curriculum representation so that content from different courses and faculties can be linked for personalization.
    The ontology is assembled largely from the authors' prior EduCOR work and is evaluated only through feedback on two modules, so its generality beyond this setting is not established.
  • Automatically generated sub-topic descriptions
    purpose: Provide text features for computing semantic similarity between lectures from different modules.
    The descriptions are generated by GPT-4o and edited by teachers; the paper reports only a precision value on this class and provides no external benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-Assisted Knowledge Graph Completion for Curriculum and Domain Modelling in Personalized Higher Education Recommendations." pith.science (2026). https://pith.science/paper/EETUHOZL

@misc{pith2026250112300,
  author       = {Pith},
  title        = {Pith review of: LLM-Assisted Knowledge Graph Completion for Curriculum and Domain Modelling in Personalized Higher Education Recommendations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EETUHOZL}},
  note         = {Machine review of arXiv:2501.12300}
}
read the original abstract

While learning personalization offers great potential for learners, modern practices in higher education require a deeper consideration of domain models and learning contexts, to develop effective personalization algorithms. This paper introduces an innovative approach to higher education curriculum modelling that utilizes large language models (LLMs) for knowledge graph (KG) completion, with the goal of creating personalized learning-path recommendations. Our research focuses on modelling university subjects and linking their topics to corresponding domain models, enabling the integration of learning modules from different faculties and institutions in the student's learning path. Central to our approach is a collaborative process, where LLMs assist human experts in extracting high-quality, fine-grained topics from lecture materials. We develop a domain, curriculum, and user models for university modules and stakeholders. We implement this model to create the KG from two study modules: Embedded Systems and Development of Embedded Systems Using FPGA. The resulting KG structures the curriculum and links it to the domain models. We evaluate our approach through qualitative expert feedback and quantitative graph quality metrics. Domain experts validated the relevance and accuracy of the model, while the graph quality metrics measured the structural properties of our KG. Our results show that the LLM-assisted graph completion approach enhances the ability to connect related courses across disciplines to personalize the learning experience. Expert feedback also showed high acceptance of the proposed collaborative approach for concept extraction and classification.

Figures

Figures reproduced from arXiv: 2501.12300 by the authors.

Figure 1
Figure 1. Proposed ontological structure of the knowledge graph, with domain, curriculum, and user models. 3.2. Automatic Content Extraction and Classification In order for teachers to classify the content of their lectures into topics and sub-topics, a great amount of time and effort needs to be invested, which is rarely available for teachers due to their busy schedules. Therefore, we developed an automated extraction and c… view at source ↗
Figure 2
Figure 2. Pipeline components for the transcription, extraction, classification, and KG construction, based on a human-AI collaborative approach. here that this ontology-based contextualization of the LLM prompt is imperative to extract an accurate representation of the learning content, ensure a sufficient level of interoperability, and enable finding meaningful similarities between the topics and sub-topics among lectures f… view at source ↗
Figure 3
Figure 3. The structure of the KG before (left) and after (right) connecting both evaluation modules through semantic Topic and Sub-Topic relations. the semantic relation extraction algorithm, to ensure high-quality relations between the topics and sub-topics. Despite this relatively small improvement on the KG structurally, qualitative expert feedback pointed out that the automatic topic extraction and the possibility to sem… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 6 canonical work pages

  1. [1]

    Meissner, L

    R. Meissner, L. Köbis, Annotated Knowledge Graphs for Teaching in Higher Education: Supporting Mentors and Mentees by Digital Systems, in: M. Bielikova, T. Mikkonen, C. Pautasso (Eds.), Web Engineering, volume 12128, Springer International Publishing, Cham, 2020, pp. 551–555. URL: http: //link.springer.com/10.1007/978-3-030-50578-3_43. doi: 10.1007/978-3-...

  2. [2]

    Ibrahim, S

    N. Ibrahim, S. Aboulela, A. Ibrahim, R. Kashef, A survey on augmenting knowledge graphs (KGs) with large language models (LLMs): models, evaluation metrics, benchmarks, and challenges, Dis- cover Artificial Intelligence 4 (2024) 76. URL: https://link.springer.com/10.1007/s44163-024-00175-8. doi:10.1007/s44163-024-00175-8

  3. [3]

    Naveed, A

    H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, A. Mian, A Comprehensive Overview of Large Language Models, 2023. URL: https://arxiv.org/abs/2307.06435. doi:10.48550/ARXIV.2307.06435, version Number: 10

  4. [4]

    S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, X. Wu, Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023). URL: https://arxiv.org/abs/2306.08302. doi:10.48550/ARXIV.2306. 08302, publisher: arXiv Version Number: 3

  5. [5]

    Abu-Rasheed, M

    H. Abu-Rasheed, M. Dornhöfer, C. Weber, G. Kismihók, U. Buchmann, M. Fathi, Building Con- textual Knowledge Graphs for Personalized Learning Recommendations Using Text Mining and Semantic Graph Completion, in: 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), IEEE, Orem, UT, USA, 2023, pp. 36–40. URL: https://ieeexplore.ieee.or...

  6. [6]

    Kejriwal, Knowledge Graphs: A Practical Review of the Research Landscape, Information 13 (2022) 161

    M. Kejriwal, Knowledge Graphs: A Practical Review of the Research Landscape, Information 13 (2022) 161. URL: https://www.mdpi.com/2078-2489/13/4/161. doi:10.3390/info13040161

  7. [7]

    V. K. Kommineni, B. König-Ries, S. Samuel, From human experts to machines: An LLM supported approach to ontology and knowledge graph construction, 2024. URL: https://arxiv.org/abs/2403. 08345. doi:10.48550/ARXIV.2403.08345, version Number: 1

  8. [8]

    Jhajj, X

    G. Jhajj, X. Zhang, J. R. Gustafson, F. Lin, M. P.-C. Lin, Educational Knowledge Graph Creation and Augmentation via LLMs, in: A. Sifaleras, F. Lin (Eds.), Generative Intelligence and Intelligent Tutoring Systems, volume 14799, Springer Nature Switzerland, Cham, 2024, pp. 292–304. URL: https://link.springer.com/10.1007/978-3-031-63031-6_25. doi: 10.1007/9...

Show all 13 references
  1. [9]

    Abu-Rasheed, C

    H. Abu-Rasheed, C. Weber, M. Fathi, Knowledge Graphs as Context Sources for LLM-Based Expla- nations of Learning Recommendations, in: 2024 IEEE Global Engineering Education Conference (EDUCON), IEEE, Kos Island, Greece, 2024, pp. 1–5. URL: https://ieeexplore.ieee.org/document/...

  2. [10]

    Stancin, P

    K. Stancin, P. Poscic, D. Jaksic, Ontologies in education – state of the art, Education and Information Technologies 25 (2020) 5301–5320. URL: https://link.springer.com/10.1007/s10639-020-10226-z. doi:10.1007/s10639-020-10226-z

  3. [11]

    Katis, H

    E. Katis, H. Kondylakis, G. Agathangelos, K. Vassilakis, Developing an Ontology for Curriculum and Syllabus, in: A. Gangemi, A. L. Gentile, A. G. Nuzzolese, S. Rudolph, M. Maleshkova, H. Paul- heim, J. Z. Pan, M. Alam (Eds.), The Semantic Web: ESWC 2018 Satellite Events, volum...

  4. [12]

    Ilkou, H

    E. Ilkou, H. Abu-Rasheed, M. Tavakoli, S. Hakimov, G. Kismihók, S. Auer, W. Nejdl, EduCOR: An Educational and Career-Oriented Recommendation Ontology, in: A. Hotho, E. Blomqvist, S. Dietze, A. Fokoue, Y. Ding, P. Barnaghi, A. Haller, M. Dragoni, H. Alani (Eds.), The Semantic W...

  5. [13]

    Radford, J

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, I. Sutskever, Robust Speech Recognition via Large-Scale Weak Supervision, 2022. URL: https://arxiv.org/abs/2212.04356. doi:10.48550/ ARXIV.2212.04356, version Number: 1

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.