REVIEW 3 major objections 35 references
From "Strings" to "Things" for Personal Knowledge Graphs: Evaluating LLM Triple Extraction for Recommendation Systems
T0 review · 3 major / 0 minor · reviewed 2026-07-12 · grok-4.5
Pith's one-line read Lightweight open-weight LLMs can turn recommendation chats into RDF preference triples that still drive useful recommendations.
desk verdict Solid, reproducible workshop paper: first systematic open-weight LLM extraction of RDF preference triples from ReDial with controlled FedTREK-LM transfer and a useful inverse-scaling observation; soft spots are missing baselines and single-downstream-system scope, not load-bearing errors. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
A prompt-based PKG construction pipeline that converts multi-turn dialogues into (User, relation, Item) triples under a fixed five-relation preference ontology, then links movie mentions to Wikidata IRIs before the graph is consumed by a federated recommendation model.
What would settle it
Re-run the same extraction and recommendation pipeline on a conversational dataset whose preference labels were collected independently of the dialogue text; if extraction F1 collapses or downstream recommendation utility no longer tracks extraction quality, the claim that dialogue alone yields faithful PKGs fails.
Extended reading notes
Core claim
Certain lightweight open-weight models extract usable RDF preference triples from conversational recommendation dialogues, and the Personal Knowledge Graphs they produce retain proportionally high downstream recommendation performance relative to their raw extraction F1; in particular, a small, balanced extractor can outperform a larger high-precision one when the recommender itself is small.
Load-bearing premise
The binary liked/seen/suggested labels already present in the dialogue dataset are treated as complete, recoverable ground-truth triples that an LLM can extract from the raw conversation text alone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a reproducible pipeline that uses lightweight open-weight LLMs (Qwen3 and Gemma-3 families, multiple sizes, 0–10-shot prompting) to extract RDF-compliant (User, relation, Item) preference triples from multi-turn recommendation dialogues, grounds movie entities via Wikidata IRIs, and assembles them into Personal Knowledge Graphs under a small domain ontology (pkg:likes, pkg:dislikes, pkg:hasSeen, pkg:hasNotSeen, pkg:wasSuggested). Extraction fidelity is measured by precision/recall/F1 against ReDial metadata-derived ground-truth triples (Table 2, Figures 3–4). Downstream utility is assessed by training the FedTREK-LM recommender on the resulting augmented datasets and comparing recommendation F1 against models trained on the original ReDial annotations (Section 4.3, Section 5.1, Figure 5). The central empirical claim is that certain models (especially Gemma-3-12B for extraction F1 and Gemma-3-1B for balanced downstream utility) produce usable PKGs whose recommendation performance is proportionally high relative to extraction quality, supporting privacy-preserving, on-device PKG construction.
Significance. If the results hold, the work supplies a concrete, open, and reproducible bridge from conversational ‘strings’ to interoperable RDF ‘things’ for personal knowledge graphs, a missing piece for decentralized personalization. Strengths that should be credited include the public code and prompt templates, the explicit RDF/Wikidata grounding and ontology (Figure 2, Algorithm 1), the full relation-wise extraction tables, and the dual evaluation of extraction fidelity plus a real downstream recommender. The observation that a 1B extractor can yield higher downstream F1 for a small recommender than larger high-precision extractors is practically relevant for on-device settings. The contribution is therefore of clear interest to the Semantic Web / conversational recommendation community, provided the transfer claims are shown to be robust beyond the single evaluation framework used.
major comments (3)
- Section 4.3 and Section 5.1 / Figure 5: The load-bearing claim that extracted PKGs yield ‘proportionally high’ downstream performance (and the inverse-scaling result that Gemma-3-1B extractions outperform larger extractors for a small recommender) rests exclusively on FedTREK-LM trained only with Qwen3-family recommenders. No other recommender architecture, no ablation of the recommender’s sensitivity to triple density/balance, and no non-LLM extraction baseline are reported. Without at least one additional downstream system or a controlled density ablation, it remains unclear whether the proportional-utility and inverse-scaling conclusions are properties of the extracted PKGs or artifacts of FedTREK-LM’s particular dependence on ‘liked’ vs. ‘suggested’ coverage.
- Table 2 and Section 5: Extraction and downstream F1 differences are reported as point estimates with no error bars, confidence intervals, or statistical significance tests across dialogues or random seeds. Given that several key comparisons (e.g., 3-shot vs. 5-shot precision trade-off, Gemma-1B vs. Gemma-12B downstream F1) are numerically close and drive the narrative, the absence of uncertainty quantification weakens the strength of the relative-performance claims.
- Section 1.2, Table 1 and Section 4.1: The evaluation treats ReDial metadata features (liked/seen/suggested) as complete, recoverable ground-truth triples that can be obtained solely from dialogue text after normalization and Wikidata linking. If the annotations encode annotator inference or context not present in the surface conversation, both the absolute extraction F1 numbers and the claimed transfer to recommendation utility are mis-calibrated. A short qualitative error analysis or an estimate of annotation–dialogue mismatch would make the ground-truth assumption falsifiable.
Circularity Check
Minor self-citation of authors' own FedTREK-LM as the sole downstream recommender; extraction F1 and labels remain independent of any fit or self-definition.
-
self citation load bearing
[§4.3 Downstream Task Evaluation / §5.1]
"Specifically, we evaluate the effectiveness of the downstream task on the FedTREK-LM recommendation system [33, 34]. This recommendation system uses a local LLM in a federated system to generate recommendations using PKGs for context, so it fits nicely with our local, LLM-based, triple-extraction pipeline. ... we construct the FedTREK-LM dataset with our augmented dataset as the base, but then we still use the original ground-truth test dataset for evaluation."
The claim that extracted PKGs yield 'proportionally high' recommendation utility (and the inverse-scaling observation that Gemma-1B extractions beat larger models) is demonstrated exclusively inside the authors' own prior FedTREK-LM system. While extraction labels and the test set remain external, the transfer-utility conclusion has no independent recommender baseline and therefore rests on a self-cited evaluator.
full rationale
The paper is an empirical evaluation of LLM triple extraction against the external ReDial human annotations (Table 1, §4.1). Precision/recall/F1 are computed by exact match of (subject, predicate, object) after Wikidata linking; nothing is fitted and then re-predicted. The downstream utility experiments replace only the training triples while always evaluating on the original ground-truth test set (§4.3, §5.1). The sole self-citation is the use of the authors' prior FedTREK-LM framework as the recommendation engine that consumes the extracted PKGs. That citation supplies an evaluation tool rather than a uniqueness theorem, ansatz, or definitional identity that forces the reported numbers. No equation, prompt, or metric reduces by construction to its own inputs. The inverse-scaling and 'proportionally high' claims are therefore framework-specific observations, not circular derivations. Score 2 reflects the single non-load-bearing self-citation; the central extraction results stand independently.
Assumptions & free parameters
free parameters (3)
- few-shot count (0/1/3/5/10) =
5-shot selected as primary for downstream
- Wikidata fuzzy-match confidence threshold
- model-size sampling
assumptions (3)
- domain assumption ReDial metadata features (liked/seen/suggested) map one-to-one onto the five pkg: relations and constitute complete ground truth recoverable from dialogue text alone
- domain assumption RDF triples grounded in Wikidata IRIs are a suitable intermediate representation for both evaluation and downstream recommendation
- domain assumption FedTREK-LM F1 on the original test set is a valid utility measure of extracted PKG quality
invented entities (1)
-
User Movie Preference Ontology (pkg:likes, pkg:dislikes, pkg:hasSeen, pkg:hasNotSeen, pkg:wasSuggested)
Cite this review
Pith. "Pith review of From "Strings" to "Things" for Personal Knowledge Graphs: Evaluating LLM Triple Extraction for Recommendation Systems." pith.science (2026). https://pith.science/paper/7TVHSCNW
@misc{pith2026260700003,
author = {Pith},
title = {Pith review of: From "Strings" to "Things" for Personal Knowledge Graphs: Evaluating LLM Triple Extraction for Recommendation Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/7TVHSCNW}},
note = {Machine review of arXiv:2607.00003}
}
read the original abstract
Personal Knowledge Graphs (PKGs) offer a privacy-preserving framework for modeling user preferences, yet constructing them from unstructured, decentralized conversational data remains a challenge. This paper bridges the gap between conversational "strings" and semantic "things" by presenting a reproducible pipeline for extracting structured user-preference triples using lightweight Large Language Models. We evaluate Qwen- and Gemma-based models on their ability to extract RDF-compliant triples linked to Wikidata identifiers from conversational data for PKG construction. Our evaluation assesses both the semantic extraction fidelity and the utility of the resulting graphs in a downstream recommendation task. We found that certain models performed well and had proportionally high downstream performance relative to their triple extraction performance.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Balog, T
K. Balog, T. Kenter, Personal knowledge graphs: A research agenda, in: Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, 2019, pp. 217–220
2019
-
[2]
M. G. Skjæveland, K. Balog, N. Bernard, W. Łajewska, T. Linjordet, An ecosystem for personal knowledge graphs: A survey and research roadmap, AI Open 5 (2024) 55–69
2024
-
[3]
S. S. Shirai, O. Seneviratne, D. L. McGuinness, A survey on personal health knowledge graphs, in: Workshop on The Personal Health Knowledge Graph, Knowledge Graph Conference, 2020, 2020
2020
-
[4]
X. Wang, X. He, Y. Cao, M. Liu, T.-S. Chua, KGAT: Knowledge graph attention network for recommendation, in: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, 2019, pp. 950–958
2019
-
[5]
Christakopoulou, F
K. Christakopoulou, F. Radlinski, K. Hofmann, Towards conversational recommender systems, in: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016, pp. 815–824
2016
-
[6]
X. Wang, X. He, Y. Cao, M. Liu, T.-S. Chua, Kgat: Knowledge graph attention network for recom- mendation, in: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Dis- covery & Data Mining, KDD ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 950–958. URL: https://doi.org/10.1145/3292500.3330989. doi:10.1145/329250...
-
[7]
P. Wang, X. Li, F. Du, H. Liu, S. Zhi, A personalized recommendation system based on knowledge graph embedding and neural network, in: 2019 3rd International Conference on Data Science and Business Analytics (ICDSBA), 2019, pp. 161–165. doi:10.1109/ICDSBA48748.2019.00042
-
[8]
R. Li, S. Ebrahimi Kahou, H. Schulz, V. Michalski, L. Charlin, C. Pal, Towards deep conversational recommendations, Advances in neural information processing systems 31 (2018)
2018
Show all 35 references
-
[9]
Hugging Face, ReDial dataset, https://huggingface.co/datasets/community-datasets/re_dial, 2024
2024
-
[10]
Spadea, O
F. Spadea, O. Seneviratne, Bursting the filter bubble with knowledge graph inversion, in: Compan- ion Publication of the 17th ACM Web Science Conference 2025, 2025, pp. 39–43
2025
-
[11]
K. Meng, D. Bau, A. Andonian, Y. Belinkov, Locating and editing factual associations in gpt, Advances in neural information processing systems 35 (2022) 17359–17372
2022
-
[12]
K. Meng, A. S. Sharma, A. J. Andonian, Y. Belinkov, D. Bau, Mass-editing memory in a transformer, in: The Eleventh International Conference on Learning Representations, 2022
2022
-
[13]
De Cao, W
N. De Cao, W. Aziz, I. Titov, Editing factual knowledge in language models, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 6491–6506
2021
-
[14]
Petroni, T
F. Petroni, T. Rocktäschel, S. Riedel, P. Lewis, A. Bakhtin, Y. Wu, A. Miller, Language models as knowledge bases?, in: K. Inui, J. Jiang, V. Ng, X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joi...
2019
-
[15]
Roberts, C
A. Roberts, C. Raffel, N. Shazeer, How much knowledge can you pack into the parameters of a language model?, in: B. Webber, T. Cohn, Y. He, Y. Liu (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computatio...
2020 doi
-
[16]
Frey, L.-P
J. Frey, L.-P. Meyer, N. Arndt, F. Brei, K. Bulert, Benchmarking the abilities of large language models for rdf knowledge graph creation and comprehension: how well do llms speak turtle?, arXiv preprint arXiv:2309.17122 (2023)
2023 arXiv
-
[17]
Z. Wang, Z. Zhang, Y. Feng, Y. Zhao, Y. Feng, X. Sun, Extract, define, canonicalize: An llm-based framework for knowledge graph construction, arXiv preprint arXiv:2404.03868 (2024)
2024 arXiv
-
[18]
Zhang, C
T. Zhang, C. Guo, Z. Chen, Y. Fan, Beyond isolation: Multi-agent synergy for improving knowledge graph construction, arXiv preprint arXiv:2312.03022 (2023)
2023 arXiv
-
[19]
G. Li, Z. Xu, Z. Shang, J. Liu, K. Ji, Y. Guo, Empirical analysis of dialogue relation extraction with large language models, arXiv preprint arXiv:2404.17802 (2024)
2024 arXiv
-
[20]
Liang, C
T. Liang, C. Jin, L. Wang, W. Fan, C. Xia, K. Chen, Y. Yin, Llm-redial: a large-scale dataset for conversational recommender systems created from user behaviors with llms, in: Findings of the Association for Computational Linguistics ACL 2024, 2024, pp. 8926–8939
2024
-
[21]
W.-S. Kim, S. Lim, G.-W. Kim, S.-M. Choi, Extracting implicit user preferences in conversational recommender systems using large language models, Mathematics 13 (2025) 221
2025
-
[22]
H. Kook, J. Kim, S. Park, J. Lee, Empowering retrieval-based conversational recommendation with contrasting user preferences, in: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technolo...
2025
-
[23]
X. Chen, S. Song, Z. Zhang, S. Liu, C. Shi, Lkpnr: Llm and kg for personalized news recommendation framework, arXiv preprint arXiv:2308.12028 (2023)
2023 arXiv
-
[24]
Z. Zhao, S. Wang, Y. Li, M. Wu, H. Lu, B. Zhang, H. Sun, J. Zhang, Breaking the barrier: Utilizing large language models for industrial recommendation systems through an inferential knowledge graph, arXiv preprint arXiv:2402.13750 (2024)
2024 arXiv
-
[25]
C.-H. Chen, D. Gruen, J. Harris, J. Hendler, D. L. McGuinness, M. Monti, N. Rastogi, O. Seneviratne, M. J. Zaki, Semantic Technologies for Clinically Relevant Personal Health Applications, Springer In- ternational Publishing, Cham, 2022, pp. 199–220. URL: https://doi.org/10.10...
2022 doi
-
[26]
doi:10.1007/978-3-031-07696-1_10
-
[27]
Seneviratne, M
O. Seneviratne, M. Shukla, Personal health knowledge graph construction using internet of medical things, in: Personal Knowledge Graphs (PKGs): Methodology, tools and applications, Computing, Institution of Engineering and Technology, 2023, pp. 295–305. URL: https://digital-li...
2023 doi
-
[28]
Seneviratne, J
O. Seneviratne, J. Harris, C.-H. Chen, D. L. McGuinness, Personal health knowledge graph for clinically relevant diet recommendations, Workshop on Personal Knowledge Graphs Co- located with the 3rd Automatic Knowledge Base Construction Conference (AKBC’21) (2021). URL: https:/...
2021 arXiv
-
[29]
Z. Yang, I. Azimi, M. J. Zaki, M. Gaur, O. Seneviratne, D. L. McGuinness, S. M. Rashid, A. M. Rahmani, Transforming Personal Health AI: Integrating Knowledge and Causal Graphs with Large Language Models, in: Proceedings of the ISCAP Conference, 2024
2024
-
[30]
Zhang, M
P. Zhang, M. Fnu, Y. Song, O. Seneviratne, Z. Yang, I. Azimi, A. M. Rahmani, Adaptive Constraint Relaxation in Personalized Nutrition Recommendations: An LLM-Driven Knowledge Graph Retrieval Approach, in: AMIA Annual Symposium Proceedings, American Medical Informatics Associat...
2025
-
[31]
Shirai, O
S. Shirai, O. Seneviratne, D. L. McGuinness, Applying personal knowledge graphs to health, in: Personal Health Knowledge Graph Workshop at Knowledge Graph Conference, 2021. URL: https://arxiv.org/abs/2104.07587
2021 arXiv
-
[32]
Experiments used Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, and Qwen3-8B
Hugging Face, Qwen3-0.6b, https://huggingface.co/Qwen/Qwen3-0.6B, 2024. Experiments used Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, and Qwen3-8B
2024
-
[33]
Includes Gemma-3-1B-IT, Gemma-3-4B-IT, and Gemma-3-12B-IT
Google, Gemma 3: Open instruction-tuned language models, https://huggingface.co/google/ gemma-3-12b-it, 2025. Includes Gemma-3-1B-IT, Gemma-3-4B-IT, and Gemma-3-12B-IT
2025
-
[34]
Spadea, O
F. Spadea, O. Seneviratne, Federated personal knowledge graph completion with lightweight large language models for personalized recommendations, in: Proceedings of the European Semantic Web Conference, 2026
2026
-
[35]
Spadea, O
F. Spadea, O. Seneviratne, TREK-LM, https://github.com/brains-group/TREK-LM, 2025
2025
Reviewed July 12, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.