RECORD
KG4RecEval: Does Knowledge Graph Really Matter for Recommender Systems?
Information Retrieval · Submitted 4 Apr 2024
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Our extensive experiments on four commonly used datasets and a number of state-of-the-art KG-based RSs reveal that: to remove, randomly distort or decrease knowledge does not necessarily decrease recommendation accuracy, even for cold-start users.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Recommender systems (RSs) are designed to provide personalized recommendations to users. Recently, knowledge graphs (KGs) have been widely introduced in RSs to improve recommendation accuracy. In this study, however, we demonstrate that RSs do not necessarily perform worse even if the KG is downgraded to the user-item interaction graph only (or removed). We propose an evaluation framework KG4RecEval to systematically evaluate how much a KG contributes to the recommendation accuracy of a KG-based RS, using our defined metric KGER (KG utilization efficiency in recommendation). We consider the scenarios where knowledge in a KG gets completely removed, randomly distorted and decreased, and also where recommendations are for cold-start users. Our extensive experiments on four commonly used datasets and a number of state-of-the-art KG-based RSs reveal that: to remove, randomly distort or decrease knowledge does not necessarily decrease recommendation accuracy, even for cold-start users. These findings inspire us to rethink how to better utilize knowledge from existing KGs, whereby we discuss and provide insights into what characteristics of datasets and KG-based RSs may help improve KG utilization efficiency. The code and supplementary material of this paper are available at: https://github.com/HotBento/KG4RecEval.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Recommender systems (RSs) are designed to provide personalized recommendations to users. | background 1.00 | 0.03 | 0.01 | asserted 1.00 |
| s1 | Recently, knowledge graphs (KGs) have been widely introduced in RSs to improve recommendation accuracy. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s2 | In this study, however, we demonstrate that RSs do not necessarily perform worse even if the KG is downgraded to the user-item interaction graph only (or removed). | result 0.95 | 0.91 | 0.80 | asserted 0.99 |
| s3 | We propose an evaluation framework KG4RecEval to systematically evaluate how much a KG contributes to the recommendation accuracy of a KG-based RS, using our defined metric KGER (KG utilization efficiency in recommendation). | contribution 0.75 | 0.95 | 0.06 | asserted 0.99 |
| s4 | We consider the scenarios where knowledge in a KG gets completely removed, randomly distorted and decreased, and also where recommendations are for cold-start users. | method 0.89 | 0.77 | 0.04 | asserted 0.58 |
| s5 | Our extensive experiments on four commonly used datasets and a number of state-of-the-art KG-based RSs reveal that: to remove, randomly distort or decrease knowledge does not necessarily decrease recommendation accuracy, even for cold-start users. | result 1.00 | 0.97 | 0.83 | asserted 0.94 |
| s6 | These findings inspire us to rethink how to better utilize knowledge from existing KGs, whereby we discuss and provide insights into what characteristics of datasets and KG-based RSs may help improve KG utilization efficiency. | interpretation 0.94 | 0.64 | 0.09 | hedged possibility 0.79 |
| s7 | The code and supplementary material of this paper are available at: https://github.com/HotBento/KG4RecEval. | contribution 0.74 | 0.32 | 0.02 | asserted 0.92 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Recommender systems (RSs) are designed to provide personalized recommendations to users. Recently, knowledge graphs (KGs) have been widely introduced in RSs to improve recommendation accuracy. In this study, however, we demonstrate that RSs do not necessarily perform worse even if the KG is downgraded to the user-item interaction graph only (or removed). We propose an evaluation framework KG4RecEval to systematically evaluate how much a KG contributes to the recommendation accuracy of a KG-based RS, using our defined metric KGER (KG utilization efficiency in recommendation). We consider the scenarios where knowledge in a KG gets completely removed, randomly distorted and decreased, and also where recommendations are for cold-start users. Our extensive experiments on four commonly used datasets and a number of state-of-the-art KG-based RSs reveal that: to remove, randomly distort or decrease knowledge does not necessarily decrease recommendation accuracy, even for cold-start users. These findings inspire us to rethink how to better utilize knowledge from existing KGs, whereby we discuss and provide insights into what characteristics of datasets and KG-based RSs may help improve KG utilization efficiency. The code and supplementary material of this paper are available at: https://github.com/HotBento/KG4RecEval.
Human review
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites
Uses the method or tool of 1 paper
-
To standardize the data format and evaluation process, we use RecBole https://recbole.io/https://recbole.io/~ [ref] , a Python-based RS library built on top of PyTorch.
Recent Advances in RecBole: Extensions with more Practical Considerations · Implementation Settings · 1.00
Cites as background 4 papers
-
Zhang et al. modify the relations in the graph based on a fact plausibility scoring function to influence downstream tasks, e.g., link prediction and recommendation [ref] .
Data Poisoning Attack against Knowledge Graph Embedding · Background · 0.88
-
For instance, Lv et al. [ref] and Faber et al. [ref] study the roles played by different elements of graphs and look for ways to make better use of the graph structure.
Should Graph Neural Networks Use Features, Edges, Or Both? · Background · 0.82
-
There are also works surveying the existing KG-based RSs, such as [ref] and [ref] , which systematically summarize the development, classification, and methods of KG-based RSs.
Deep Learning on Knowledge Graph for Recommender System: A Survey · Background · 1.00
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.