A Graph Perspective to Probe Structural Patterns of Knowledge in Large Language Models

Franck Dernoncourt; Mahantesh M Halappanavar; Nesreen K. Ahmed; Ryan A. Rossi; Utkarsh Sahu; Yao Ma; Yongjia Lei; Yu Wang; Zhisheng Qi

REVIEW 5 major objections 6 minor 40 references

A Graph Perspective to Probe Structural Patterns of Knowledge in Large Language Models

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM knowledge, mapped onto a knowledge graph, is structured enough that a graph model can predict which facts the model does not know, and fine-tuning on those predicted-unknown triplets outperforms random selection (69.04 vs 62.09 on…

desk verdict The graph-based triplet selection idea is worth a serious look and the average fine-tuning gain is plausible, but the paper overclaims consistency and leaves the main empirical result unverifiable without a clear train/eval holdout statement. read the letter →

arxiv 2505.19286 v2 pith:H5L3UPNY submitted 2025-05-25 cs.CL cs.LGcs.SI

Utkarsh Sahu , Zhisheng Qi , Yongjia Lei , Ryan A. Rossi , Franck Dernoncourt , Nesreen K. Ahmed , Mahantesh M Halappanavar , Yao Ma

show 1 more author

Yu Wang

This is my paper

classification cs.CLcs.LGcs.SI

keywords knowledgegraphslargelanguagemodelsknowledgeabilityhomophilygraphneuralnetworksfine-tuningdataselectionprobingnodedegree

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the facts a large language model knows are not scattered uniformly: when entities from a knowledge graph are scored by how often the model recognizes their triplets, connected entities look alike, and heavily connected entities tend to be better known. The authors turn this observed structure into a practical loop. They prompt an LLM on a small sample of triplets, aggregate the True/False answers into entity-level knowledgeability scores, train a graph neural network to predict scores for the remaining entities, and fine-tune the LLM on triplets involving entities predicted to be least known. Across five knowledge graphs and three 7-8B open models, this Graph-FT selection beats random triplet selection, with average evaluation accuracy 69.04 versus 62.09 for random selection and 49.64 for the base model. The payoff of the claimed discovery is that graph topology alone can locate the weakest parts of an LLM's knowledge without querying the model on every fact.

What carries the argument

The load-bearing objects are the entity knowledgeability score $K(v_i)$, defined as the average over incident triplets of the LLM's binary self-report, and the knowledge homophily measure $H_{v_i} = 1 - \frac{1}{|N(v_i)|} \sum_{v_j \in N(v_i)} |K(v_i) - K(v_j)|$. $K$ turns the binary probes into graph node labels, homophily justifies using a message-passing graph neural network (GraphSAGE with text or one-hot features) to regress $K$ on the graph, and the predicted $K$ drives the Graph-FT triplet selection strategy by choosing entities with the smallest predicted knowledgeability.

What would settle it

Take a held-out set of triplets with verified gold labels, run the same True/False probe on a model, and compare: if the probe's accuracy is not far above chance for facts the model demonstrably knows, then $K$ measures response bias rather than knowledge. Separately, recompute homophily on degree-preserving random edge permutations of the same graphs; if permuted graphs show equally high homophily, the pattern is generated by the score-averaging formula rather than by topology.

Watch

Extended reading notes

Core claim

We show that LLM knowledge on a knowledge graph is structured: entity-level knowledgeability scores follow a trimodal distribution, correlate positively with degree, and display homophily, meaning topologically close entities have similar knowledgeability. Because of this homophily, a GNN regressor trained on a subset of entities can estimate knowledgeability for all entities from local neighborhood context, avoiding exhaustive prompting. By ranking entities by predicted ignorance (one minus predicted knowledgeability) and fine-tuning on their triplets, we outperform random triplet selection, with Graph-FT averaging 69.04 versus Random-FT at 62.09 and the base model at 49.64.

Load-bearing premise

The load-bearing premise is that a model's binary True/False self-report is a valid measure of whether it knows a fact, and that the neighbor similarity in scores is genuine graph structure rather than an artifact of averaging each entity's score from the same incident triplets that link it to its neighbors.

Editorial extensions

If this is right

Fine-tuning on triplets selected by predicted ignorance improves accuracy over random selection, with the largest gains on domain-specific graphs such as MVPKG and PharmKG8K.
Only 20% of the fine-tuning budget needs to be probed by the LLM; the GNN extrapolates scores for the rest from graph structure, reducing the cost of knowledge checking.
Entity knowledgeability grows with degree and centrality, so well-connected entities are more likely to be known, quantifying the head-versus-tail imbalance in LLM factual knowledge.
Knowledge homophily appears consistently across five LLMs, so graph-based knowledgeability estimation transfers across models rather than being tied to one model.
Adding timestamps to triplets lowers knowledgeability and shifts homophily, indicating that time-sensitive facts form a distinct weak region that the same selection loop can target.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the self-assessed scores truly track knowledge, the same graph regression could prioritize entities for knowledge editing or retrieval-augmented generation, not just fine-tuning data selection.
The initial 20% probe set could be chosen actively rather than randomly, potentially reducing probing further while keeping the GNN's extrapolation accurate.
Because homophily persists across LLMs, knowledgeability scores from a large proprietary model could guide fine-tuning of smaller open models without re-probing every fact.
Temporal knowledge shows lower knowledgeability and weaker homophily, so time-sensitive subgraphs are a natural target for the same predicted-ignorance selection loop.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea is genuinely useful: use a GNN to estimate which entities an LLM knows least about, then fine-tune on triplets involving those entities. The average gain (Graph-FT 69.04 vs Random-FT 62.09 vs base 49.64) is large enough that this deserves to be checked properly. The entity-level homophily analysis is also new to me, and the multi-model, multi-dataset sweep is solid work. Credit where due: the GNN regression consistently beats MLPs, the code is public, and the authors cite the relevant prior work including Head-to-Tail.

But there are soft spots, and one is load-bearing. Appendix D.2 never states that the evaluation triplets are excluded from the fine-tuning set. If the same triplets appear in both, the headline result could just be memorization of answer labels. The authors need to state the holdout explicitly and report evaluation and budget sizes per dataset. This is fixable but until it is stated, the empirical claim is not verifiable from the text.

The other concerns are real but less severe. The knowledgeability measure is binary self-report with no validation against ground truth factuality; that's an interpretational limitation, not necessarily a fatal one. The homophily measure is partly mechanical because each edge's triplet score contributes to both endpoints' entity scores in Eq. (1), so a permutation null model is needed to show the effect exceeds chance. The paper's claim that Graph-FT "consistently outperforms" Random-FT is contradicted by its own Table 2 in three of fifteen cells (T-Rex Qwen, WD50 Mistral, CoDEx Llama). And there's a sampling confound: Graph-FT selects entities while Random-FT samples triplets, and the low-degree tie-break changes the degree distribution, so an entity-random control would isolate the knowledgeability benefit. No error bars or significance tests either, which is a minor complaint for a paper of this scope.

Who should read this: researchers working on LLM knowledge probing, efficient fine-tuning, or GNNs on knowledge graphs. It deserves a serious referee rather than a desk reject, but only after the authors clarify the holdout and add a null model. I'd send it to review with a request for major revision.

Referee Report

5 major / 6 minor

Summary. This paper proposes a graph-based framework for probing the structural patterns of knowledge encoded in large language models (LLMs). It defines triplet-level knowledgeability as a binary True/False self-report elicited from the LLM (Section 2.1), aggregates these scores to entity-level knowledgeability in Eq. (1), and reports three structural findings: trimodal knowledgeability distributions, a positive degree–knowledgeability correlation, and knowledge homophily where topologically close entities have similar scores (Eq. (2)). It then trains GNN regressors to estimate entity knowledgeability from a small labeled subset and uses the predicted 'ignorance' scores to select triplets for fine-tuning, reporting that this Graph-FT strategy outperforms Random-FT and the base model across five knowledge graphs and three open-weight 7-8B LLMs (Table 2, averages 69.04 vs 62.09 vs 49.64). The paper concludes that graph topology can be used to target the weakest parts of an LLM's knowledge.

Significance. If the central claims hold, the paper makes a useful contribution by connecting graph structure to LLM knowledge and by demonstrating a concrete application: selecting less-known triplets for fine-tuning. The evaluation spans five knowledge graphs and five LLMs, the code is claimed to be public, and the GNN-based regression pipeline is a sensible design. However, the quantitative findings currently rest on three unverified pillars: the self-report measure is not validated against ground-truth factuality, the homophily measure is partially mechanical given Eq. (1) and lacks a null-model baseline, and the headline fine-tuning result does not state that evaluation triplets are held out from the fine-tuning budget. These issues are load-bearing for the paper's main claims, so the significance is conditional on resolving them.

major comments (5)

[Appendix D.2, Table 2] The central fine-tuning result is unverifiable as reported because the manuscript never states that the evaluation triplets are excluded from the fine-tuning budget. The evaluation set is described only as 'randomly sampling a fixed number of triplets,' and the budget selection procedure (initial 20% query plus remaining 80%) does not mention any removal of evaluation triplets; if overlap exists, the reported Graph-FT average of 69.04 could reflect label memorization rather than the benefit of selecting less-known triplets. Please specify the holdout procedure, report per-dataset evaluation and budget sizes, and re-run the experiments if overlap occurred.
[Section 2.2, Eqs. (1)-(2), Finding 2] The homophily finding is partially forced by construction: K(vi) in Eq. (1) is the average of all incident triplet scores, so the score of every edge (vi, rij, vj) enters both K(vi) and K(vj), mechanically increasing the neighbor similarity measured by Eq. (2). The paper provides no permutation or null-model baseline (e.g., randomly permuting entity scores while preserving the graph or comparing against a degree-matched random score assignment), so the claim that knowledgeability is 'correlated among connected entities' above chance is not established. Please add such a baseline and quantify the effect size relative to it.
[Section 2.1, Prompt 1] The binary True/False response from the LLM is used as ground-truth triplet knowledgeability without any validation against external factuality (e.g., held-out KG facts, human evaluation, or an independent QA-based probe). This is load-bearing because all subsequent entity scores, homophily measurements, regression targets, and fine-tuning selections inherit any systematic response bias of the self-report. Please provide a validation study or reinterpret the scores as 'self-reported knowledge' and discuss the implications for the findings.
[Table 2, Section 4] The claim that Graph-FT 'consistently outperforms' Random-FT is not supported by Table 2: Random-FT is better on T-Rex/Qwen2.5 (84.80 vs 83.25), WD50/Mistral (56.25 vs 55.12), and CoDEx/Llama3 (78.75 vs 75.62). Moreover, no error bars, seeds, or significance tests are reported for any Table 2 number, so the headline average gap (69.04 vs 62.09) may be within run-to-run variability. Please report multiple seeds with variance and paired significance tests.
[Section D.2] The two fine-tuning strategies differ in selection unit and tie-breaking, not only in knowledgeability: Graph-FT selects entities (breaking ties by lowest degree) while Random-FT samples triplets uniformly, so the Graph-FT gain could reflect differences in degree distribution or topical diversity rather than the value of targeting less-known knowledge. An entity-random control that uses the exact Graph-FT selection procedure but with random ignorance scores is needed to isolate the mechanism.

minor comments (6)

[Section 2.2, Eq. (2)] The text says 'neighbors N(vj)' but the summation is over neighbors N(vi) of the central node; please fix the notation.
[Finding 2, Figure 2(d)] The text states that average graph homophily 'remains above 0.5' while the Figure 2(d) caption says 'exceeds 0.6'; please reconcile the threshold.
[Finding 4, Table 1] Finding 4 states regression absolute errors are 'between 0.15 and 0.25', but Table 1 values imply a wider range (e.g., MVPKG w/o t at 70% corresponds to MAE 0.30); please correct the reported range.
[Appendix D.2, Table 2] The model names are inconsistent: the fine-tuning section uses 'Llama 3.1 8B' while Table 2 and the abstract use 'Llama3 8B'; please standardize.
[Figure 2] The caption labels (a) and (b) jointly as 'Distribution of node knowledgeability/homophily', which is ambiguous; please label each panel separately.
[Section 2.3, Eq. (3)] The GNN update in Eq. (3) uses MPl and TRl but does not specify the layer types, hidden sizes, or number of layers; please add these details for reproducibility.

Circularity Check

1 steps flagged · score 5.0 of 10

Knowledge homophily is partly forced by the definition of entity knowledgeability; the Graph-FT fine-tuning result is not circular by construction, though its evaluation holdout is unstated.

self definitional [Section 2.2, Eqs. (1)-(2), and Finding 2 in Figure 2(b)/(d).]
"K(vi) = |T (vi)|−1 X (vi,rij ,vj )∈T (vi) K(vi, rij, vj) (1) ... Hvi = 1− 1 |N (vi)| X vj ∈N (vi) |K(vi) − K(vj)|, (2). In Figure 2(b), these distributions are all right-skewed, with a peak around 0.8, suggesting that nodes and their neighbors tend to share similar knowledgeability scores."

Eq. (1) defines K(vi) as the mean of the binary triplet scores incident to vi. For any edge (vi, vj), the same triplet score K(vi, rij, vj) enters both K(vi) and K(vj), which Eq. (2) then compares through |K(vi) − K(vj)|. Therefore adjacent entity scores always share one component, mechanically shrinking the average absolute difference and inflating H even if triplet scores were independent random variables. The paper reports homophily consistently above 0.5 and concludes that factual recognition is 'not randomly distributed' in the graph, but it provides no permutation or null-model baseline to separate this definitional overlap from genuine knowledge correlation. The homophily 'discovery' is hence partly forced by construction.

full rationale

Aside from the homophily metric, the paper's remaining derivation chain is largely self-contained. Triplet scores come from an external self-report prompt; entity scores are a defined aggregation; the GNN regression is trained on a subset of entity scores and evaluated on held-out entity scores; Graph-FT's selection uses GNN predictions as inputs, not as outputs. No load-bearing self-citation chain or imported uniqueness theorem appears. However, the homophily finding, one of the paper's two headline novel structural patterns and the stated motivation for the GNN regression, is partially definitional: Eq. (1) feeds every edge's triplet score into both endpoint knowledgeability values, which Eq. (2) then compares, mechanically raising neighbor similarity. A second, non-circularity concern is that Appendix D.2 never states that the random evaluation triplets are excluded from the fine-tuning budget; if the budget overlaps the evaluation set, the Table 2 Graph-FT gains (69.04 vs 62.09 vs 49.64) could reflect label memorization rather than targeted knowledge selection. Because this is an omitted holdout control rather than an equation-level reduction, it is counted as a correctness risk, not as an additional circularity step.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central analysis rests on the self-reported knowledge metric, the aggregation choice, the largest-connected-component reduction, and the popularity explanation. The GNN and fine-tuning experiments introduce no new entities, but the regression targets are defined by the same unvalidated probing labels.

free parameters (3)

Fine-tuning triplet budget per dataset = not specified
Budget is 'adjusted according to the domain and size of the dataset' (Appendix D.2), making comparisons across datasets dependent on hand-chosen budgets.
Initial query fraction = 20%
The 20% initial probe / 80% selection split is a design choice not derived from data; results may depend on it.
GNN hyperparameters = not reported
The paper states identical hyperparameters within each dataset but does not list them, so the regression results are not independently tunable.

assumptions (4)

domain assumption A binary True/False self-assessment by the LLM accurately reflects whether it knows the stated fact.
Prompt 1 (Section 2.1) treats model self-report as ground truth; no external factuality check is performed.
ad hoc to paper Entity knowledgeability is appropriately defined as the unweighted mean of its incident triplet scores.
Eq. (1) in Section 2.2; this choice creates overlap between neighbors' scores and is not justified against alternatives.
domain assumption The largest strongly connected component is representative of the full knowledge graph for these analyses.
Appendix C reduces each dataset to an LCC, changing degree distributions (e.g., PharmKG avg degree 132 to 28), which may affect degree-knowledgeability correlations.
ad hoc to paper High-degree entities are more likely to appear in pre-training corpora, explaining the degree-knowledge correlation.
Used in Finding 2 and Section F; no corpus exposure data is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Graph Perspective to Probe Structural Patterns of Knowledge in Large Language Models." pith.science (2026). https://pith.science/paper/H5L3UPNY

@misc{pith2026250519286,
  author       = {Pith},
  title        = {Pith review of: A Graph Perspective to Probe Structural Patterns of Knowledge in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H5L3UPNY}},
  note         = {Machine review of arXiv:2505.19286}
}

read the original abstract

Large language models have been extensively studied as neural knowledge bases for their knowledge access, editability, reasoning, and explainability. However, few works focus on the structural patterns of their knowledge. Motivated by this gap, we investigate these structural patterns from a graph perspective. We quantify the knowledge of LLMs at both the triplet and entity levels, and analyze how it relates to graph structural properties such as node degree. Furthermore, we uncover the knowledge homophily, where topologically close entities exhibit similar levels of knowledgeability, which further motivates us to develop graph machine learning models to estimate entity knowledge based on its local neighbors. This model further enables valuable knowledge checking by selecting triplets less known to LLMs. Empirical results show that using selected triplets for fine-tuning leads to superior performance.

Figures

Figures reproduced from arXiv: 2505.19286 by the authors.

**Figure 2.** (a)/(b): Distribution of node knowledgeability/homophily for each dataset; (c): Node knowledgeability increases as node degree increases. The results here are based on GPT3.5, and results for other LLMs hold similar observations in Appendix E. (d): Average homophily for all datasets given by different LLMs exceeds 0.6. scores. We perform message-passing (MP) and feature transformation (TR) followed by regression: K… view at source ↗

**Figure 3.** Relation between regression performance and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: LLaMa (a): Distribution of node knowledgeability for each dataset; (b): Distribution of node homophily for each dataset; (c): Node knowledgeability increases as node degree increases [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

**Figure 5.** Figure 5: Deepseek (a): Distribution of node knowledgeability for each dataset; (b): Distribution of node homophily for each dataset; (c): Node knowledgeability increases as node degree increases [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: Gemini (a): Distribution of node knowledgeability for each dataset; (b): Distribution of node homophily for each dataset; (c): Node knowledgeability increases as node degree increases [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

**Figure 7.** Figure 7: GPT4o (a): Distribution of node knowledgeability for each dataset; (b): Distribution of node homophily for each dataset; (c): Node knowledgeability increases as node degree increases. F KG vs Topology Analysis across models For each node, we calculate its correspondi…

**Figure 8.** Figure 8: GPT4o - Relationship between Mean Knowledgeability and [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

**Figure 9.** Figure 9: LLaMa - Relationship between Mean Knowledgeability and [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

**Figure 10.** Figure 10: Gemini - Relationship between Mean Knowledgeability and [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]

**Figure 11.** Figure 11: Deepseek - Relationship between Mean Knowledgeability and [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

**Figure 12.** Figure 12: GPT3.5 - Relationship between Mean Knowledgeability and [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 18 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin L Edelman. 2024. Distinguishing the knowable from the unknowable with language models. arXiv preprint arXiv:2402.03563

arXiv 2024
[4]

Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. 2022. A review on language models as knowledge bases. arXiv preprint arXiv:2204.06031

arXiv 2022
[5]

Jeffrey R Binder, Rutvik H Desai, William W Graves, and Lisa L Conant. 2009. Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral cortex, 19(12):2767--2796

work page 2009
[6]

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2021. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696

arXiv 2021
[7]

Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)

work page 2018
[8]

Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625--630

2024

Show all 40 references

[9]

Mikhail Galkin, Priyansh Trivedi, Gaurav Maheshwari, Ricardo Usbeck, and Jens Lehmann. 2020. Message passing for hyper-relational knowledge graphs. arXiv preprint arXiv:2009.10847

2020 arXiv
[10]

Mona M Garvert, Raymond J Dolan, and Timothy EJ Behrens. 2017. A map of abstract relational knowledge in the human hippocampal--entorhinal cortex. elife, 6:e17086

2017
[11]

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2020. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913

2020 arXiv
[12]

Qiyuan He, Yizhong Wang, and Wenya Wang. 2024. Can language models act as knowledge bases at scale? arXiv preprint arXiv:2402.14273

2024 arXiv
[13]

Benjamin Heinzerling and Kentaro Inui. 2020. Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries. arXiv preprint arXiv:2008.09036

2020 arXiv
[14]

Renske S Hoedemaker and Peter C Gordon. 2017. The onset and time course of semantic priming during rapid recognition of visual words. Journal of Experimental Psychology: Human Perception and Performance, 43(5):881

2017
[15]

Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Fr \'e d \'e ric E Theunissen, and Jack L Gallant. 2016. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600):453--458

2016
[16]

Shengbin Jia, Yang Xiang, Xiaojun Chen, and Kun Wang. 2019. Triple trustworthiness measurement for knowledge graph. In The World Wide Web Conference, pages 2865--2871

2019
[17]

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

2022 arXiv
[18]

Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958

2021 arXiv
[19]

Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, et al. 2025. Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems. arXiv pre...

2025 arXiv
[20]

Linhao Luo, Thuy-Trang Vu, Dinh Phung, and Gholamreza Haffari. 2023. Systematic assessment of factual knowledge in large language models. arXiv preprint arXiv:2310.11638

2023 arXiv
[21]

Yao Ma, Xiaorui Liu, Neil Shah, and Jiliang Tang. 2021. Is homophily a necessity for graph neural networks? International Conference on Learning Representations

2021
[22]

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359--17372

2022
[23]

Xinyi Mou, Zejun Li, Hanjia Lyu, Jiebo Luo, and Zhongyu Wei. 2024. Unifying local and global knowledge: Empowering large language models as political experts with knowledge graphs. In Proceedings of the ACM Web Conference 2024, pages 2603--2614

2024
[24]

Vishwas Mruthyunjaya, Pouya Pezeshkpour, Estevam Hruschka, and Nikita Bhutani. 2023. Rethinking language models as symbolic knowledge graphs. arXiv preprint arXiv:2308.13676

2023 arXiv
[25]

Fabio Petroni, Tim Rockt \"a schel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International J...

2019
[26]

Pouya Pezeshkpour. 2023. Measuring and modifying factual knowledge in large language models. In 2023 International Conference on Machine Learning and Applications (ICMLA), pages 831--838. IEEE

2023
[27]

Thorsten Rings, Timo Br \"o hl, and Klaus Lehnertz. 2022. Network structure from a characterization of interactions in complex systems. Scientific Reports, 12(1):11742

2022
[28]

Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418--5426

2020
[29]

Tara Safavi and Danai Koutra. 2020. Codex: A comprehensive knowledge graph completion benchmark. arXiv preprint arXiv:2009.07810

2020 arXiv
[30]

Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. 2023. Knowledge unlearning for llms: Tasks, methods, and challenges. arXiv preprint arXiv:2311.15766

2023 arXiv
[31]

Linxin Song, Xuwei Ding, Jieyu Zhang, Taiwei Shi, Ryotaro Shimizu, Rahul Gupta, Yang Liu, Jian Kang, and Jieyu Zhao. 2025. Discovering knowledge deficiencies of language models on massive knowledge base. arXiv preprint arXiv:2503.23361

2025 arXiv
[32]

Kai Sun, Yifan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. 2024. Head-to-tail: How knowledgeable are large language models ( LLM s)? A . K . A . will LLM s replace knowledge graphs? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Comp...

2024
[33]

Yu Wang and Tyler Derr. 2021. Tree decomposed graph neural network. In Proceedings of the 30th ACM international conference on information & knowledge management, pages 2040--2049

2021
[34]

Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis Ioannidis, Karthik Subbian, James Y Zou, and Jure Leskovec. 2024. Stark: Benchmarking llm retrieval on textual and relational knowledge bases. Advances in Neural Information Processing Sys...

2024
[35]

Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuan-Jing Huang. 2023. Do large language models know what they don’t know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653--8665

2023
[36]

Chenhan Yuan, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024. Back to the future: Towards explainable temporal reasoning with large language models. In Proceedings of the ACM Web Conference 2024, pages 1963--1974

2024
[37]

Danna Zheng, Mirella Lapata, and Jeff Z Pan. 2024. Large language models as reliable knowledge bases? arXiv preprint arXiv:2407.13578

2024 arXiv
[38]

Shangshang Zheng, He Bai, Yizhe Zhang, Yi Su, Xiaochuan Niu, and Navdeep Jaitly. 2023. Kglens: Towards efficient and effective knowledge probing of large language models with knowledge graphs. arXiv preprint arXiv:2312.11539

2023 arXiv
[39]

Shuangjia Zheng, Jiahua Rao, Ying Song, Jixian Zhang, Xianglu Xiao, Evandro Fei Fang, Yuedong Yang, and Zhangming Niu. 2021. Pharmkg: a dedicated knowledge graph benchmark for bomedical data mining. Briefings in bioinformatics

2021
[40]

Yanxu Zhu, Jinlin Xiao, Yuhang Wang, and Jitao Sang. 2024. Kg-fpq: Evaluating factuality hallucination in llms with knowledge graph-based false premise questions. arXiv preprint arXiv:2407.05868

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin L Edelman. 2024. Distinguishing the knowable from the unknowable with language models. arXiv preprint arXiv:2402.03563

arXiv 2024

[4] [4]

Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. 2022. A review on language models as knowledge bases. arXiv preprint arXiv:2204.06031

arXiv 2022

[5] [5]

Jeffrey R Binder, Rutvik H Desai, William W Graves, and Lisa L Conant. 2009. Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral cortex, 19(12):2767--2796

work page 2009

[6] [6]

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2021. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696

arXiv 2021

[7] [7]

Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)

work page 2018

[8] [8]

Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625--630

2024