REVIEW 3 major objections 4 minor 20 references
IRIS claims that a frozen LLM's internal states can be distilled into reusable identity vectors that align entities across knowledge graphs without any alignment-specific training or candidate-wise inference, reaching 100%, 99.38%, 98.31%,
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 01:58 UTC pith:HZJHHB6R
load-bearing objection Clean, well-specified idea for training-free entity alignment from frozen LLM representations, but the near-perfect benchmark scores may lean on the LLM's memorized knowledge of famous entities, so the 'reusable identity encoder' claim needs a harder test. the 3 major comments →
IRIS: Reusable Identity Representations from Frozen LLMs for Entity Alignment
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
IRIS's central claim is that identity, not description, can be read out of a frozen LLM's hidden states. For each entity it constructs a structured context from its resident KG, asks the LLM to complete a canonical full English name and type, builds several name views, and reads weighted post-MLP hidden states of label and type tokens at layers chosen by an intrinsic-dimension criterion. The resulting vectors live in a shared space; alignment is pure cosine ranking. On D-Y-15K V2, DBP-WIKI, ICEWS-WIKI, and ICEWS-YAGO, the best IRIS variants score 100.00, 99.38, 98.31, and 97.99 Hits@1, beating the strongest LLM baseline by 10.31 and 4.49 points on the two ICEWS benchmarks.
What carries the argument
Contextual identity completion: a generation template that recovers a complementary full English name and entity type from the local KG context. Multi-view extraction: original name, aliases, and completed name each populate a 'label' slot in a fixed prompt; post-MLP hidden states of label/type tokens are pooled with position-dependent weights. Intrinsic-dimension layer selection: a label-free TwoNN estimator finds the first local minimum after the initial expansion in layer-wise intrinsic dimension, and a fixed layer window is used for all entities and datasets. Multi-view fusion: original-name representation is averaged with the mean of auxiliary-view representations.
Load-bearing premise
The load-bearing premise is that the frozen LLM can, from a serialized local KG context alone, recognize the entity and produce its canonical full English name and type; if it cannot, the completed labels differ across graphs and the signatures are no longer comparable.
What would settle it
Build two small KGs whose entities are recent or obscure (so the LLM has little or no parametric memory of them), run IRIS unchanged, and check Hits@1; a collapse toward chance would indicate the benchmark scores rely on memorized entity knowledge rather than on the identity-elicitation mechanism. An even sharper test is to compare IRIS with and without the contextual-completion step on such entities.
If this is right
- If correct, entity alignment can be done by encoding each entity once and comparing vectors, removing pair-specific training and candidate-wise LLM inference.
- The same cached representations can be reused when a KG is aligned against a different counterpart KG.
- The method requires no alignment labels or parameter updates, so it applies to KGs where supervised training data are scarce.
- Consistent results across two frozen backbones suggest the approach is not tied to one LLM.
- Gains on heterogeneous event-centric KGs imply the method handles descriptive variability better than structure-based baselines.
Where Pith is reading between the lines
- Editorial: the near-perfect scores likely reflect the LLM's parametric familiarity with famous entities; a stress test on obscure or private KGs with minimal aliases would reveal how much of the identity signal comes from pretraining knowledge versus the elicitation mechanism itself.
- Editorial: because the completion step asks for a canonical English name, the method may be weaker for low-resource or strongly non-English KGs unless translated aliases are available; a cross-lingual extension might need a multilingual identity anchor.
- Editorial: the layer-selection heuristic (first local minimum after initial intrinsic-dimension expansion) could be tested on other LLM families as a cheap proxy for optimal readout depth, potentially transferring to embedding tasks beyond entity alignment.
- Editorial: one can probe the identity space by measuring whether the same entity's signature is stable across paraphrased contexts; if not, the 'stability' claim would need qualification.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces IRIS, a training-free framework that constructs entity signatures from frozen LLMs without alignment-specific training or candidate-wise inference. The method serializes each entity's local KG context, asks the LLM to complete the entity's full English name and type, extracts identity-oriented representations from label/type tokens at selected Transformer layers, and fuses multi-view representations. Experiments on four EA benchmarks with Llama 3.1-8B and Qwen 2.5-7B report near-perfect Hits@1 (97.99–100.00), surpassing LLM-based baselines by large margins on the two ICEWS benchmarks. Ablations isolate the contributions of token weighting, contextual identity completion, and readout-layer selection.
Significance. The central idea is original and potentially important: aligning entities through reusable, independently computed signatures from a frozen LLM avoids repeated pair-specific computation and could make EA more scalable. The method is clearly specified (Eqs. 1–6), and the ablations are systematic and show consistent gains from each component. The readout-layer selection via intrinsic dimension is a novel label-free criterion. However, the empirical evaluation has major gaps: the load-bearing identity-completion step is not validated on entities outside LLM pretraining; the layer-window rule appears to be selected on test benchmarks; and the near-ceiling results lack variance and reproducibility artifacts. These issues must be resolved before the paper's central claim is fully supported.
major comments (3)
- [Contextual Identity Completion] The method's signatures are anchored on the completed label and type tokens (Eq. (2)), and Table 2 shows removing this completion degrades Hits@1 by 0.62–1.46 points. However, the four benchmarks contain well-known entities heavily covered in Llama/Qwen pretraining data. The near-perfect results may therefore reflect parametric memory rather than identity evidence from the resident KG. This is load-bearing for the 'reusable identity encoder' claim. Please report (a) completion success and cross-KG consistency rates per dataset; (b) results on entities outside the LLM's pretraining coverage (e.g., synthetic/fictional entities or recent low-resource entities); and (c) a string-matching baseline using only the completed names to quantify the contribution of generated surface forms.
- [Effect of Readout-Layer Selection] Table 3 compares four layer-window rules (L1–L4) on the same four EA benchmarks and selects L4 because it achieves the highest Hits@1 in every setting. Although the intrinsic-dimension trajectory is computed on WikiText-103 without labels, the choice of the rule (first local minimum after expansion) among the four candidates is made on the test benchmarks, effectively tuning a core model choice on the evaluation data. This weakens the claim that the readout window is a principled, label-free choice. Please validate L4 on a held-out benchmark or development split, and report sensitivity of Hits@1 to the window size and position.
- [Experimental Setup] The paper reports single-run results at or near ceiling (e.g., 100.00, 99.38 in Table 1) without error bars, multiple seeds, or bootstrap intervals. No code or data is released, and baseline numbers are taken from prior papers rather than re-run in the IRIS pipeline. Consequently, the claimed improvements (e.g., +10.31 on ICEWS-WIKI) cannot be independently verified or statistically assessed. Please release code and data, report variance over at least three runs, and either re-run baselines under identical conditions or justify why the cited numbers are directly comparable (e.g., identical preprocessing and candidate sets).
minor comments (4)
- [Author Information] The corresponding-author email 'sqlite@nju.edu.cn' appears to be a placeholder; please correct it.
- [Implementation Details] The ablation table headers (Table 2) would be clearer if the component being ablated (token-aware weighting, contextual identity completion) were shown as a subheader; currently the 'Without/With' pairs are easy to misread.
- [Notation] The model name is inconsistent: 'Qwen3.5-4B' in the method text versus 'Qwen 3.5-4B' in tables. Standardize the spacing.
- [Figure 1] Figure 1 is dense; the font in the pyramid and the equation overlays is very small. Consider enlarging the figure or splitting it into two panels for readability.
Circularity Check
No circularity: IRIS derives signatures from resident-KG context and frozen-LLM hidden states, without fitting to alignment labels.
full rationale
The derivation chain is self-contained. Each entity signature is built from the resident KG alone (structured context, name views, completed label/type), then passed through a frozen LLM; Eq. (2) is a deterministic weighted average of post-MLP hidden states, and Eq. (3) is a fixed fusion of view representations. Alignment is then ordinary cosine similarity (Eq. 4). No alignment label, counterpart KG, candidate set, or cross-graph supervision is used to construct the representations, so the reported Hits@1 numbers are not forced by construction from the inputs. The token weights, fusion weight, edit-distance threshold, and layer window are fixed design choices; the intrinsic-dimension layer selection is calibrated on external unlabeled WikiText-103 rather than on entity-alignment labels. Even if some choices were selected after inspecting benchmark tables, that is benchmark tuning or selection bias, not circular reasoning: none of the reported predictions is equivalent to a fitted parameter by equation. The contextual identity completion does rely on the frozen LLM's parametric knowledge, which raises a legitimate out-of-distribution risk on obscure or private entities, but that is an external-validity concern, not a circular derivation. No self-citations are load-bearing, and no uniqueness theorem or prior result by the same authors is invoked to force the method. The paper's central claim therefore has independent empirical content.
Axiom & Free-Parameter Ledger
free parameters (7)
- w0 =
1.0
- alpha =
0.5
- w_sym =
0.25
- w_type =
0.75
- lambda =
0.5
- edit-distance threshold =
floor(min(|a|,|b|)/3)
- layer window selection rule (L4) =
first local minimum after initial ID expansion; layer windows per backbone (e.g., [1,2,3,4] for Llama 3.1-8B)
axioms (5)
- domain assumption Entity identity can be inferred from serialized local KG context (name, type, properties, relations) plus LLM parametric knowledge.
- domain assumption Post-MLP hidden states of label and type tokens at selected transformer layers encode identity-oriented semantics.
- domain assumption Intrinsic-dimension trajectory computed on WikiText-103 transfers to entity-alignment readout-layer selection.
- domain assumption Cosine similarity over L2-normalized signatures ranks correct counterparts correctly.
- domain assumption Baseline results taken from prior papers under supposedly unified settings are accurate and comparable.
read the original abstract
Entity alignment (EA) identifies entities across knowledge graphs (KGs) that refer to the same real-world object. Conventional EA methods mainly exploit explicit graph structures and textual fields, which often provide insufficient semantic understanding to recognize the same entity under heterogeneous descriptions and distinguish it from semantically similar entities. Although large language models (LLMs) offer deeper entity understanding, existing LLM-based EA methods largely use this capability for auxiliary generation or candidate-conditioned decisions. Consequently, such understanding is not distilled into a stable and directly comparable identity space, leaving alignment tied to specific KG pairs or candidate sets and requiring repeated processing as the matching context changes. To address these limitations, we propose IRIS (Identity Representations from Internal States), a training-free framework that constructs for each entity an iris-like signature encoding its distinctive and stable identity characteristics. IRIS derives these signatures by eliciting identity-oriented contextual representations from a frozen LLM, thereby forming a shared space in which each entity is encoded once and can be aligned across different KGs through direct similarity comparison, without pair-dependent representation construction or candidate-wise LLM inference. Across four established EA benchmarks and two frozen LLM backbones, the best IRIS variants achieve Hits@1 scores of 100.00, 99.38, 98.31, and 97.99 on D-Y-15K V2, DBP-WIKI, ICEWS-WIKI, and ICEWS-YAGO, respectively.
Figures
Reference graph
Works this paper leans on
-
[1]
Llm2vec: Large language models are secretly powerful text encoders.arXiv preprint arXiv:2404.05961,
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders.arXiv preprint arXiv:2404.05961,
-
[6]
The llama 3 herd of models.arXiv preprint arXiv:2407.21783,
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhi- navPandey,AbhishekKadian,AhmadAl-Dahle,AieshaLet- man,AkhilMathur,AlanSchelten,AlexVaughan,etal. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,
-
[7]
Unlocking the power of large language models for entity alignment
Xuhui Jiang, Yinghan Shen, Zhichao Shi, Chengjin Xu, Wei Li, Zixuan Li, Jian Guo, Huawei Shen, and Yuanzhuo Wang. Unlocking the power of large language models for entity alignment. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages...
-
[8]
doi: 10.1609/aaai.v39i11.33294. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training llms as generalist embedding models. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors,International Conference on Learning Representations, volume 2025, pages 79310– 79333,
-
[9]
Xin Mao, Wenting Wang, Huimin Xu, Yuanbin Wu, and Man Lan
URL https://proceedings.iclr.cc/paper_files/ paper/2025/file/c4bf73386022473a652a18941e9ea6f8- Paper-Conference.pdf. Xin Mao, Wenting Wang, Huimin Xu, Yuanbin Wu, and Man Lan. Relational reflection entity alignment. InProceedings of the 29th ACM international conference on information & knowledge management, pages 1095–1104,
2025
-
[12]
URL https: //openreview.net/forum?id=Byj72udxe. 8 Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search.arXiv preprint arXiv:2202.08904,
-
[13]
URL https: //arxiv.org/abs/2412.15115. Qwen Team. Qwen3.5: Towards native multimodal agents, February
-
[15]
doi: 10.18653/v1/2024.findings-eacl.58
Association for Computational Lin- guistics. doi: 10.18653/v1/2024.findings-eacl.58. URL https://aclanthology.org/2024.findings-eacl.58/. Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Gra- ham Neubig, and Aditi Raghunathan. Repetition improves language model embeddings. InThe Thirteenth Interna- tionalConferenceonLearningRepresentations,2025. URL ht...
-
[16]
Zequn Sun, Wei Hu, Qingheng Zhang, and Yuzhong Qu. Bootstrapping entity alignment with knowledge graph em- bedding.InProceedingsoftheTwenty-SeventhInternational Joint Conference on Artificial Intelligence, IJCAI-18, pages 4396–4402.InternationalJointConferencesonArtificialIn- telligenceOrganization,2018. doi:10.24963/ijcai.2018/611. URL https://doi.org/10...
Pith/arXiv arXiv 2018
-
[17]
URL https://doi.org/10.24963/ijcai.2020/439
doi: 10.24963/ijcai.2020/439. URL https://doi.org/10.24963/ijcai.2020/439. Main track. Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. Thegeometryofhiddenrepresentationsoflargetransformer models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural...
-
[19]
URL https://doi.org/10.24963/ijcai.2019/733
doi: 10.24963/ijcai.2019/733. URL https://doi.org/10.24963/ijcai.2019/733. Qingheng Zhang, Zequn Sun, Wei Hu, Muhao Chen, Ling- bing Guo, and Yuzhong Qu. Multi-view knowledge graph embedding for entity alignment. InProceedings of the Twenty-Eighth International Joint Conference on Artifi- cial Intelligence, IJCAI-19, pages 5429–5435. Interna- tional Joint...
-
[20]
URL https://doi.org/10.24963/ijcai.2019/754
doi: 10.24963/ijcai.2019/754. URL https://doi.org/10.24963/ijcai.2019/754. 9
-
[2017]
doi: 10.24963/ijcai. 2017/209. URL https://doi.org/10.24963/ijcai.2017/209. Shengyuan Chen, Qinggang Zhang, Junnan Dong, Wen Hua, Qing Li, and Xiao Huang. Entity alignment with noisy annotations from large language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processin...
doi:10.24963/ijcai 2017
-
[2019]
Association for Computational Linguistics. doi: 10.18653/v1/P19-1140. URL https://aclanthology.org/P19-1140/. Muhao Chen, Yingtao Tian, Mohan Yang, and Carlo Zaniolo. Multilingualknowledgegraphembeddingsforcross-lingual knowledge alignment. InProceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 1511–1517,
-
[2020]
Boost- ing the speed of entity alignment 10×: Dual attention matching network with normalized hard sample mining
Xin Mao, Wenting Wang, Yuanbin Wu, and Man Lan. Boost- ing the speed of entity alignment 10×: Dual attention matching network with normalized hard sample mining. InProceedings of the Web Conference 2021, WWW ’21, pages 821–832, New York, NY, USA,
2021
-
[2021]
Association for Computing Machinery. ISBN 9781450383127. doi: 10.1145/3442381.3449897. URL https://doi.org/10.1145/ 3442381.3449897. StephenMerity,CaimingXiong,JamesBradbury,andRichard Socher. Pointer sentinel mixture models. InInternational Conference on Learning Representations,
-
[2023]
doi: 10.52202/075280-2230. URL https://proceedings.neurips.cc/paper_files/paper/ 2023/file/a0e66093d7168b40246af1cddc025daa-Paper- Conference.pdf. Zhichun Wang, Qingsong Lv, Xiaohan Lan, and Yu Zhang. Cross-lingual knowledge graph alignment via graph con- volutional networks. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors,P...
-
[2024]
doi: 10.52202/079017-0482. URL https://proceedings.neurips.cc/paper_files/paper/ 2024/file/1b57aaddf85ab01a2445a79c9edc1f4b-Paper- Conference.pdf. Jingwei Cheng, Chenglong Lu, Linyan Yang, Guoqing Chen, and Fu Zhang. EasyEA: Large language model is all you need in entity alignment between knowledge graphs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova...
-
[2025]
Associ- ation for Computational Linguistics. ISBN 979-8-89176- 256-5. doi: 10.18653/v1/2025.findings-acl.1080. URL https://aclanthology.org/2025.findings-acl.1080/. ElenaFacco,Mariad’Errico,AlexRodriguez,andAlessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information.Scientific reports, 7(1): 12140,
-
[2026]
AntonRazzhigaev,MatveyMikhalchuk,ElizavetaGoncharova, IvanOseledets,DenisDimitrov,andAndreyKuznetsov
URL https://qwen.ai/blog?id=qwen3.5. AntonRazzhigaev,MatveyMikhalchuk,ElizavetaGoncharova, IvanOseledets,DenisDimitrov,andAndreyKuznetsov. The shape of learning: Anisotropy and intrinsic dimensions in transformer-based models. In Yvette Graham and Matthew Purver, editors,Findings of the Association for Computa- tional Linguistics: EACL 2024, pages 868–874...
2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.