Pith. sign in

REVIEW 3 major objections 4 minor 20 references

IRIS claims that a frozen LLM's internal states can be distilled into reusable identity vectors that align entities across knowledge graphs without any alignment-specific training or candidate-wise inference, reaching 100%, 99.38%, 98.31%,

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 01:58 UTC pith:HZJHHB6R

load-bearing objection Clean, well-specified idea for training-free entity alignment from frozen LLM representations, but the near-perfect benchmark scores may lean on the LLM's memorized knowledge of famous entities, so the 'reusable identity encoder' claim needs a harder test. the 3 major comments →

arxiv 2607.25579 v1 pith:HZJHHB6R submitted 2026-07-28 cs.CL cs.AI

IRIS: Reusable Identity Representations from Frozen LLMs for Entity Alignment

classification cs.CL cs.AI
keywords entity alignmentknowledge graphfrozen LLMinternal statesidentity representationtraining-freemulti-view representationentity resolution
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Entity alignment usually needs pair-specific training or candidate-conditioned LLM calls. IRIS instead extracts, from a frozen decoder-only LLM, a per-entity 'iris-like' signature using only that entity's own knowledge graph context, then aligns by direct cosine similarity. The paper reports near-perfect Hits@1 on four benchmarks, with the largest gains on the heterogeneous ICEWS datasets. A sympathetic reader cares because it suggests semantic identity can be captured once, cached, and reused across arbitrary KG pairs without retraining or repeated LLM inference.

Core claim

IRIS's central claim is that identity, not description, can be read out of a frozen LLM's hidden states. For each entity it constructs a structured context from its resident KG, asks the LLM to complete a canonical full English name and type, builds several name views, and reads weighted post-MLP hidden states of label and type tokens at layers chosen by an intrinsic-dimension criterion. The resulting vectors live in a shared space; alignment is pure cosine ranking. On D-Y-15K V2, DBP-WIKI, ICEWS-WIKI, and ICEWS-YAGO, the best IRIS variants score 100.00, 99.38, 98.31, and 97.99 Hits@1, beating the strongest LLM baseline by 10.31 and 4.49 points on the two ICEWS benchmarks.

What carries the argument

Contextual identity completion: a generation template that recovers a complementary full English name and entity type from the local KG context. Multi-view extraction: original name, aliases, and completed name each populate a 'label' slot in a fixed prompt; post-MLP hidden states of label/type tokens are pooled with position-dependent weights. Intrinsic-dimension layer selection: a label-free TwoNN estimator finds the first local minimum after the initial expansion in layer-wise intrinsic dimension, and a fixed layer window is used for all entities and datasets. Multi-view fusion: original-name representation is averaged with the mean of auxiliary-view representations.

Load-bearing premise

The load-bearing premise is that the frozen LLM can, from a serialized local KG context alone, recognize the entity and produce its canonical full English name and type; if it cannot, the completed labels differ across graphs and the signatures are no longer comparable.

What would settle it

Build two small KGs whose entities are recent or obscure (so the LLM has little or no parametric memory of them), run IRIS unchanged, and check Hits@1; a collapse toward chance would indicate the benchmark scores rely on memorized entity knowledge rather than on the identity-elicitation mechanism. An even sharper test is to compare IRIS with and without the contextual-completion step on such entities.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, entity alignment can be done by encoding each entity once and comparing vectors, removing pair-specific training and candidate-wise LLM inference.
  • The same cached representations can be reused when a KG is aligned against a different counterpart KG.
  • The method requires no alignment labels or parameter updates, so it applies to KGs where supervised training data are scarce.
  • Consistent results across two frozen backbones suggest the approach is not tied to one LLM.
  • Gains on heterogeneous event-centric KGs imply the method handles descriptive variability better than structure-based baselines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the near-perfect scores likely reflect the LLM's parametric familiarity with famous entities; a stress test on obscure or private KGs with minimal aliases would reveal how much of the identity signal comes from pretraining knowledge versus the elicitation mechanism itself.
  • Editorial: because the completion step asks for a canonical English name, the method may be weaker for low-resource or strongly non-English KGs unless translated aliases are available; a cross-lingual extension might need a multilingual identity anchor.
  • Editorial: the layer-selection heuristic (first local minimum after initial intrinsic-dimension expansion) could be tested on other LLM families as a cheap proxy for optimal readout depth, potentially transferring to embedding tasks beyond entity alignment.
  • Editorial: one can probe the identity space by measuring whether the same entity's signature is stable across paraphrased contexts; if not, the 'stability' claim would need qualification.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces IRIS, a training-free framework that constructs entity signatures from frozen LLMs without alignment-specific training or candidate-wise inference. The method serializes each entity's local KG context, asks the LLM to complete the entity's full English name and type, extracts identity-oriented representations from label/type tokens at selected Transformer layers, and fuses multi-view representations. Experiments on four EA benchmarks with Llama 3.1-8B and Qwen 2.5-7B report near-perfect Hits@1 (97.99–100.00), surpassing LLM-based baselines by large margins on the two ICEWS benchmarks. Ablations isolate the contributions of token weighting, contextual identity completion, and readout-layer selection.

Significance. The central idea is original and potentially important: aligning entities through reusable, independently computed signatures from a frozen LLM avoids repeated pair-specific computation and could make EA more scalable. The method is clearly specified (Eqs. 1–6), and the ablations are systematic and show consistent gains from each component. The readout-layer selection via intrinsic dimension is a novel label-free criterion. However, the empirical evaluation has major gaps: the load-bearing identity-completion step is not validated on entities outside LLM pretraining; the layer-window rule appears to be selected on test benchmarks; and the near-ceiling results lack variance and reproducibility artifacts. These issues must be resolved before the paper's central claim is fully supported.

major comments (3)
  1. [Contextual Identity Completion] The method's signatures are anchored on the completed label and type tokens (Eq. (2)), and Table 2 shows removing this completion degrades Hits@1 by 0.62–1.46 points. However, the four benchmarks contain well-known entities heavily covered in Llama/Qwen pretraining data. The near-perfect results may therefore reflect parametric memory rather than identity evidence from the resident KG. This is load-bearing for the 'reusable identity encoder' claim. Please report (a) completion success and cross-KG consistency rates per dataset; (b) results on entities outside the LLM's pretraining coverage (e.g., synthetic/fictional entities or recent low-resource entities); and (c) a string-matching baseline using only the completed names to quantify the contribution of generated surface forms.
  2. [Effect of Readout-Layer Selection] Table 3 compares four layer-window rules (L1–L4) on the same four EA benchmarks and selects L4 because it achieves the highest Hits@1 in every setting. Although the intrinsic-dimension trajectory is computed on WikiText-103 without labels, the choice of the rule (first local minimum after expansion) among the four candidates is made on the test benchmarks, effectively tuning a core model choice on the evaluation data. This weakens the claim that the readout window is a principled, label-free choice. Please validate L4 on a held-out benchmark or development split, and report sensitivity of Hits@1 to the window size and position.
  3. [Experimental Setup] The paper reports single-run results at or near ceiling (e.g., 100.00, 99.38 in Table 1) without error bars, multiple seeds, or bootstrap intervals. No code or data is released, and baseline numbers are taken from prior papers rather than re-run in the IRIS pipeline. Consequently, the claimed improvements (e.g., +10.31 on ICEWS-WIKI) cannot be independently verified or statistically assessed. Please release code and data, report variance over at least three runs, and either re-run baselines under identical conditions or justify why the cited numbers are directly comparable (e.g., identical preprocessing and candidate sets).
minor comments (4)
  1. [Author Information] The corresponding-author email 'sqlite@nju.edu.cn' appears to be a placeholder; please correct it.
  2. [Implementation Details] The ablation table headers (Table 2) would be clearer if the component being ablated (token-aware weighting, contextual identity completion) were shown as a subheader; currently the 'Without/With' pairs are easy to misread.
  3. [Notation] The model name is inconsistent: 'Qwen3.5-4B' in the method text versus 'Qwen 3.5-4B' in tables. Standardize the spacing.
  4. [Figure 1] Figure 1 is dense; the font in the pyramid and the equation overlays is very small. Consider enlarging the figure or splitting it into two panels for readability.

Circularity Check

0 steps flagged

No circularity: IRIS derives signatures from resident-KG context and frozen-LLM hidden states, without fitting to alignment labels.

full rationale

The derivation chain is self-contained. Each entity signature is built from the resident KG alone (structured context, name views, completed label/type), then passed through a frozen LLM; Eq. (2) is a deterministic weighted average of post-MLP hidden states, and Eq. (3) is a fixed fusion of view representations. Alignment is then ordinary cosine similarity (Eq. 4). No alignment label, counterpart KG, candidate set, or cross-graph supervision is used to construct the representations, so the reported Hits@1 numbers are not forced by construction from the inputs. The token weights, fusion weight, edit-distance threshold, and layer window are fixed design choices; the intrinsic-dimension layer selection is calibrated on external unlabeled WikiText-103 rather than on entity-alignment labels. Even if some choices were selected after inspecting benchmark tables, that is benchmark tuning or selection bias, not circular reasoning: none of the reported predictions is equivalent to a fitted parameter by equation. The contextual identity completion does rely on the frozen LLM's parametric knowledge, which raises a legitimate out-of-distribution risk on obscure or private entities, but that is an external-validity concern, not a circular derivation. No self-citations are load-bearing, and no uniqueness theorem or prior result by the same authors is invoked to force the method. The paper's central claim therefore has independent empirical content.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

IRIS introduces no new physical or conceptual entities. The 'signature' is a weighted average of LLM hidden states. The main burden is carried by a collection of hand-set hyperparameters and domain assumptions about LLM hidden states and benchmark comparability.

free parameters (7)
  • w0 = 1.0
    Base weight for the first regular label token in Eq. 5; manually set.
  • alpha = 0.5
    Progressive weight increase for label tokens in Eq. 5; manually set.
  • w_sym = 0.25
    Weight for label tokens containing special symbols; fixed by hand.
  • w_type = 0.75
    Weight for type tokens; fixed by hand.
  • lambda = 0.5
    Fusion weight between original-name and auxiliary-view representations in Eq. 6; manually chosen.
  • edit-distance threshold = floor(min(|a|,|b|)/3)
    Threshold for retaining auxiliary name views; manually chosen and fixed across datasets.
  • layer window selection rule (L4) = first local minimum after initial ID expansion; layer windows per backbone (e.g., [1,2,3,4] for Llama 3.1-8B)
    The rule is justified by Table 3 on the same EA benchmarks, so it is effectively a hand-picked hyperparameter validated on the target data.
axioms (5)
  • domain assumption Entity identity can be inferred from serialized local KG context (name, type, properties, relations) plus LLM parametric knowledge.
    The entire method depends on the contextual identity-completion step producing a canonical name/type from local context alone (Method).
  • domain assumption Post-MLP hidden states of label and type tokens at selected transformer layers encode identity-oriented semantics.
    The representation readout (Eq. 2) assumes these hidden states are useful identity vectors; no theorem grounds this.
  • domain assumption Intrinsic-dimension trajectory computed on WikiText-103 transfers to entity-alignment readout-layer selection.
    Layer windows are chosen using ID on WikiText-103, but the optimality of L4 is only shown on EA benchmarks.
  • domain assumption Cosine similarity over L2-normalized signatures ranks correct counterparts correctly.
    Alignment is performed purely by cosine similarity (Eq. 4); this assumes the shared space is sufficiently isotropic.
  • domain assumption Baseline results taken from prior papers under supposedly unified settings are accurate and comparable.
    Baselines are copied from LLM4EA and ChatEA papers; no independent re-run is performed.

pith-pipeline@v1.3.0-alltime-deepseek · 12613 in / 8476 out tokens · 82498 ms · 2026-08-01T01:58:56.952644+00:00 · methodology

0 comments
read the original abstract

Entity alignment (EA) identifies entities across knowledge graphs (KGs) that refer to the same real-world object. Conventional EA methods mainly exploit explicit graph structures and textual fields, which often provide insufficient semantic understanding to recognize the same entity under heterogeneous descriptions and distinguish it from semantically similar entities. Although large language models (LLMs) offer deeper entity understanding, existing LLM-based EA methods largely use this capability for auxiliary generation or candidate-conditioned decisions. Consequently, such understanding is not distilled into a stable and directly comparable identity space, leaving alignment tied to specific KG pairs or candidate sets and requiring repeated processing as the matching context changes. To address these limitations, we propose IRIS (Identity Representations from Internal States), a training-free framework that constructs for each entity an iris-like signature encoding its distinctive and stable identity characteristics. IRIS derives these signatures by eliciting identity-oriented contextual representations from a frozen LLM, thereby forming a shared space in which each entity is encoded once and can be aligned across different KGs through direct similarity comparison, without pair-dependent representation construction or candidate-wise LLM inference. Across four established EA benchmarks and two frozen LLM backbones, the best IRIS variants achieve Hits@1 scores of 100.00, 99.38, 98.31, and 97.99 on D-Y-15K V2, DBP-WIKI, ICEWS-WIKI, and ICEWS-YAGO, respectively.

Figures

Figures reproduced from arXiv: 2607.25579 by Ge Wang, Shengtao Li, Shouqian Shi, Xinran Liu, Xin-Wei Yao.

Figure 1
Figure 1. Figure 1: Overview of IRIS. (a) For each entity, IRIS independently organizes identity evidence from its resident KG and applies [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 6 canonical work pages

  1. [1]

    Llm2vec: Large language models are secretly powerful text encoders.arXiv preprint arXiv:2404.05961,

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders.arXiv preprint arXiv:2404.05961,

  2. [6]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhi- navPandey,AbhishekKadian,AhmadAl-Dahle,AieshaLet- man,AkhilMathur,AlanSchelten,AlexVaughan,etal. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

  3. [7]

    Unlocking the power of large language models for entity alignment

    Xuhui Jiang, Yinghan Shen, Zhichao Shi, Chengjin Xu, Wei Li, Zixuan Li, Jian Guo, Huawei Shen, and Yuanzhuo Wang. Unlocking the power of large language models for entity alignment. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages...

  4. [8]

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping

    doi: 10.1609/aaai.v39i11.33294. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training llms as generalist embedding models. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors,International Conference on Learning Representations, volume 2025, pages 79310– 79333,

  5. [9]

    Xin Mao, Wenting Wang, Huimin Xu, Yuanbin Wu, and Man Lan

    URL https://proceedings.iclr.cc/paper_files/ paper/2025/file/c4bf73386022473a652a18941e9ea6f8- Paper-Conference.pdf. Xin Mao, Wenting Wang, Huimin Xu, Yuanbin Wu, and Man Lan. Relational reflection entity alignment. InProceedings of the 29th ACM international conference on information & knowledge management, pages 1095–1104,

  6. [12]

    8 Niklas Muennighoff

    URL https: //openreview.net/forum?id=Byj72udxe. 8 Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search.arXiv preprint arXiv:2202.08904,

  7. [13]

    Qwen Team

    URL https: //arxiv.org/abs/2412.15115. Qwen Team. Qwen3.5: Towards native multimodal agents, February

  8. [15]

    doi: 10.18653/v1/2024.findings-eacl.58

    Association for Computational Lin- guistics. doi: 10.18653/v1/2024.findings-eacl.58. URL https://aclanthology.org/2024.findings-eacl.58/. Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Gra- ham Neubig, and Aditi Raghunathan. Repetition improves language model embeddings. InThe Thirteenth Interna- tionalConferenceonLearningRepresentations,2025. URL ht...

  9. [16]

    Zequn Sun, Wei Hu, Qingheng Zhang, and Yuzhong Qu. Bootstrapping entity alignment with knowledge graph em- bedding.InProceedingsoftheTwenty-SeventhInternational Joint Conference on Artificial Intelligence, IJCAI-18, pages 4396–4402.InternationalJointConferencesonArtificialIn- telligenceOrganization,2018. doi:10.24963/ijcai.2018/611. URL https://doi.org/10...

  10. [17]

    URL https://doi.org/10.24963/ijcai.2020/439

    doi: 10.24963/ijcai.2020/439. URL https://doi.org/10.24963/ijcai.2020/439. Main track. Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. Thegeometryofhiddenrepresentationsoflargetransformer models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural...

  11. [19]

    URL https://doi.org/10.24963/ijcai.2019/733

    doi: 10.24963/ijcai.2019/733. URL https://doi.org/10.24963/ijcai.2019/733. Qingheng Zhang, Zequn Sun, Wei Hu, Muhao Chen, Ling- bing Guo, and Yuzhong Qu. Multi-view knowledge graph embedding for entity alignment. InProceedings of the Twenty-Eighth International Joint Conference on Artifi- cial Intelligence, IJCAI-19, pages 5429–5435. Interna- tional Joint...

  12. [20]

    URL https://doi.org/10.24963/ijcai.2019/754

    doi: 10.24963/ijcai.2019/754. URL https://doi.org/10.24963/ijcai.2019/754. 9

  13. [2017]

    2017/209

    doi: 10.24963/ijcai. 2017/209. URL https://doi.org/10.24963/ijcai.2017/209. Shengyuan Chen, Qinggang Zhang, Junnan Dong, Wen Hua, Qing Li, and Xiao Huang. Entity alignment with noisy annotations from large language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processin...

  14. [2019]

    doi: 10.18653/v1/P19-1140

    Association for Computational Linguistics. doi: 10.18653/v1/P19-1140. URL https://aclanthology.org/P19-1140/. Muhao Chen, Yingtao Tian, Mohan Yang, and Carlo Zaniolo. Multilingualknowledgegraphembeddingsforcross-lingual knowledge alignment. InProceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 1511–1517,

  15. [2020]

    Boost- ing the speed of entity alignment 10×: Dual attention matching network with normalized hard sample mining

    Xin Mao, Wenting Wang, Yuanbin Wu, and Man Lan. Boost- ing the speed of entity alignment 10×: Dual attention matching network with normalized hard sample mining. InProceedings of the Web Conference 2021, WWW ’21, pages 821–832, New York, NY, USA,

  16. [2021]

    ISBN 9781450383127

    Association for Computing Machinery. ISBN 9781450383127. doi: 10.1145/3442381.3449897. URL https://doi.org/10.1145/ 3442381.3449897. StephenMerity,CaimingXiong,JamesBradbury,andRichard Socher. Pointer sentinel mixture models. InInternational Conference on Learning Representations,

  17. [2023]

    URL https://proceedings.neurips.cc/paper_files/paper/ 2023/file/a0e66093d7168b40246af1cddc025daa-Paper- Conference.pdf

    doi: 10.52202/075280-2230. URL https://proceedings.neurips.cc/paper_files/paper/ 2023/file/a0e66093d7168b40246af1cddc025daa-Paper- Conference.pdf. Zhichun Wang, Qingsong Lv, Xiaohan Lan, and Yu Zhang. Cross-lingual knowledge graph alignment via graph con- volutional networks. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors,P...

  18. [2024]

    URL https://proceedings.neurips.cc/paper_files/paper/ 2024/file/1b57aaddf85ab01a2445a79c9edc1f4b-Paper- Conference.pdf

    doi: 10.52202/079017-0482. URL https://proceedings.neurips.cc/paper_files/paper/ 2024/file/1b57aaddf85ab01a2445a79c9edc1f4b-Paper- Conference.pdf. Jingwei Cheng, Chenglong Lu, Linyan Yang, Guoqing Chen, and Fu Zhang. EasyEA: Large language model is all you need in entity alignment between knowledge graphs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova...

  19. [2025]

    ISBN 979-8-89176- 256-5

    Associ- ation for Computational Linguistics. ISBN 979-8-89176- 256-5. doi: 10.18653/v1/2025.findings-acl.1080. URL https://aclanthology.org/2025.findings-acl.1080/. ElenaFacco,Mariad’Errico,AlexRodriguez,andAlessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information.Scientific reports, 7(1): 12140,

  20. [2026]

    AntonRazzhigaev,MatveyMikhalchuk,ElizavetaGoncharova, IvanOseledets,DenisDimitrov,andAndreyKuznetsov

    URL https://qwen.ai/blog?id=qwen3.5. AntonRazzhigaev,MatveyMikhalchuk,ElizavetaGoncharova, IvanOseledets,DenisDimitrov,andAndreyKuznetsov. The shape of learning: Anisotropy and intrinsic dimensions in transformer-based models. In Yvette Graham and Matthew Purver, editors,Findings of the Association for Computa- tional Linguistics: EACL 2024, pages 868–874...