On Entity Identification in Language Models

Benjamin Heinzerling; Kentaro Inui; Masaki Sakata; Sho Yokoi; Takumi Ito

REVIEW 4 major objections 5 minor 13 references

On Entity Identification in Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Autoregressive language models identify named entities internally, clustering mentions of the same entity and separating different ones with AUC scores of 0.8–0.9 for ambiguous mentions.

desk verdict A useful entity-identity probing framework with a real methodological hole: all headline scores ride on LDA fit to the labels being tested, so the true separability numbers are lower than reported; the behavioral experiments partially rescue the thesis. read the letter →

arxiv 2506.02701 v4 pith:4U54PIA4 submitted 2025-06-03 cs.CL

Masaki Sakata , Benjamin Heinzerling , Sho Yokoi , Takumi Ito , Kentaro Inui This is my paper

classification cs.CL

keywords entityidentificationmentionambiguityvariabilityinternalrepresentationslow-dimensionalsubspacepurityinverselanguagemodelinterpretability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether language models internally know who is who: whether mentions such as 'Obama' and 'Barack Obama' are encoded as the same real-world person, and whether a name like 'Georgia' that can be a country or a U.S. state is encoded differently by context. The authors measure this by treating entity mentions as classes and testing, with Purity and Inverse Purity cluster metrics, whether hidden states of the same entity form compact clusters that stay separated from other entities. Across five autoregressive Transformers they report AUC scores of 0.8–0.9 under mention ambiguity and 0.66–0.8 under mention variability, outperforming surface-only baselines on both axes, with the separating information concentrated in a low-dimensional linear subspace near layer 8 of a 32-layer model. If these results hold, LM hidden states carry entity identity as separable geometry that text-only training produces spontaneously, tying representation structure to real-world, entity-centric knowledge.

What carries the argument

The measuring instrument is an F1 score built from local Purity and local Inverse Purity. For each entity, the mention embeddings define a gold class; assigning every embedding to the nearest entity centroid defines a cluster. Purity asks how much a cluster contains a single class, Inverse Purity asks how much a class is contained in a single cluster, and their harmonic mean scores how cleanly the geometry separates entities. This is applied along two difficulty axes: mention ambiguity, the entropy of the mention-to-entity candidate distribution, and mention variability, the average normalized Levenshtein distance among surface forms of one entity, with AUC summarizing separation across difficulty bins. All headline scores are computed in a 20-dimensional space produced by linear discriminant analysis, chosen because LMs keep high F1 at low dimension while random embeddings only saturate at high dimension.

What would settle it

Retrain the 20-dimensional LDA projector on entity labels from a randomly held-out half of the mentions and score Purity/IP F1 on the other half, or replace LDA with an unsupervised random projection to the same dimension; if AUC drops well below the reported 0.8–0.9 for ambiguity and 0.66–0.8 for variability, the low-dimensional subspace claim is largely an artifact of label-informed projection. A parallel check is to measure nearest-centroid entity agreement directly in the full native dimension, where the curse of dimensionality would need a different statistical treatment.

Watch

Extended reading notes

Core claim

The central claim is that autoregressive language models identify entities in their internal representations: the hidden state of a mention's last token places that mention near other mentions of the same entity and away from mentions of different entities, with F1-derived AUC scores of roughly 0.8–0.9 for ambiguous mentions and 0.66–0.8 for variable ones, clearly above baseline embeddings that only encode surface forms. Layer-by-layer analysis shows the separation is built up through contextualization and peaks in early layers, around one-fifth of model depth, then declines in later layers, which the authors attribute to the pull of next-token prediction. Projecting the hidden states to 20 dimensions with linear discriminant analysis leaves the separation almost intact, which the paper takes as evidence that entity identity is stored compactly in a low-dimensional linear subspace. The same geometry predicts behavior: entities whose mentions are poorly clustered yield inconsistent answers to factual prompts (roughly 71% consistency at low mention variability versus 39% at high variability), and better-separated representations improve accuracy when patched into an entity-disambiguation prompt.

Load-bearing premise

The result depends on the assumption that the 20-dimensional projection via linear discriminant analysis, fitted on the same gold entity labels that are later used for scoring, does not leak label information that inflates the measured clustering; if that assumption fails, the reported separation overstates how well LMs distinguish entities in their native representation space.

Editorial extensions

If this is right

Entity identity is readable directly from LM hidden states, so entity linking and disambiguation could be performed with clustering geometry instead of task-specific fine-tuning.
Entity knowledge is localized in early layers (around layer 8 of 32) in a low-dimensional subspace, so interpretability work seeking entity identity should target those layers and a handful of dimensions.
Representation separation is behaviorally consequential: factual prediction consistency for an entity tracks how tightly its mentions cluster, providing an internal signal for where a model will contradict itself.
Patching early-layer entity representations into prompts drives binary entity disambiguation to roughly 74 percent accuracy from a 50 percent chance baseline, suggesting the subspace can be steered or edited.
Cross-model similarity of entity representation structure supports convergence toward a shared, entity-centric geometry, consistent with the Platonic Representation Hypothesis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A decisive check the paper does not run: train the LDA projector on gold labels from one half of the mentions and score on the other half, or project with a random matrix of equal dimension; if separation collapses, the headline numbers partly reflect label-informed projection rather than native geometry.
The same metrics give a cross-lingual test the authors list as future work: if 'Tokyo,' '東京,' and '东京' cluster together in one model, entity-identity geometry is language-independent, and if they do not, the isomorphism claim needs qualification.
The output-consistency result implies a practical diagnostic: entities whose mentions fail to cluster could be flagged as low-confidence for fact recall, a reliability signal that avoids sampling many paraphrases.
The early-layer subspace invites a causal manipulation: surgical, rank-constrained edits that sharpen the subspace should raise entity-disambiguation accuracy and blurring it should lower it, which the paper's correlational evidence alone does not settle.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing you should know: the main quantitative claims -- LMs achieve AUC 0.8-0.9 for ambiguity and 0.66-0.8 for variability, and entity information lives in a low-dimensional subspace at early layers -- are all computed after projecting representations to 20 dimensions with LDA trained on the ZELDA gold entity labels used for scoring. That makes the separation partly circular. The subspace claim is the most affected: the 20-d subspace is literally constructed from the labels it is then used to predict. The random baseline is not LDA-projected, so the LM-versus-random comparison is unequal. The fix is straightforward: use a label-free projection (PCA or random) or fit LDA on a held-out split, and apply the same treatment to all baselines.

The paper does not collapse, though. The Purity/IP framework applied to mention ambiguity and variability is a genuinely new framing for entity-centric probing, and the code and data are public. The word-prediction consistency experiment (71% vs 39% consistency for low- vs high-variability entities) is a real behavioral prediction that does not depend on LDA, and it supports the qualitative conclusion that representation geometry affects model behavior. The PatchScope entity disambiguation result is weaker evidence because it also uses LDA embeddings, and the F1-vs-accuracy correlation is small (r ~ 0.18). The limitations section is honest about sample size, English-only data, and the neural collapse caveat. The isomorphism discussion is speculative but clearly framed as interpretation.

Soft spots, in order: (1) the LDA circularity is the main issue; (2) no error bars or significance tests anywhere, which makes the AUC differences hard to interpret; (3) selecting the best layer per model with the highest AUC risks optimistic selection; (4) the 'linearly separable' claim in Section 5.1 is partly an artifact of LDA -- of course the projection makes labels linearly separable. A label-free check is needed before the early-layer subspace claim is taken at face value.

The paper deserves a serious referee. It should go to review, but the authors should be asked to redo the headline analysis with a non-circular projection, or show the main trends survive when the projection is fit on held-out labels. I would promote the behavioral consistency experiment as primary evidence and treat the LDA-based numbers as an upper bound. I'd bring this to a reading group focused on interpretability methodology, mostly as a cautionary example of how easily labels leak into a projection.

Referee Report

4 major / 5 minor

Summary. The paper operationalizes entity identification in autoregressive LMs as the clustering of contextualized mention embeddings, using Purity/Inverse Purity F1 and AUC as difficulty-controlled metrics on the ZELDA dataset. Across five LMs it reports F1/AUC values of 0.66-0.9 for mention ambiguity and variability, with LMs outperforming static baselines; it further claims that entity information is linearly separable in a 20-dimensional subspace at early layers, that representation geometry is similar across models, and that better separation predicts more consistent word prediction and better entity disambiguation under PatchScope.

Significance. Entity identification is an important and underexplored question, and the paper offers a clear formulation, a public code release, a multi-model comparison, and a robustness check via the Adjusted Rand Index. If the 20-dimensional subspace claim were established without label leakage, the early-layer geometry result would be a valuable interpretability contribution. The behavioral experiments (output consistency and PatchScope) are a thoughtful complement to the geometric metrics. However, the headline geometric evidence currently depends on a supervised projection fitted on the same gold labels used for evaluation, so the significance of the central claim is conditional on fixing this leakage.

major comments (4)

[Section 3.4, Appendix B.7.1] The headline F1/AUC numbers in Sections 4 and 5.1 are all computed in a 20-dimensional space obtained by LDA fitted on the ZELDA gold entity labels of the same dataset that is later scored. Because LDA maximizes between-entity separation, these scores measure how well a supervised projection can separate the labels, not how entity identity is organized in the model's native representation. The random and static baselines are not given the same supervised projection, so the LM-versus-baseline comparison is unequal. This directly affects the central claim that entity information is compactly represented in a low-dimensional linear subspace. Please redo the evaluation with a label-free projection (e.g., PCA or random projection), or with LDA fitted only on a training split and applied to a held-out split, and report raw-space metrics for at least one model.
[Appendix A, Eq. (1)-(8)] The Purity/IP computation uses gold entity centroids computed from the same mentions being scored and assigns each mention to the nearest such centroid. This is an in-sample measure: it does not separate the estimation of cluster structure from evaluation, so even raw-space F1 values are optimistically biased. The authors should either use held-out mention or entity splits for centroid estimation or explicitly reframe the metric as an internal cluster-quality measure rather than evidence that LMs generalize entity identity across mentions.
[Section 5.2, Appendix B.8] The PatchScope description is internally inconsistent: it states 'Embedding configuration: 20-dimensional LDA embeddings' while also describing the source representation as the final token representation from the entity mention at layer l, which is 5120-dimensional for Llama-2-13B. If the 20-dimensional LDA projection is patched into the residual stream, the dimension mismatch must be explained; if the full hidden state is patched, the text should say so, since the latter would make the PatchScope result independent of the LDA issue and should be emphasized as separate evidence. As written, the experiment cannot be reproduced and its role as an independent check is unclear.
[Section 4.1, Section 4.2] AUC scores in Figure 5 and Tables 8-9 are reported for the layer with the highest AUC on the same gold labels used for scoring, which is a form of test-set model selection. The improvement from layer 0 to an intermediate layer is a real and interesting trend, but the absolute 'best-layer' AUC overstates typical performance. Please report a fixed layer schedule, or use cross-validated layer selection, and show the sensitivity of the conclusions to the chosen layer.

minor comments (5)

[Section 4.1, Section 4.2] The sentences 'In the upper-left region of Figure 5 shows ...' and 'In the upper-right region of Figure 5 shows ...' are grammatically incomplete; they should read 'The upper-left region of Figure 5 shows ...'.
[Section 5.1] The sentence 'Our results is further evidence to support this hypothesis' should read 'Our results are further evidence supporting this hypothesis'.
[Figure 5] Several labels and annotations in Figure 5 are garbled, including 'l16(AUC084)' and 'Llama213Blayer9'; these should be corrected and made consistent across panels.
[Abstract, Section 4] The abstract reports 'F1 scores ranging from 0.66 to 0.9,' but the main text and tables primarily report AUC values; please clarify which quantity is being summarized.
[Section 5.2] The output-consistency experiment uses only 20 entities per variability group and reports no variance or significance test; please add confidence intervals or a small statistical test to support the 71% versus 39% comparison.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline separability scores are computed in an LDA subspace fit on the same gold entity labels used for scoring, so the low-dimensional subspace and linear-separability claims are partly constructed by the projection.

self definitional [Section 3.4, Section 5.1, Appendix B.7.1]
"We reduced the LM and FastText representations with linear discriminant analysis (LDA). Random embeddings, in contrast, were generated directly at each target dimensionality. ... Therefore, we conduct subsequent analyses in 20-dimensional space where the curse of dimensionality has minimal impact. ... As in §4, the embedding dimensions are reduced to 20 dimensions using LDA."

LDA is a supervised projection that uses the gold ZELDA entity labels to maximize between-entity separation. The same labels are then used to compute Purity/IP F1 scores and AUCs in §4 and §5.1, so the headline numbers measure how well the label-fitted subspace separates the labels, not how entity identity is organized in the model's native representation. The §5.1 claim that entity information lives in a low-dimensional subspace is partly true by construction because the subspace is defined by those labels. Baselines (random, unique mention, FastText) are not LDA-projected, making the LM-versus-baseline comparison unequal.
fitted input called prediction [Section 5.1]
"As in §4, the embedding dimensions are reduced to 20 dimensions using LDA. ... Linear Probing achieved F1 scores of around 0.9. This suggests that entity representations are encoded in an (almost) linearly separable form."

The linear probe is trained on 20-dimensional LDA embeddings, i.e., on a projection explicitly chosen to make entity classes linearly separable. Reporting probe F1 ≈ 0.9 as evidence that the original representations are 'almost linearly separable' is circular: the linear probe can only confirm separation in the LDA subspace, which is constructed for that purpose. It does not test the native high-dimensional representation space.

full rationale

The paper's core behavioral results (word-prediction consistency, Patchscopes disambiguation) are genuine predictions and not fitted, which is why this is not a 8-10. However, every headline F1/AUC and the linear-probing claim in §4–§5.1 are computed after projecting representations with LDA, a supervised method that uses the same gold entity labels that define the evaluation. Because LDA is fit on the full ZELDA dataset (no held-out fit for the projection is described), the 20-dimensional 'low-dimensional subspace' is constructed to separate entities, so the finding that entity information is linearly separable there is partly built into the pipeline. The Purity/IP metric also computes centroids from the gold class labels (§A, Eq. 1–2), making it an in-sample cluster-quality measure; this is standard but not a predictive test. The random/unique/FastText baselines are not LDA-projected, so the reported LM-vs-baseline gaps overstate the native representational advantage. The Patchscopes experiments inherit the same 20-d LDA embeddings, so they provide only partial independent support. The paper's own limitation section notes that Purity/IP is relative to the prepared data, but it does not address the LDA label leakage. Overall, the central subspace/separability claim is partially circular, while the behavioral experiments carry independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central quantitative claims rest on ZELDA annotations, the last-token/repetition representation convention, the Euclidean geometry of LDA-projected embeddings, and the Purity/IP clustering metric. The main free choices are the LDA target dimension (20), the five-instance entity filter, the zero-entropy mention exclusion, and the per-model selection of the best layer for reporting. No invented entities are introduced.

free parameters (4)

LDA target dimensionality = 20
Chosen from slope analysis on the same data (Section 3.4); all main F1/AUC scores reported in 20-dimensional LDA space.
Entity frequency filter = at least 5 instances
Entities with fewer than five mentions are excluded (Section 3.3), which removes hard cases and affects centroid estimation.
Best-layer selection = per-model best AUC layer (e.g., Llama-2 7B layer 8)
F1/AUC reported at the layer with highest AUC for each model (Section 4, Figure 5), a post hoc choice that can overstate performance.
Mention ambiguity subset = H > 0
Mentions with zero entropy are excluded (Section 3.5.1), focusing only on ambiguous mentions and potentially making the task easier.

assumptions (4)

domain assumption ZELDA's mention-to-entity annotations are a correct gold standard for entity identity.
All purity/IP scores treat ZELDA entity labels as ground truth; if labels are noisy, reported separability is mismeasured (Section 3.3).
domain assumption The hidden state of the last subword token of a mention, taken from the second occurrence after sentence repetition, is a sufficient entity representation.
This is assumed in Section 3.2 following Meng et al. (2022), Geva et al. (2023), and Springer et al. (2024); if repetition changes representations, the clusters may not reflect normal LM processing.
ad hoc to paper LDA projection to 20 dimensions preserves the entity-relevant geometry of the representations.
Section 3.4 uses LDA fit on the same entity labels to reduce dimensionality; the claim that entity information lives in a low-dimensional linear subspace depends on this projection being faithful rather than label-leaking.
domain assumption Euclidean distance in the projected space is the right similarity for entity identity.
Purity/IP clustering assigns each embedding to the nearest centroid by Euclidean distance (Appendix A); different distance choices could change the scores.

how reviews work

0 comments

Cite this review

Pith. "Pith review of On Entity Identification in Language Models." pith.science (2026). https://pith.science/paper/4U54PIA4

@misc{pith2026250602701,
  author       = {Pith},
  title        = {Pith review of: On Entity Identification in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4U54PIA4}},
  note         = {Machine review of arXiv:2506.02701}
}

read the original abstract

We analyze the extent to which internal representations of language models (LMs) identify and distinguish mentions of named entities, focusing on the many-to-many correspondence between entities and their mentions. We first formulate two problems of entity mentions -- ambiguity and variability -- and propose a framework analogous to clustering quality metrics. Specifically, we quantify through cluster analysis of LM internal representations the extent to which mentions of the same entity cluster together and mentions of different entities remain separated. Our experiments examine five Transformer-based autoregressive models, showing that they effectively identify and distinguish entities with metrics analogous to precision and recall ranging from 0.66 to 0.9. Further analysis reveals that entity-related information is compactly represented in a low-dimensional linear subspace at early LM layers. Additionally, we clarify how the characteristics of entity representations influence word prediction performance. These findings are interpreted through the lens of isomorphism between LM representations and entity-centric knowledge structures in the real world, providing insights into how LMs internally organize and use entity information.

Figures

Figures reproduced from arXiv: 2506.02701 by the authors.

**Figure 1.** Illustration of entity identification. If “Barack Obama” or “Joe Biden” are represented in various mentions and contexts but still form a cluster as a single entity in the representation space, then we observe successful entity identification. Such entity identification in the representation space suggests that the LM’s representations reflect certain aspects of the structure of real-world, entity-centric knowled… view at source ↗

**Figure 2.** Two major factors that can make entity identi [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Overview of local Purity and local IP based on [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Comparison of F1 scores across different [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 6.** Figure 6: F1 scores based on Purity, IP, and Linear Prob [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 7.** Figure 7: Results of RSA experiments. Each value rep [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 8.** Figure 8: (a) Entity disambiguation accuracy for patch [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 10.** Figure 10: UMAP visualization of the ambiguous mention “Georgia.” In layer 0 of Llama-2 7B, the embeddings are [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

**Figure 12.** Figure 12: Results for all models: mention ambiguity and the separability of entity representations (F1 score). [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]

**Figure 13.** Figure 13: Results for all models: mention variability and the separability of entity representations (F1 score) [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 8 canonical work pages

[1]

}: be := 1 |X e| X xe∈X e xe

For each entity e ∈ E(e.g., ”Biden”), com- pute the centroid be ∈ B of its mention em- beddings X e = {x1 e, x2 e, . . .}: be := 1 |X e| X xe∈X e xe. (1)

work page
[2]

In our experiments, we employed Eu- clidean distances, which are commonly used in NLP

For each embedding x, determine the Cluster(e) it belongs to by assigning it to the nearest centroid be: Cluster(e) := {x | be = arg min b′∈B d(x, b′)}, (2) where d(x, y) represents the distance func- tion. In our experiments, we employed Eu- clidean distances, which are commonly used in NLP. This step is equivalent to an interme- diate stage of k-means c...

work page
[3]

class,” and the blue-shaded area is referred to as the “cluster,

Define the gold class for entity e as: Class(e) := X e. (3) 12In Figure 3, the red-shaded area is referred to as the “class,” and the blue-shaded area is referred to as the “cluster,” which are used to calculate Equations 4 and 6. We calculate the local purity and local IP by considering each embedding’s “class” as the entity it refers to and each embeddi...

work page
[4]

Compute the local purity of the Cluster(e) by calculating the fraction of embeddings in the cluster that belong to the most frequent class ˆe: local Purity(e) := |Cluster(e) ∩ Class(ˆe)| |Cluster(e)| , (4) where Class(ˆe) is the most frequent class in the cluster, and |Cluster(e)| is the total num- ber of embeddings in the Cluster(e). Purity The overall p...

work page
[5]

Association for Computational Linguistics

Do language models know the way to Rome? In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 510–517, Punta Cana, Dominican Repub- lic. Association for Computational Linguistics. Raja Marjieh, Ilia Sucholutsky, Pol van Rijn, Nori Ja- coby, and Thomas L. Griffiths. 2023. Large language models predi...

arXiv 2023
[11]

We use the definitions of Cluster(e) from Equation 2 and Class(e) from Equation 3 for each entity e

work page
[12]

Does X refer to A or B? One-word answer only. A: {option_a} B: {op- tion_b} Answer:

Compute the local inverse purity local IP(e) by calculating the fraction of embeddings from the gold class Class(e) that are con- tained within the cluster Cluster(e): local IP(e) := |Cluster(e) ∩ Class(e)| |Class(e)| , (6) where |Class(e)| is the total number of em- beddings in the gold class Class(e). Inverse Purity (IP) The overall inverse purity, repr...

work page 2021
[1985]

Georgia

to evaluate the extent to which our models’ internal representations facilitate entity identifica- tion. The Adjusted Rand Index is a measure of the similarity between two data clusterings, accounting for chance. In this context, it serves as an alter- native approach to assessing the quality of entity identification, complementing our primary metrics. D....

work page

Show all 13 references

[2013]

In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751, Atlanta, Georgia

Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751, Atlanta, Georgia. Association for Computational L...

2013 arXiv
[2021]

In Proceedings of the 25th Conference on Computa- tional Natural Language Learning, pages 109–132, Online

Can language models encode perceptual struc- ture without grounding? a case study in color. In Proceedings of the 25th Conference on Computa- tional Natural Language Learning, pages 109–132, Online. Association for Computational Linguistics. Guillaume Alain and Yoshua Bengio. ...

2017 arXiv
[2022]

Preprint, arXiv:2202.08384

Limitations of neural collapse for under- standing generalization in deep learning. Preprint, arXiv:2202.08384. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume La...

2023 arXiv
[2023]

In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12397–12420, Singapore

Bridging information-theoretic and geometric compression in language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12397–12420, Singapore. Association for Computational Linguis- tics. Roi Cohen, Mor Geva, Jonathan Bera...

2023 arXiv
[2024]

John Hewitt and Christopher D

OpenReview.net. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word represen- tations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, V...

2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

}: be := 1 |X e| X xe∈X e xe

For each entity e ∈ E(e.g., ”Biden”), com- pute the centroid be ∈ B of its mention em- beddings X e = {x1 e, x2 e, . . .}: be := 1 |X e| X xe∈X e xe. (1)

work page

[2] [2]

In our experiments, we employed Eu- clidean distances, which are commonly used in NLP

For each embedding x, determine the Cluster(e) it belongs to by assigning it to the nearest centroid be: Cluster(e) := {x | be = arg min b′∈B d(x, b′)}, (2) where d(x, y) represents the distance func- tion. In our experiments, we employed Eu- clidean distances, which are commonly used in NLP. This step is equivalent to an interme- diate stage of k-means c...

work page

[3] [3]

class,” and the blue-shaded area is referred to as the “cluster,

Define the gold class for entity e as: Class(e) := X e. (3) 12In Figure 3, the red-shaded area is referred to as the “class,” and the blue-shaded area is referred to as the “cluster,” which are used to calculate Equations 4 and 6. We calculate the local purity and local IP by considering each embedding’s “class” as the entity it refers to and each embeddi...

work page

[4] [4]

Compute the local purity of the Cluster(e) by calculating the fraction of embeddings in the cluster that belong to the most frequent class ˆe: local Purity(e) := |Cluster(e) ∩ Class(ˆe)| |Cluster(e)| , (4) where Class(ˆe) is the most frequent class in the cluster, and |Cluster(e)| is the total num- ber of embeddings in the Cluster(e). Purity The overall p...

work page

[5] [5]

Association for Computational Linguistics

Do language models know the way to Rome? In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 510–517, Punta Cana, Dominican Repub- lic. Association for Computational Linguistics. Raja Marjieh, Ilia Sucholutsky, Pol van Rijn, Nori Ja- coby, and Thomas L. Griffiths. 2023. Large language models predi...

arXiv 2023

[6] [11]

We use the definitions of Cluster(e) from Equation 2 and Class(e) from Equation 3 for each entity e

work page

[7] [12]

Does X refer to A or B? One-word answer only. A: {option_a} B: {op- tion_b} Answer:

Compute the local inverse purity local IP(e) by calculating the fraction of embeddings from the gold class Class(e) that are con- tained within the cluster Cluster(e): local IP(e) := |Cluster(e) ∩ Class(e)| |Class(e)| , (6) where |Class(e)| is the total number of em- beddings in the gold class Class(e). Inverse Purity (IP) The overall inverse purity, repr...

work page 2021

[8] [1985]

Georgia

to evaluate the extent to which our models’ internal representations facilitate entity identifica- tion. The Adjusted Rand Index is a measure of the similarity between two data clusterings, accounting for chance. In this context, it serves as an alter- native approach to assessing the quality of entity identification, complementing our primary metrics. D....

work page