Quantifying Similarity between Relations with Fact Distribution
Pith reviewed 2026-05-24 18:51 UTC · model grok-4.3
The pith
Divergence between conditional distributions over entity pairs quantifies similarity between relations in knowledge bases.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The similarity between two relations equals the divergence between the conditional probability distributions over entity pairs that instantiate each relation; a sampling procedure from a neural network that parameterizes these distributions yields a practical approximation whose outputs align with human similarity judgments.
What carries the argument
Divergence between conditional probability distributions over entity pairs, parameterized by a simple neural network and approximated by sampling.
If this is right
- The scores can detect redundant relations extracted by open information extraction models.
- Even the strongest relational classification models still confuse very similar relations.
- The measure can be inserted into negative sampling and softmax classification to reduce errors on similar relations.
Where Pith is reading between the lines
- The same distributional divergence idea could be tested on other structured prediction settings such as event or attribute similarity.
- Knowledge-base merging pipelines might use these scores to decide when two relation schemas should be aligned or collapsed.
- Model confusion matrices for relation extraction could be re-ranked by this measure to prioritize error analysis on the most semantically close pairs.
Load-bearing premise
Semantic similarity between relations is captured by low divergence between the distributions of entity pairs they relate.
What would settle it
If the computed divergences show no significant correlation with a fresh set of human similarity ratings collected independently of the original experiments, the central claim would be falsified.
read the original abstract
We introduce a conceptually simple and effective method to quantify the similarity between relations in knowledge bases. Specifically, our approach is based on the divergence between the conditional probability distributions over entity pairs. In this paper, these distributions are parameterized by a very simple neural network. Although computing the exact similarity is in-tractable, we provide a sampling-based method to get a good approximation. We empirically show the outputs of our approach significantly correlate with human judgments. By applying our method to various tasks, we also find that (1) our approach could effectively detect redundant relations extracted by open information extraction (Open IE) models, that (2) even the most competitive models for relational classification still make mistakes among very similar relations, and that (3) our approach could be incorporated into negative sampling and softmax classification to alleviate these mistakes. The source code and experiment details of this paper can be obtained from https://github.com/thunlp/relation-similarity.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a method to quantify similarity between relations in knowledge bases by measuring the divergence between conditional probability distributions over entity pairs, parameterized by a simple neural network. It provides a sampling-based approximation to the intractable exact computation and reports that the resulting similarity scores significantly correlate with human judgments. The method is then applied to detect redundant relations from Open IE systems, to analyze errors among similar relations in competitive relational classification models, and to improve negative sampling and softmax-based classification.
Significance. If the reported human correlation and task improvements hold under the sampling approximation, the work supplies a practical, data-driven similarity measure for relations that can be directly integrated into KB construction and classification pipelines. The public release of code and experiment details is a clear strength that supports reproducibility and follow-on work.
minor comments (3)
- [§3.1] §3.1: the neural network architecture used to parameterize P(entity-pairs | relation) is described only at a high level; an explicit diagram or layer sizes would clarify the 'very simple' claim and aid replication.
- [Table 2] Table 2: the reported correlation coefficients lack confidence intervals or p-values, making it difficult to assess whether the 'significant' correlation is robust across the sampled entity pairs.
- [§4.3] §4.3: when the similarity measure is incorporated into negative sampling, the paper does not state how many negative samples are drawn per positive or whether the sampling is performed once or dynamically during training.
Simulated Author's Rebuttal
We thank the referee for the positive summary, recognition of the method's practicality, and recommendation for minor revision. We appreciate the note on reproducibility via code release.
Circularity Check
No significant circularity; measure is explicitly defined and externally validated
full rationale
The paper defines relation similarity directly as divergence between NN-parameterized P(entity-pairs | relation) distributions, provides a sampling approximation, and evaluates the resulting scores via human correlation and task utility. No step reduces a claimed prediction or first-principles result to a fitted parameter or self-citation by construction; the core hypothesis is the object of empirical testing rather than an unverified premise. Code release and direct human judgments supply independent external anchors.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Divergence between conditional probability distributions over entity pairs captures relation similarity
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.