REVIEW 3 major objections 5 minor 36 references
Towards a More Generalized Approach in Open Relation Extraction
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that a single two-phase framework can handle OpenRE when unlabeled data mixes known and novel relations, and reports consistent wins over prior baselines on three benchmarks.
desk verdict A sensible, incremental OpenRE method whose headline gains come with an oracle cluster count; worth a serious referee if robustness to unknown K is required. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Semantic Autoencoder (SAE), an encoder-decoder that projects relation representations into a latent space whose axes are one-hot vectors of the known relations; instances that score low on similarity to every known one-hot vector are treated as outliers. Those outliers are clustered by a Gaussian Mixture Model (GMM) with the number of components set to $|C_{\mathrm{novel}}|$, and the high-confidence cluster members become weak labels for novel relations. The second phase combines three losses — supervised cross-entropy $L_c$ for classification, a triplet margin loss $L_{lm}$ on labeled positive pairs, and a clustering exemplar loss $L_e$ that pulls each instance toward multi-granularity centroids — trained with a continual-learning schedule so novel relations are absorbed after known-relation warm-up. At inference, known-relation predictions are taken directly while novel relations are assigned by Faiss K-Means clustering of the learned representations.
What would settle it
Run MixORE on the same FewRel, TACRED, and Re-TACRED splits with $|C_{\mathrm{novel}}|$ set to values above and below the true value of 6 (for example 3 and 12) and compare known-relation F1 and novel-clustering B3 F1; the claim that the framework generalizes to arbitrary mixed data would be weakened if performance degrades sharply with a misspecified count, since real deployments do not know the count in advance.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a single model can learn known and novel relations together from a mixed unlabeled pool without assuming novel relations are rare or long-tailed. The authors claim that MixORE consistently outperforms competitive baselines in known relation classification across all three benchmarks, and achieves the strongest overall novel-relation clustering performance, with the largest gains on TACRED where it beats the second-best clustering baseline by 3.73% B3 F1, 2.59% V-measure F1, and 4.72% ARI. The mechanism is a division of labor: a Semantic Autoencoder maps known relations onto one-hot semantic vectors so novel instances fall out as low-similarity outliers, GMM turns those outliers into weak-label clusters, and the OW-SS joint learning phase uses cross-entropy, triplet margin, and clustering exemplar losses to sharpen known-relation features while keeping novel-relation features diverse.
Load-bearing premise
The method must be told how many novel relations are hiding in the unlabeled data before it runs, because that number sets both the Gaussian mixture component count and the final clustering count; the authors state in the Limitations section that the framework cannot determine this number automatically, so the claimed generality holds only when the cluster count is known or correctly guessed.
Editorial extensions
If this is right
- OpenRE systems no longer need the unlabeled pool to be pre-split into known and novel buckets; a single pass over mixed data can serve both goals.
- Weak labels produced by unsupervised clustering, even at moderate purity (0.556–0.636 in the paper's analysis), are sufficient to train a model that classifies known relations well.
- Continually training on weakly labeled novel relations after a known-relation warm-up substantially improves novel clustering; the ablation attributes drops of 8.34, 6.48, and 13.01 points in B3 F1, V-measure F1, and ARI to removing that component.
- Combining classification and clustering losses lets one encoder balance compact known-relation features with diverse novel-relation features, narrowing the known-versus-novel performance gap observed in prior work.
Reading between the lines
- Beyond the paper: a natural next step is to replace the fixed $|C_{\mathrm{novel}}|$ input with an adaptive cluster-count estimator; if that works, the approach would apply to fully open deployments where the number of emerging relations is unknown.
- Beyond the paper: the same semantic-autoencoder outlier-then-cluster pipeline could be reused for other open-world tasks such as novel event type discovery or open intent detection, wherever labeled classes are few and unlabeled data is mixed.
- Beyond the paper: because the model tolerates noisy weak labels, one could test whether larger unlabeled pools with lower outlier purity still yield gains, which would tell practitioners how much detection precision is actually required.
- Beyond the paper: modeling hierarchical relations explicitly, which the authors list as a limitation, might be integrated as a structured prior in the one-hot latent space instead of treating all relations as independent.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes MixORE, a two-phase framework for open relation extraction (OpenRE) where unlabeled data contains a mixture of known and novel relations. Phase 1 (Novel Relation Detection) projects relation representations through a Semantic Autoencoder into a one-hot latent space, flags the 5% of instances with the lowest mapping scores as outliers, clusters them with a Gaussian Mixture Model using the true |Cnovel| as the component count, and selects high-confidence instances as weak labels. Phase 2 (OW-SS Joint Learning) continually trains a BERT relation encoder with a cross-entropy classification loss, a triplet margin loss on labeled data, and a clustering exemplar loss, then clusters novel-relation representations with Faiss K-Means at inference using the same |Cnovel| as K. Experiments on FewRel, TACRED, and Re-TACRED compare against six baselines including KNoRD, reporting consistent gains in known-relation F1 and competitive novel-relation clustering (Table 2), with an ablation study and an analysis of weak-label quality.
Significance. The paper addresses a genuine limitation of prior OpenRE work by relaxing the long-tail assumption on novel relations and treating the unlabeled pool as an arbitrary mixture of known and novel instances. If the empirical results hold under sensitivity analysis, the two-phase design is a useful contribution to practical OpenRE, and the release of source code, the ablation study, and the weak-label quality analysis are strengths that support reproducibility. The main significance, however, is conditional on the method's behavior when the number of novel relations is not supplied in advance, which the paper itself acknowledges as an open limitation.
major comments (3)
- [§4.2–§4.4, Limitations] The number of novel relations |Cnovel| is used as (i) the GMM component count in Eq. (3), (ii) the classifier output size in Eq. (4), and (iii) the number of clusters K in the Faiss K-Means in §4.4. The Limitations section states that the method 'cannot automatically determine the number of novel relations' in the unlabeled data. Consequently, the reported gains in Table 2 are conditional on knowing a test-set property that is unknown in the generalized OpenRE setting the paper advocates. This is especially consequential because Table 4 shows that weak-label detection identifies only 3–5 of the 6 novel relations on the three datasets, so the strong final clustering numbers cannot be attributed mainly to reliable weak labels; they depend on the oracle K in §4.4. Please add a sensitivity analysis with misspecified K (e.g., K in {2,...,10}) to show graceful degradation, or propose a principled criterion for estimating K from the outlier pool.
- [Table 2, §5.4] The paper reports the average of two random runs with no variance, confidence intervals, or significance tests. The headline claim that MixORE surpasses KNoRD by 5.90%, 3.14%, and 4.87% in known-relation F1 (FewRel, TACRED, Re-TACRED) is presented without any measure of uncertainty, making it unclear whether these differences exceed run-to-run noise. Please report results over multiple seeds with standard deviations and, ideally, paired significance tests against KNoRD.
- [§4.2, §5.6] The 5% outlier threshold is justified only by an analogy to the conventional significance level in hypothesis testing, which is not a substantive justification. The outlier pool directly determines which instances enter GMM and become weak labels, and if the true proportion of novel instances deviates from 5%, the threshold will either discard true novel instances or admit many known instances. Please report the actual novel-instance proportions in the three datasets and perform a sensitivity analysis over the threshold value.
minor comments (5)
- [§4.3, after Eq. (6)] The sentence 'τ is a is a temperature hyperparameter (Wu et al., 2018)' contains a duplicated article; it should read 'τ is a temperature hyperparameter.'
- [§4.3, Eq. (4)] The notation Dc is used both for the number of instances in the current epoch and as the summation bound in Eq. (4); please clarify whether Dc includes both labeled and weakly labeled instances and how it varies across epochs.
- [Table 2, Table 3] The column headers for B3 precision/recall/F1 are split across lines in a way that makes the tables difficult to parse; reformatting the headers would improve readability.
- [§5.2, Baselines] The descriptions of the modified baselines MatchPrompt′, TABs′, HiURE∗, and AugURE∗ would benefit from additional detail on the Hungarian-alignment step, particularly how the number of clusters is chosen for the alignment and whether the known-relation labels influence that choice.
- [§1, Introduction] The introduction should state more explicitly that the setting of mixed known and novel instances is shared with KNoRD and that the claimed contribution is the relaxation of the long-tail assumption, not the setting itself; this would help position the work relative to Hogan et al. (2023).
Circularity Check
No derivation-level circularity; oracle-K input is an admitted limitation, not a forced result.
full rationale
The claimed results are not derived from the model's own inputs by construction: SAE projection, GMM weak-label generation, the OW-SS losses, and Faiss K-Means are standard components evaluated against external ground-truth labels, so the Table 2 F1/B3/V/ARI numbers could have been different and are not forced by the equations. The one near-circular concern is that |Cnovel| is used as the GMM component count in Eq. (3), as the classifier output dimension in Eq. (4), and as the final Faiss K-Means K in Sec. 4.4, while Sec. 3 defines the goal as 'identifying |Cnovel| novel relation clusters.' The Limitations section explicitly states the method 'cannot automatically determine the number of novel relations' and 'relies on a pre-defined number of clusters.' This is a test-set-information leak and a genuine generality limitation, but it does not make the reported clustering quality or known-relation F1 true by definition; knowing K still leaves the assignment problem nontrivial. The self-citation to Wang et al. (2023) for the augmentation technique is not load-bearing. Therefore no significant circularity is present; the score is 2 to reflect the mild oracle-K caveat.
Assumptions & free parameters
free parameters (5)
- Outlier proportion (5%) =
0.05
- Number of novel relations |Cnovel| =
6 per dataset
- Triplet margin gamma =
0.75
- Temperature tau =
0.02
- SAE weighting lambda =
100
assumptions (6)
- ad hoc to paper Known instances cluster near their one-hot vectors in the SAE latent space.
- domain assumption Novel instances appear as outliers in the SAE mapping.
- domain assumption Novel instances of the same relation cluster in the latent space.
- ad hoc to paper The 5% lowest mapping scores approximate the novel instances.
- domain assumption Relations are independent (no hierarchy or dependencies).
- standard math Bartels-Stewart solver, K-Means, and GMM are valid for their uses.
Cite this review
Pith. "Pith review of Towards a More Generalized Approach in Open Relation Extraction." pith.science (2026). https://pith.science/paper/YLB4HNML
@misc{pith2026250522801,
author = {Pith},
title = {Pith review of: Towards a More Generalized Approach in Open Relation Extraction},
year = {2026},
howpublished = {\url{https://pith.science/paper/YLB4HNML}},
note = {Machine review of arXiv:2505.22801}
}
read the original abstract
Open Relation Extraction (OpenRE) seeks to identify and extract novel relational facts between named entities from unlabeled data without pre-defined relation schemas. Traditional OpenRE methods typically assume that the unlabeled data consists solely of novel relations or is pre-divided into known and novel instances. However, in real-world scenarios, novel relations are arbitrarily distributed. In this paper, we propose a generalized OpenRE setting that considers unlabeled data as a mixture of both known and novel instances. To address this, we propose MixORE, a two-phase framework that integrates relation classification and clustering to jointly learn known and novel relations. Experiments on three benchmark datasets demonstrate that MixORE consistently outperforms competitive baselines in known relation classification and novel relation clustering. Our findings contribute to the advancement of generalized OpenRE research and real-world applications.
Figures
Reference graph
Works this paper leans on
-
[1]
Amit Bagga and Breck Baldwin. 1998. Entity-based cross-document coreferencing using the vector space model. In Annual Meeting of the Association for Computational Linguistics
work page 1998
-
[2]
Richard H. Bartels and G. W. Stewart. 1972. Solution of the matrix equation ax+xb=c [F4] (algorithm 432). Commun. ACM , 15(9):820--826
work page 1972
-
[3]
Kaidi Cao, Maria Brbic, and Jure Leskovec. 2022. Open-world semi-supervised learning. In ICLR . OpenReview.net
work page 2022
-
[4]
Li Cui, Deqing Yang, Jiaxin Yu, Chengwei Hu, Jiayang Cheng, Jingjie Yi, and Yanghua Xiao. 2021. Refining sample embeddings with relation prototypes to enhance continual relation extraction. In ACL/IJCNLP (1) , pages 232--243. Association for Computational Linguistics
work page 2021
-
[5]
Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, ...
work page 2019
-
[6]
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In EMNLP , pages 4803--4809. Association for Computational Linguistics
work page 2018
-
[7]
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhanced bert with disentangled attention. In ICLR . OpenReview.net
work page 2021
-
[8]
William Hogan, Jiacheng Li, and Jingbo Shang. 2023. Open-world semi-supervised generalized relation discovery aligned in a real-world setting. In EMNLP , pages 14227--14242. Association for Computational Linguistics
work page 2023
Show all 36 references
-
[9]
Hubert and Phipps Arabie
Lawrence J. Hubert and Phipps Arabie. 1985. Comparing partitions. Journal of Classification, 2:193--218
1985
-
[10]
Jeff Johnson, Matthijs Douze, and Herv \' e J \' e gou. 2021. Billion-scale similarity search with gpus. IEEE Trans. Big Data , 7(3):535--547
2021
-
[11]
Elyor Kodirov, Tao Xiang, and Shaogang Gong. 2017. Semantic autoencoder for zero-shot learning. In CVPR , pages 4447--4456. IEEE Computer Society
2017
-
[12]
Harold W. Kuhn. 2010. The hungarian method for the assignment problem. In 50 Years of Integer Programming, pages 29--47. Springer
2010
-
[13]
Sha Li, Heng Ji, and Jiawei Han. 2022. Open relation and event type discovery with type abstraction. In EMNLP , pages 6864--6877. Association for Computational Linguistics
2022
-
[14]
Fangchao Liu, Lingyong Yan, Hongyu Lin, Xianpei Han, and Le Sun. 2021. Element intervention for open relation extraction. In ACL/IJCNLP (1) , pages 4683--4693. Association for Computational Linguistics
2021
-
[15]
Shuliang Liu, Xuming Hu, Chenwei Zhang, Shu'ang Li, Lijie Wen, and Philip S. Yu. 2022. Hiure: Hierarchical exemplar contrastive learning for unsupervised relation extraction. In NAACL-HLT , pages 5970--5980. Association for Computational Linguistics
2022
-
[16]
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net
2019
-
[17]
Ramshaw, Richard M
Scott Miller, Michael Crystal, Heidi Fox, Lance A. Ramshaw, Richard M. Schwartz, Rebecca Stone, and Ralph M. Weischedel. 1998. BBN: description of the SIFT system as used for MUC-7 . In Seventh Message Understanding Conference: Proceedings of a Conference Held in Fairfax, Virg...
1998
-
[18]
Fabian Pedregosa, Ga \" e l Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake VanderPlas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duc...
2011
-
[19]
Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen - tau Yih. 2017. Cross-sentence n-ary relation extraction with graph lstms. Trans. Assoc. Comput. Linguistics, 5:101--115
2017
-
[20]
Andrew Rosenberg and Julia Hirschberg. 2007. V-measure: A conditional entropy-based external cluster evaluation measure. In EMNLP-CoNLL 2007, Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learnin...
2007
-
[21]
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-...
2019
-
[22]
George Stoica, Emmanouil Antonios Platanios, and Barnab'as P'oczos. 2021. https://api.semanticscholar.org/CorpusID:233296843 Re-tacred: Addressing shortcomings of the tacred dataset . In AAAI Conference on Artificial Intelligence
2021
-
[23]
Somin Wadhwa, Silvio Amir, and Byron C. Wallace. 2023. Revisiting relation extraction in the era of large language models. In ACL (1) , pages 15566--15589. Association for Computational Linguistics
2023
-
[24]
Jiaxin Wang, Lingling Zhang, Jun Liu, Xi Liang, Yujie Zhong, and Yaqiang Wu. 2022. Matchprompt: Prompt-based open relation extraction with semantic consistency guided clustering. In EMNLP , pages 7875--7888. Association for Computational Linguistics
2022
-
[25]
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. 2024. A comprehensive survey of continual learning: Theory, method and application. IEEE Trans. Pattern Anal. Mach. Intell. , 46(8):5362--5383
2024
-
[26]
Qing Wang, Kang Zhou, Qiao Qiao, Yuepei Li, and Qi Li. 2023. Improving unsupervised relation extraction by augmenting diverse sentence pairs. In EMNLP , pages 12136--12147. Association for Computational Linguistics
2023
-
[27]
Ting Wu, Jingyi Liu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Enhancing contrastive learning with noise-guided attack: Towards continual relation extraction in the wild. In ACL (1) , pages 2227--2239. Association for Computational Linguistics
2024
-
[28]
Yu, and Dahua Lin
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. 2018. Unsupervised feature learning via non-parametric instance discrimination. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 3733--3742. ...
2018
-
[29]
Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin, and Leyu Lin. 2020. Denoising relation extraction from document-level distant supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Onl...
2020
-
[30]
Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2002. Kernel methods for relation extraction. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing, EMNLP 2002, Philadelphia, PA, USA, July 6-7, 2002 , pages 71--78
2002
-
[31]
Xinwei Zhang, Jianwen Jiang, Yutong Feng, Zhi - Fan Wu, Xibin Zhao, Hai Wan, Mingqian Tang, Rong Jin, and Yue Gao. 2022. Grow and merge: A unified framework for continuous categories discovery. In NeurIPS
2022
-
[32]
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In EMNLP , pages 35--45. Association for Computational Linguistics
2017
-
[33]
Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June ...
2021
-
[34]
Kang Zhou, Qiao Qiao, Yuepei Li, and Qi Li. 2023. Improving distantly supervised relation extraction by natural language inference. In AAAI , pages 14047--14055. AAAI Press
2023
-
[35]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[36]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.