REVIEW 5 major objections 6 minor 32 references
Reinforcement Learning-based N-ary Cross-Sentence Relation Extraction
T0 review · 5 major / 6 minor · reviewed 2026-08-27 · deepseek-v4-flash
Pith's one-line read This paper claims that a two-level reinforcement-learning sentence selector can clean noisy distant-supervision labels for n-ary cross-sentence relation extraction, raising accuracy on both consecutive and non-consecutive sentence groups.
desk verdict Novel RL-based denoiser for n-ary cross-sentence relation extraction, but the theoretical proof is invalid and the WikiText test split leaks; the empirical claims are not yet solid. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key mechanism is the sentence distribution estimator (SDE), a two-level reinforcement-learning agent that plays the role of a data cleaner: it takes sentence groups labeled by distant supervision and outputs the subgroup that actually expresses the relation. At the main-sentence level, a sigmoid policy scores the primary sentence from its PCNN-derived encoding; at the supplementary level, a second policy scores each candidate sentence using three indicators—exponential distance from the main sentence, entity variety relative to the fact's entity set, and cosine similarity to the main sentence—with Monte Carlo rollouts providing intermediate rewards. The relation extractor (RE) is the reward function and the downstream classifier: it encodes each sentence with Bi-LSTM, applies piecewise CNN, then combines multi-head self-attention and a gated nonlinear transformation to classify the relation; the two models are trained alternately. The paper also replaces the strong consecutive-sentence distant supervision assumption with a weaker one (a sentence containing at least one main entity or two supplementary entities is labeled with the relation), which is what brings non-consecutive sentences into the training set.
What would settle it
Take the WikiText data and artificially flip labels for all sentences mentioning a particular entity pair (correlated noise) while keeping the correct labels unchanged; if the SDE-RE training still assigns selection probability P(1|xp) overwhelmingly to true sentences rather than the systematically flipped ones, the noise model holds; if the selector instead learns to favor the flipped sentences, the zero-mean uncorrelated-noise assumption is violated.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that iteratively training a sentence selector and a relation extractor makes the selector concentrate probability on correctly labeled sentences: after training, Pθ,γ(1|xp) ≫ Pθ,γ(1|xn), where xp are true positive instances and xn are false positives introduced by distant supervision. The selector, called the sentence distribution estimator (SDE), is a two-level reinforcement learning agent: a main-sentence policy scores the primary sentence, and a supplementary-sentence policy decides, via three hand-designed indicators (position distance, entity variety, and cosine similarity to the main sentence), which additional sentences carry the relation. The relation extractor (RE) is a hybrid of Bi-LSTM encoding, piecewise CNN, multi-head self-attention, and a gated fusion of attention with a nonlinear transformation, designed to handle both consecutive and non-consecutive sentence groups. The proof sketch in the appendix argues that with zero-mean, uncorrelated noise, the RE converges to the true distribution and the SDE then assigns higher selection probability to clean data; the experiments on PubMed and the new WikiText dataset are put forward as evidence that this convergence holds in practice and improves accuracy.
Load-bearing premise
The entire convergence guarantee rests on assuming that the labeling noise is zero-mean and uncorrelated, so mislabeled sentences are just the true distribution plus random noise; if distant-supervision errors are systematic and correlated, as they often are, the guarantee that the selector will concentrate on clean data may collapse.
Editorial extensions
If this is right
- If the SDE-RE training loop genuinely concentrates selection probability on correctly labeled sentences, distant-supervision noise can be reduced without manual annotation, so larger automatically labeled corpora become usable for multi-sentence relation extraction.
- The weaker distant supervision assumption implies that relation extraction can be applied to facts stated in non-consecutive sentences, expanding the coverage of knowledge-base construction beyond adjacent sentence windows.
- The reported accuracy gaps—e.g., 66.4% vs 52.2% on WikiText—suggest that both the selector and the universal relation extractor contribute; the ablations with random selection and without indicators quantify their roles.
- On the PubMed dataset, the method sets new state-of-the-art numbers on most ternary and binary relation extraction settings, including multi-class tasks where the proposed model stays roughly 1.8% above the strongest baseline.
Reading between the lines
- The editor's inference: the proof of convergence relies on the noise being zero-mean and uncorrelated, so the guarantee is fragile exactly in the regime that motivated the work—distantly labeled sentences from a knowledge base often fail systematically (e.g., co-reference errors), so the practical benefit may depend on how close real noise is to that idealized model.
- Pith editor inference: the SDE could be reused as a general denoising module for any distant-supervision pipeline, not only n-ary cross-sentence extraction; the two-level structure with entity-variety transition might transfer to bag-level classifiers in traditional relation extraction.
- If the WikiText dataset is made public, it provides a new benchmark where the strong-consecutiveness assumption is violated, which could set a realistic test for future cross-sentence and document-level relation extraction models.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses n-ary cross-sentence relation extraction under distant supervision. It proposes a two-level reinforcement-learning "sentence distribution estimator" (SDE) to select likely correctly labeled sentence groups, a weaker distant supervision assumption that also labels non-consecutive sentences, and a hybrid relation extraction (RE) model combining Bi-LSTM, PCNN, self-attention, soft attention, and a gate layer. The central claim, formalized as Proposition 1, is that iteratively training SDE and RE makes the selection probability of correctly labeled sentences much larger than that of incorrectly labeled ones (Eq. 12). The authors report accuracy gains on PubMed (e.g., 88.6 vs 88.0 for ternary binary-class) and on a newly created WikiText dataset (66.4 vs 52.2 for the strongest baseline), and they provide a manual inspection of SDE-selected sentences as supporting evidence.
Significance. If the claims were fully established, the work would make a useful contribution: it relaxes the consecutive-sentence assumption underlying much cross-sentence relation extraction, introduces RL-based instance selection to the n-ary setting, and contributes a new non-consecutive benchmark. The RE model alone is competitive with recent PubMed baselines, and the manual inspection of SDE probabilities is a sensible sanity check. However, the theoretical guarantee in Proposition 1 is not currently supported, and the WikiText experimental design creates train/test overlap; the practical significance of the method therefore remains uncertain until these points are resolved.
major comments (5)
- [Appendix, Lemma 1 (Eq. 14-16)] The derivation of max R = max R_p + min R_n is invalid. The objective after subtracting the average reward is a single expectation under the shared policy π_{θ,γ}; the sets τ_p and τ_n depend on r and on π, so the two sums cannot be optimized independently. For a two-trajectory example with rewards 1 and 0 and r = 0.5, the true objective max_π [0.5π - 0.5(1-π)] equals 0.5, whereas the claimed RHS is 0.5 + (-0.5) = 0. Proposition 1 is therefore not proved by the argument given.
- [Appendix, Theoretical Analysis] The proof is explicitly incomplete: the footnote states "The full proof is a work in progress." In addition, Lemma 2 models noise as additive feature noise ξ on p_X(x), whereas distant supervision errors are label errors; the Taylor-expansion argument in Eqs. 19-20 does not establish that incorrectly labeled sentences receive lower selection probability after iterative training. If Proposition 1 is a load-bearing claim, a rigorous proof is needed; otherwise the proposition should be reclassified as a heuristic motivation.
- [Experiments, WikiText dataset] The test/train split is designed so that "the instance that has sentences in the test dataset also has sentences in the training dataset." This creates overlap between training and test facts, which can inflate test accuracy through leakage of relation-relevant information. The authors should report results on a split that is disjoint at the fact (or at least entity) level, or justify in detail why the overlap does not affect the conclusions.
- [Table 3, PubMed results] The differences between RE and RE with SDE are small (e.g., 88.6±0.1 vs 88.0±0.3; 90.1±0.2 vs 89.1±0.2) and no significance tests are reported. The conclusion states that the results are "significantly better" than baselines, but this is not supported by any statistical test. Add paired significance tests (e.g., McNemar or paired bootstrap over the five folds) or temper the strength of the claims.
- [Evaluation on SDE model] The manual inspection of 100 sentences is described only briefly: there is no information on how the sentences were selected, whether the annotator was blind to the SDE probabilities, whether any second annotator was used, or how the "strong positive correlation" in Fig. 2 was quantified. Because this is direct empirical evidence for Proposition 1, the evaluation should be made more rigorous and the correlation should be reported numerically with a significance test.
minor comments (6)
- [Abstract] The sentence "This estimator selects correctly labeled sentences to alleviate the effect of noisy data is a two-level agent reinforcement learning model" is ungrammatical and should be split or rewritten.
- [Algorithm 1] Step 11 says "do step 18~19" but steps 18-19 are defined later in the algorithm; this makes the procedure hard to follow and should be renumbered.
- [Eq. 1] The dimension notation for W^T is inconsistent: if W^T ∈ R^{ds×1}, then W^T s_i is not defined for s_i of dimension ds; use W ∈ R^{ds×1} and W^T s_i or correct the dimensions.
- [Table 3] The table headings "Binary class" and "Multi-class" with subheadings "Ternary Binary" are confusing; define "Single" vs "Cross" and clarify the relation to the task setup.
- [WikiText dataset] The text says "standard derivation" in the description of the five splits; this should be "standard deviation."
- [Eq. 8] Initializing h_0, c_0 from N(0,1) is unusual and is not discussed; if this is intentional, provide a justification, otherwise use the standard zero initialization.
Circularity Check
No significant circularity: the central claim is a substantive convergence claim, and the paper's flaws are correctness risks, not definitional reductions.
full rationale
Proposition 1 and its appendix proof do not reduce to their own inputs by construction. The target quantity Pθ,γ(1|xp) is defined through a formal data distribution pX with additive zero-mean noise ξ, while the SDE reward in Eq. 4 is the RE model's cross-entropy; the proof attempts to connect high reward to xp through Lemma 2 rather than simply defining 'correctly labeled' as 'high reward.' The SDE is also independently checked against 100 manually inspected test sentences in Fig. 2, so the empirical claim is not fitted to itself. The appendix Lemma 1 contains a genuine mathematical error: maximizing the sum of positive-reward and negative-reward terms is not equivalent to independently maximizing the first and minimizing the second, because the same policy appears in both sums. Lemma 2 also misapplies Bishop (1995) by modeling distant-supervision label noise as additive input noise. These are correctness risks that undermine the proof, but they are not circularity: the paper does not invoke an unverified self-citation chain, and no equation or fitted parameter is renamed as a prediction. The WikiText test split, which ensures test instances share facts with training instances, is a data-leakage concern rather than a circular reduction. Because no load-bearing step can be exhibited as equivalent to its own input by definition or by a self-citation chain, the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (3)
- Monte Carlo sample times N =
5
- Supplementary policy sample times =
3
- RE/SDE training ratio =
20:8
assumptions (4)
- domain assumption A sentence with at least one main entity or two supplementary entities is annotated with the relation of those entities.
- domain assumption Each sentence group has exactly one main sentence and several supplementary sentences.
- ad hoc to paper Distant supervision noise is zero-mean and uncorrelated; incorrectly labeled sentences are drawn from pX(x)+xi.
- ad hoc to paper max R = max Rp + min Rn for trajectories with positive and negative reward advantages.
Cite this review
Pith. "Pith review of Reinforcement Learning-based N-ary Cross-Sentence Relation Extraction." pith.science (2026). https://pith.science/paper/HC5ACWHS
@misc{pith2026200912683,
author = {Pith},
title = {Pith review of: Reinforcement Learning-based N-ary Cross-Sentence Relation Extraction},
year = {2026},
howpublished = {\url{https://pith.science/paper/HC5ACWHS}},
note = {Machine review of arXiv:2009.12683}
}
read the original abstract
The models of n-ary cross sentence relation extraction based on distant supervision assume that consecutive sentences mentioning n entities describe the relation of these n entities. However, on one hand, this assumption introduces noisy labeled data and harms the models' performance. On the other hand, some non-consecutive sentences also describe one relation and these sentences cannot be labeled under this assumption. In this paper, we relax this strong assumption by a weaker distant supervision assumption to address the second issue and propose a novel sentence distribution estimator model to address the first problem. This estimator selects correctly labeled sentences to alleviate the effect of noisy data is a two-level agent reinforcement learning model. In addition, a novel universal relation extractor with a hybrid approach of attention mechanism and PCNN is proposed such that it can be deployed in any tasks, including consecutive and nonconsecutive sentences. Experiments demonstrate that the proposed model can reduce the impact of noisy data and achieve better performance on general n-ary cross sentence relation extraction task compared to baseline models.
Figures
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address author booktitle chapter doi edition editor eid howpublished institution isbn issn journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Bishop, C. M. 1995. Training with noise is equivalent to Tikhonov regularization. Neural computation 7(1): 108--116
work page 1995
-
[4]
Feng, J.; Huang, M.; Zhao, L.; Yang, Y.; and Zhu, X. 2018. Reinforcement learning for relation classification from noisy data. In Thirty-Second AAAI Conference on Artificial Intelligence
work page 2018
-
[5]
Graves, A.; and Schmidhuber, J. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural networks 18(5-6): 602--610
work page 2005
-
[6]
Gupta, P.; Rajaram, S.; Sch \"u tze, H.; and Runkler, T. 2019. Neural relation extraction within and across sentence boundaries. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 6513--6520
work page 2019
-
[7]
Hochreiter, S.; and Schmidhuber, J. 1997. Long short-term memory. Neural computation 9(8): 1735--1780
work page 1997
-
[8]
Hu, L.; Zhang, L.; Shi, C.; Nie, L.; Guan, W.; and Yang, C. 2019. Improving Distantly-Supervised Relation Extraction with Joint Label Embedding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3812--3820
work page 2019
Show all 32 references
-
[9]
Ji, G.; Liu, K.; He, S.; Zhao, J.; et al. 2017. Distant supervision for relation extraction with sentence-level attention and entity descriptions. In AAAI, volume 3060
2017
-
[10]
Jia, R.; Wong, C.; and Poon, H. 2019. Document-Level N -ary Relation Extraction with Multiscale Representation Learning. arXiv preprint arXiv:1904.02347
2019 arXiv
-
[11]
Li, H.; Krause, S.; Xu, F.; Moro, A.; Uszkoreit, H.; and Navigli, R. 2015. Improvement of n-ary Relation Extraction by Adding Lexical Semantics to Distant-Supervision Rule Learning. In ICAART (2), 317--324
2015
-
[12]
Mandya, A.; Bollegala, D.; Coenen, F.; and Atkinson, K. 2018. Combining Long Short Term Memory and Convolutional Neural Network for Cross-Sentence n-ary Relation Extraction. In Automated Knowledge Base Construction (AKBC)
2018
-
[13]
Mesquita, F.; Schmidek, J.; and Barbosa, D. 2013. Effectiveness and efficiency of open relation extraction. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 447--457
2013
-
[14]
S.; and Dean, J
Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, 3111--3119
2013
-
[15]
Mintz, M.; Bills, S.; Snow, R.; and Jurafsky, D. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of th...
2009
-
[16]
Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, 8026--8037
2019
-
[17]
Peng, N.; Poon, H.; Quirk, C.; Toutanova, K.; and Yih, W.-t. 2017. Cross-sentence n-ary relation extraction with graph lstms. Transactions of the Association for Computational Linguistics 5: 101--115
2017
-
[18]
Qin, P.; Xu, W.; and Wang, W. Y. 2018. Robust distant supervision relation extraction via deep reinforcement learning. arXiv preprint arXiv:1805.09927
2018 arXiv
-
[19]
Quirk, C.; and Poon, H. 2016. Distant supervision for relation extraction beyond the sentence boundary. arXiv preprint arXiv:1609.04873
2016 arXiv
-
[20]
J.; Guez, A.; Sifre, L.; Van Den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; et al
Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.; Van Den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; et al. 2016. Mastering the game of Go with deep neural networks and tree search. nature 529(7587): 484--489
2016
-
[21]
Song, L.; Zhang, Y.; Wang, Z.; and Gildea, D. 2018. N-ary relation extraction using graph state lstm. arXiv preprint arXiv:1808.09101
2018 arXiv
-
[22]
S.; McAllester, D
Sutton, R. S.; McAllester, D. A.; Singh, S. P.; and Mansour, Y. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, 1057--1063
2000
-
[23]
Takamatsu, S.; Sato, I.; and Nakagawa, H. 2012. Reducing wrong labels in distant supervision for relation extraction. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 721--729
2012
-
[24]
C.; and Varoquaux, G
Van Der Walt, S.; Colbert, S. C.; and Varoquaux, G. 2011. The NumPy array: a structure for efficient numerical computation. Computing in Science & Engineering 13(2): 22
2011
-
[25]
N.; Kaiser, .; and Polosukhin, I
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. In Advances in neural information processing systems, 5998--6008
2017
-
[26]
Vrande c i \'c , D.; and Kr \"o tzsch, M. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM 57(10): 78--85
2014
-
[27]
Yang, K.; He, L.; Dai, X.; Huang, S.; and Chen, J. 2019. Exploiting Noisy Data in Distant Supervision Relation Classification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol...
2019
-
[28]
Ye, Z.-X.; and Ling, Z.-H. 2019. Distant supervision relation extraction with intra-bag and inter-bag attentions. arXiv preprint arXiv:1904.00143
2019 arXiv
-
[29]
Zeng, D.; Liu, K.; Chen, Y.; and Zhao, J. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceedings of the 2015 conference on empirical methods in natural language processing, 1753--1762
2015
-
[30]
Zhang, Y.; Guo, Z.; and Lu, W. 2019. Attention guided graph convolutional networks for relation extraction. arXiv preprint arXiv:1906.07510
2019 arXiv
-
[31]
Zhang, Y.; Qi, P.; and Manning, C. D. 2018. Graph convolution over pruned dependency trees improves relation extraction. arXiv preprint arXiv:1809.10185
2018 arXiv
-
[32]
Zhao, D.; Wang, J.; Zhang, Y.; Wang, X.; Lin, H.; and Yang, Z. 2020. Incorporating representation learning and multihead attention to improve biomedical cross-sentence n-ary relation extraction. BMC bioinformatics 21(1): 1--17
2020
Reviewed August 27, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.