Pith. sign in

REVIEW 5 major objections 6 minor 32 references

Reinforcement Learning-based N-ary Cross-Sentence Relation Extraction

T0 review · 5 major / 6 minor · reviewed 2026-08-27 · deepseek-v4-flash

Pith's one-line read This paper claims that a two-level reinforcement-learning sentence selector can clean noisy distant-supervision labels for n-ary cross-sentence relation extraction, raising accuracy on both consecutive and non-consecutive sentence groups.

desk verdict Novel RL-based denoiser for n-ary cross-sentence relation extraction, but the theoretical proof is invalid and the WikiText test split leaks; the empirical claims are not yet solid. read the letter →

arxiv 2009.12683 v1 pith:HC5ACWHS submitted 2020-09-26 cs.LG cs.CLstat.ML

classification cs.LGcs.CLstat.ML
keywords n-arycross-sentencerelationextractiondistantsupervisionreinforcementlearningsentencedistributionestimatornoisereductionattentionmechanismPCNNknowledgebase
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper takes on n-ary cross-sentence relation extraction—pulling out facts that involve more than two entities and are stated across several sentences—under distant supervision, where labels come automatically from a knowledge base and are often wrong. It claims that the strong assumption used by earlier systems (only consecutive sentences mentioning the full entity set describe the relation) can be relaxed to a weaker one that also admits non-consecutive sentences, provided a trained 'sentence distribution estimator' filters out mislabeled sentences. The estimator is a two-level reinforcement-learning agent that selects which sentences truly express the relation, and it is trained alternately with a new 'universal relation extractor' that combines attention, piecewise CNN, and LSTM encoding. The paper's experimental claim is that this interplay yields higher accuracy than strong baselines: 88.6 vs 88.0 on PubMed ternary binary-class, 89.2 vs 88.5 on binary class with cross-sentence setting, and 66.4% vs 52.2% on a new WikiText dataset of non-consecutive sentences. A sympathetic reader would care because it suggests a way to automatically clean noisy distant-supervision data and to handle facts that are spread out in text, not just adjacent sentences.

What carries the argument

The key mechanism is the sentence distribution estimator (SDE), a two-level reinforcement-learning agent that plays the role of a data cleaner: it takes sentence groups labeled by distant supervision and outputs the subgroup that actually expresses the relation. At the main-sentence level, a sigmoid policy scores the primary sentence from its PCNN-derived encoding; at the supplementary level, a second policy scores each candidate sentence using three indicators—exponential distance from the main sentence, entity variety relative to the fact's entity set, and cosine similarity to the main sentence—with Monte Carlo rollouts providing intermediate rewards. The relation extractor (RE) is the reward function and the downstream classifier: it encodes each sentence with Bi-LSTM, applies piecewise CNN, then combines multi-head self-attention and a gated nonlinear transformation to classify the relation; the two models are trained alternately. The paper also replaces the strong consecutive-sentence distant supervision assumption with a weaker one (a sentence containing at least one main entity or two supplementary entities is labeled with the relation), which is what brings non-consecutive sentences into the training set.

What would settle it

Take the WikiText data and artificially flip labels for all sentences mentioning a particular entity pair (correlated noise) while keeping the correct labels unchanged; if the SDE-RE training still assigns selection probability P(1|xp) overwhelmingly to true sentences rather than the systematically flipped ones, the noise model holds; if the selector instead learns to favor the flipped sentences, the zero-mean uncorrelated-noise assumption is violated.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that iteratively training a sentence selector and a relation extractor makes the selector concentrate probability on correctly labeled sentences: after training, Pθ,γ(1|xp) ≫ Pθ,γ(1|xn), where xp are true positive instances and xn are false positives introduced by distant supervision. The selector, called the sentence distribution estimator (SDE), is a two-level reinforcement learning agent: a main-sentence policy scores the primary sentence, and a supplementary-sentence policy decides, via three hand-designed indicators (position distance, entity variety, and cosine similarity to the main sentence), which additional sentences carry the relation. The relation extractor (RE) is a hybrid of Bi-LSTM encoding, piecewise CNN, multi-head self-attention, and a gated fusion of attention with a nonlinear transformation, designed to handle both consecutive and non-consecutive sentence groups. The proof sketch in the appendix argues that with zero-mean, uncorrelated noise, the RE converges to the true distribution and the SDE then assigns higher selection probability to clean data; the experiments on PubMed and the new WikiText dataset are put forward as evidence that this convergence holds in practice and improves accuracy.

Load-bearing premise

The entire convergence guarantee rests on assuming that the labeling noise is zero-mean and uncorrelated, so mislabeled sentences are just the true distribution plus random noise; if distant-supervision errors are systematic and correlated, as they often are, the guarantee that the selector will concentrate on clean data may collapse.

Editorial extensions

If this is right

  • If the SDE-RE training loop genuinely concentrates selection probability on correctly labeled sentences, distant-supervision noise can be reduced without manual annotation, so larger automatically labeled corpora become usable for multi-sentence relation extraction.
  • The weaker distant supervision assumption implies that relation extraction can be applied to facts stated in non-consecutive sentences, expanding the coverage of knowledge-base construction beyond adjacent sentence windows.
  • The reported accuracy gaps—e.g., 66.4% vs 52.2% on WikiText—suggest that both the selector and the universal relation extractor contribute; the ablations with random selection and without indicators quantify their roles.
  • On the PubMed dataset, the method sets new state-of-the-art numbers on most ternary and binary relation extraction settings, including multi-class tasks where the proposed model stays roughly 1.8% above the strongest baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The editor's inference: the proof of convergence relies on the noise being zero-mean and uncorrelated, so the guarantee is fragile exactly in the regime that motivated the work—distantly labeled sentences from a knowledge base often fail systematically (e.g., co-reference errors), so the practical benefit may depend on how close real noise is to that idealized model.
  • Pith editor inference: the SDE could be reused as a general denoising module for any distant-supervision pipeline, not only n-ary cross-sentence extraction; the two-level structure with entity-variety transition might transfer to bag-level classifiers in traditional relation extraction.
  • If the WikiText dataset is made public, it provides a new benchmark where the strong-consecutiveness assumption is violated, which could set a realistic test for future cross-sentence and document-level relation extraction models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Request a human review

A listed scientist reviews the paper for a fee and the review publishes here regardless of verdict. See the reviewers or get listed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper addresses n-ary cross-sentence relation extraction under distant supervision. It proposes a two-level reinforcement-learning "sentence distribution estimator" (SDE) to select likely correctly labeled sentence groups, a weaker distant supervision assumption that also labels non-consecutive sentences, and a hybrid relation extraction (RE) model combining Bi-LSTM, PCNN, self-attention, soft attention, and a gate layer. The central claim, formalized as Proposition 1, is that iteratively training SDE and RE makes the selection probability of correctly labeled sentences much larger than that of incorrectly labeled ones (Eq. 12). The authors report accuracy gains on PubMed (e.g., 88.6 vs 88.0 for ternary binary-class) and on a newly created WikiText dataset (66.4 vs 52.2 for the strongest baseline), and they provide a manual inspection of SDE-selected sentences as supporting evidence.

Significance. If the claims were fully established, the work would make a useful contribution: it relaxes the consecutive-sentence assumption underlying much cross-sentence relation extraction, introduces RL-based instance selection to the n-ary setting, and contributes a new non-consecutive benchmark. The RE model alone is competitive with recent PubMed baselines, and the manual inspection of SDE probabilities is a sensible sanity check. However, the theoretical guarantee in Proposition 1 is not currently supported, and the WikiText experimental design creates train/test overlap; the practical significance of the method therefore remains uncertain until these points are resolved.

major comments (5)
  1. [Appendix, Lemma 1 (Eq. 14-16)] The derivation of max R = max R_p + min R_n is invalid. The objective after subtracting the average reward is a single expectation under the shared policy π_{θ,γ}; the sets τ_p and τ_n depend on r and on π, so the two sums cannot be optimized independently. For a two-trajectory example with rewards 1 and 0 and r = 0.5, the true objective max_π [0.5π - 0.5(1-π)] equals 0.5, whereas the claimed RHS is 0.5 + (-0.5) = 0. Proposition 1 is therefore not proved by the argument given.
  2. [Appendix, Theoretical Analysis] The proof is explicitly incomplete: the footnote states "The full proof is a work in progress." In addition, Lemma 2 models noise as additive feature noise ξ on p_X(x), whereas distant supervision errors are label errors; the Taylor-expansion argument in Eqs. 19-20 does not establish that incorrectly labeled sentences receive lower selection probability after iterative training. If Proposition 1 is a load-bearing claim, a rigorous proof is needed; otherwise the proposition should be reclassified as a heuristic motivation.
  3. [Experiments, WikiText dataset] The test/train split is designed so that "the instance that has sentences in the test dataset also has sentences in the training dataset." This creates overlap between training and test facts, which can inflate test accuracy through leakage of relation-relevant information. The authors should report results on a split that is disjoint at the fact (or at least entity) level, or justify in detail why the overlap does not affect the conclusions.
  4. [Table 3, PubMed results] The differences between RE and RE with SDE are small (e.g., 88.6±0.1 vs 88.0±0.3; 90.1±0.2 vs 89.1±0.2) and no significance tests are reported. The conclusion states that the results are "significantly better" than baselines, but this is not supported by any statistical test. Add paired significance tests (e.g., McNemar or paired bootstrap over the five folds) or temper the strength of the claims.
  5. [Evaluation on SDE model] The manual inspection of 100 sentences is described only briefly: there is no information on how the sentences were selected, whether the annotator was blind to the SDE probabilities, whether any second annotator was used, or how the "strong positive correlation" in Fig. 2 was quantified. Because this is direct empirical evidence for Proposition 1, the evaluation should be made more rigorous and the correlation should be reported numerically with a significance test.
minor comments (6)
  1. [Abstract] The sentence "This estimator selects correctly labeled sentences to alleviate the effect of noisy data is a two-level agent reinforcement learning model" is ungrammatical and should be split or rewritten.
  2. [Algorithm 1] Step 11 says "do step 18~19" but steps 18-19 are defined later in the algorithm; this makes the procedure hard to follow and should be renumbered.
  3. [Eq. 1] The dimension notation for W^T is inconsistent: if W^T ∈ R^{ds×1}, then W^T s_i is not defined for s_i of dimension ds; use W ∈ R^{ds×1} and W^T s_i or correct the dimensions.
  4. [Table 3] The table headings "Binary class" and "Multi-class" with subheadings "Ternary Binary" are confusing; define "Single" vs "Cross" and clarify the relation to the task setup.
  5. [WikiText dataset] The text says "standard derivation" in the description of the five splits; this should be "standard deviation."
  6. [Eq. 8] Initializing h_0, c_0 from N(0,1) is unusual and is not discussed; if this is intentional, provide a justification, otherwise use the standard zero initialization.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is a substantive convergence claim, and the paper's flaws are correctness risks, not definitional reductions.

full rationale

Proposition 1 and its appendix proof do not reduce to their own inputs by construction. The target quantity Pθ,γ(1|xp) is defined through a formal data distribution pX with additive zero-mean noise ξ, while the SDE reward in Eq. 4 is the RE model's cross-entropy; the proof attempts to connect high reward to xp through Lemma 2 rather than simply defining 'correctly labeled' as 'high reward.' The SDE is also independently checked against 100 manually inspected test sentences in Fig. 2, so the empirical claim is not fitted to itself. The appendix Lemma 1 contains a genuine mathematical error: maximizing the sum of positive-reward and negative-reward terms is not equivalent to independently maximizing the first and minimizing the second, because the same policy appears in both sums. Lemma 2 also misapplies Bishop (1995) by modeling distant-supervision label noise as additive input noise. These are correctness risks that undermine the proof, but they are not circularity: the paper does not invoke an unverified self-citation chain, and no equation or fitted parameter is renamed as a prediction. The WikiText test split, which ensures test instances share facts with training instances, is a data-leakage concern rather than a circular reduction. Because no load-bearing step can be exhibited as equivalent to its own input by definition or by a self-citation chain, the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a novel labeling assumption, a hand-designed two-level RL structure, and an unproven theoretical analysis of noise. The free parameters are small design choices; the axioms include the weak supervision rule and the noise model, the latter being essential to the proof but not independently verified.

free parameters (3)
  • Monte Carlo sample times N = 5
    Set to 5 in Experimental Settings; used as the number of rollouts for the intermediate reward in the supplementary policy.
  • Supplementary policy sample times = 3
    Set to 3 in Experimental Settings; number of samples for the transition simulation in the supplementary agent.
  • RE/SDE training ratio = 20:8
    In WikiText experiments, SDE is trained 8 times every 20 RE training iterations; chosen by hand and may affect convergence.
assumptions (4)
  • domain assumption A sentence with at least one main entity or two supplementary entities is annotated with the relation of those entities.
    This weaker distant supervision rule, introduced in the Introduction, is used to label all WikiText data; the main results depend on this labeling.
  • domain assumption Each sentence group has exactly one main sentence and several supplementary sentences.
    Assumed in the Sentence Distribution Estimator; the main sentence contains the main entities and the supplementary sentences provide qualifier information.
  • ad hoc to paper Distant supervision noise is zero-mean and uncorrelated; incorrectly labeled sentences are drawn from pX(x)+xi.
    Stated in the appendix and required for Lemma 2, which is the basis of the claim that RE converges to the clean distribution.
  • ad hoc to paper max R = max Rp + min Rn for trajectories with positive and negative reward advantages.
    Used in Lemma 1 to argue that SDE assigns high probability to high-reward data; the decomposition is asserted without a rigorous derivation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforcement Learning-based N-ary Cross-Sentence Relation Extraction." pith.science (2026). https://pith.science/paper/HC5ACWHS

@misc{pith2026200912683,
  author       = {Pith},
  title        = {Pith review of: Reinforcement Learning-based N-ary Cross-Sentence Relation Extraction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HC5ACWHS}},
  note         = {Machine review of arXiv:2009.12683}
}
read the original abstract

The models of n-ary cross sentence relation extraction based on distant supervision assume that consecutive sentences mentioning n entities describe the relation of these n entities. However, on one hand, this assumption introduces noisy labeled data and harms the models' performance. On the other hand, some non-consecutive sentences also describe one relation and these sentences cannot be labeled under this assumption. In this paper, we relax this strong assumption by a weaker distant supervision assumption to address the second issue and propose a novel sentence distribution estimator model to address the first problem. This estimator selects correctly labeled sentences to alleviate the effect of noisy data is a two-level agent reinforcement learning model. In addition, a novel universal relation extractor with a hybrid approach of attention mechanism and PCNN is proposed such that it can be deployed in any tasks, including consecutive and nonconsecutive sentences. Experiments demonstrate that the proposed model can reduce the impact of noisy data and achieve better performance on general n-ary cross sentence relation extraction task compared to baseline models.

Figures

Figures reproduced from arXiv: 2009.12683 by the authors.

Figure 1
Figure 1. The flowchart of the proposed model. On the right side, the sentence distribution estimator consists of main policy [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. The value of weights on each training iteration [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 2
Figure 2. The probability distribution of sentences [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 26 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter doi edition editor eid howpublished institution isbn issn journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bishop, C. M. 1995. Training with noise is equivalent to Tikhonov regularization. Neural computation 7(1): 108--116

  4. [4]

    Feng, J.; Huang, M.; Zhao, L.; Yang, Y.; and Zhu, X. 2018. Reinforcement learning for relation classification from noisy data. In Thirty-Second AAAI Conference on Artificial Intelligence

  5. [5]

    Graves, A.; and Schmidhuber, J. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural networks 18(5-6): 602--610

  6. [6]

    Gupta, P.; Rajaram, S.; Sch \"u tze, H.; and Runkler, T. 2019. Neural relation extraction within and across sentence boundaries. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 6513--6520

  7. [7]

    Hochreiter, S.; and Schmidhuber, J. 1997. Long short-term memory. Neural computation 9(8): 1735--1780

  8. [8]

    Hu, L.; Zhang, L.; Shi, C.; Nie, L.; Guan, W.; and Yang, C. 2019. Improving Distantly-Supervised Relation Extraction with Joint Label Embedding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3812--3820

Show all 32 references
  1. [9]

    Ji, G.; Liu, K.; He, S.; Zhao, J.; et al. 2017. Distant supervision for relation extraction with sentence-level attention and entity descriptions. In AAAI, volume 3060

  2. [10]

    Jia, R.; Wong, C.; and Poon, H. 2019. Document-Level N -ary Relation Extraction with Multiscale Representation Learning. arXiv preprint arXiv:1904.02347

  3. [11]

    Li, H.; Krause, S.; Xu, F.; Moro, A.; Uszkoreit, H.; and Navigli, R. 2015. Improvement of n-ary Relation Extraction by Adding Lexical Semantics to Distant-Supervision Rule Learning. In ICAART (2), 317--324

  4. [12]

    Mandya, A.; Bollegala, D.; Coenen, F.; and Atkinson, K. 2018. Combining Long Short Term Memory and Convolutional Neural Network for Cross-Sentence n-ary Relation Extraction. In Automated Knowledge Base Construction (AKBC)

  5. [13]

    Mesquita, F.; Schmidek, J.; and Barbosa, D. 2013. Effectiveness and efficiency of open relation extraction. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 447--457

  6. [14]

    S.; and Dean, J

    Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, 3111--3119

  7. [15]

    Mintz, M.; Bills, S.; Snow, R.; and Jurafsky, D. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of th...

  8. [16]

    Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, 8026--8037

  9. [17]

    Peng, N.; Poon, H.; Quirk, C.; Toutanova, K.; and Yih, W.-t. 2017. Cross-sentence n-ary relation extraction with graph lstms. Transactions of the Association for Computational Linguistics 5: 101--115

  10. [18]

    Qin, P.; Xu, W.; and Wang, W. Y. 2018. Robust distant supervision relation extraction via deep reinforcement learning. arXiv preprint arXiv:1805.09927

  11. [19]

    Quirk, C.; and Poon, H. 2016. Distant supervision for relation extraction beyond the sentence boundary. arXiv preprint arXiv:1609.04873

  12. [20]

    J.; Guez, A.; Sifre, L.; Van Den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; et al

    Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.; Van Den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; et al. 2016. Mastering the game of Go with deep neural networks and tree search. nature 529(7587): 484--489

  13. [21]

    Song, L.; Zhang, Y.; Wang, Z.; and Gildea, D. 2018. N-ary relation extraction using graph state lstm. arXiv preprint arXiv:1808.09101

  14. [22]

    S.; McAllester, D

    Sutton, R. S.; McAllester, D. A.; Singh, S. P.; and Mansour, Y. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, 1057--1063

  15. [23]

    Takamatsu, S.; Sato, I.; and Nakagawa, H. 2012. Reducing wrong labels in distant supervision for relation extraction. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 721--729

  16. [24]

    C.; and Varoquaux, G

    Van Der Walt, S.; Colbert, S. C.; and Varoquaux, G. 2011. The NumPy array: a structure for efficient numerical computation. Computing in Science & Engineering 13(2): 22

  17. [25]

    N.; Kaiser, .; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. In Advances in neural information processing systems, 5998--6008

  18. [26]

    Vrande c i \'c , D.; and Kr \"o tzsch, M. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM 57(10): 78--85

  19. [27]

    Yang, K.; He, L.; Dai, X.; Huang, S.; and Chen, J. 2019. Exploiting Noisy Data in Distant Supervision Relation Classification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol...

  20. [28]

    Ye, Z.-X.; and Ling, Z.-H. 2019. Distant supervision relation extraction with intra-bag and inter-bag attentions. arXiv preprint arXiv:1904.00143

  21. [29]

    Zeng, D.; Liu, K.; Chen, Y.; and Zhao, J. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceedings of the 2015 conference on empirical methods in natural language processing, 1753--1762

  22. [30]

    Zhang, Y.; Guo, Z.; and Lu, W. 2019. Attention guided graph convolutional networks for relation extraction. arXiv preprint arXiv:1906.07510

  23. [31]

    Zhang, Y.; Qi, P.; and Manning, C. D. 2018. Graph convolution over pruned dependency trees improves relation extraction. arXiv preprint arXiv:1809.10185

  24. [32]

    Zhao, D.; Wang, J.; Zhang, Y.; Wang, X.; Lin, H.; and Yang, Z. 2020. Incorporating representation learning and multihead attention to improve biomedical cross-sentence n-ary relation extraction. BMC bioinformatics 21(1): 1--17

Pith tools

Reviewed August 27, 2026 · model on record in the stance chip above.