Pith. sign in

REVIEW 4 major objections 5 minor 45 references

Enhancing Relation Extraction via Supervised Rationale Verification and Feedback

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A small rationale supervisor can catch the biased relation predictions of large language models and correct them with re-selected demonstrations, adding up to 10.65 points of micro-F1 over standard in-context learning.

desk verdict The empirical claims hold up, but the paper's central bias-detection mechanism is under-validated: the supervision signal is collected under different conditions than the test-time predictions, and the authors never show the supervisor is actually separating unbiased from genuinely biased rationales at inference. read the letter →

arxiv 2412.07289 v2 pith:SBEPEMIM submitted 2024-12-10 cs.CL cs.AI

Yongqi Li , Xin Miao , Shen Zhou , Mayi Xu , Yuyang Ren , Tieyun Qian This is my paper · ORCID
classification cs.CLcs.AI
keywords relationextractionlargelanguagemodelsrationaleverificationautomatedfeedbackin-contextlearningcausalinterventioncontrastivefew-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Relation extraction is the task of deciding which semantic relation holds between two entities mentioned in a sentence, and large language models often get this wrong because their pre-trained knowledge nudges them toward stereotyped relations. The paper claims that this bias shows up in the model's rationale, the explanation it gives for its prediction, and that the bias can be detected and corrected without retraining the large model. It builds a small rationale supervisor that is contrastively trained on two kinds of rationales: unbiased ones, produced by asking the LLM to explain the golden relation label, and biased ones, produced by feeding demonstrations with deliberately different labels. At inference the supervisor flags rationales that resemble known biased ones and retrieves new in-context demonstrations from the labeled set as feedback, repeating until the rationale looks unbiased. The paper reports average micro-F1 gains of 10.65 points over random-demonstration in-context learning, 6.49 points over semantic-similarity retrieval, and 3.24 points over task-specific retrieval, with the same loop transferring to document-level relation extraction and to event detection.

What carries the argument

The load-bearing object is the rationale supervisor, a small pre-trained encoder trained with a supervised contrastive loss over rationale pairs. The training data come from a causal intervention-and-observation step: label-guided intervention blocks the model's bias path and asks it to explain the golden label, producing the unbiased rationale pool, while diversified intervention feeds demonstrations with different labels to elicit a spread of biased rationales. At inference the supervisor computes $p_b = \max_{r^b \in S_b} \operatorname{sim}(r, r^b) - \max_{r^u \in S_u} \operatorname{sim}(r, r^u)$, where $S_b$ and $S_u$ are the stored biased and unbiased rationales for the predicted relation; $p_b > 0$ means the current rationale is closer to the biased region. Feedback is then generated by taking the $k$ labeled samples whose biased rationales are most similar to the current rationale and using their unbiased versions as new in-context demonstrations, so the correction signal is a demonstration set rather than a textual error message.

What would settle it

A decisive test would be to hold out a set of samples that each have both a label-guided 'unbiased' rationale and a demonstration-guided 'biased' rationale for the same sentence, then measure whether the trained supervisor ranks the biased rationale above the unbiased one with above-chance accuracy; if it does not, the collected rationales are not encoding relation bias in the intended way.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the relation bias of an LLM is visible in the rationale it produces and can be exploited by a lightweight supervised model. The paper induces a label-guided unbiased rationale for each labeled sample by prompting the LLM to explain why the golden relation holds, and it elicits deliberately wrong rationales by feeding demonstrations with different labels, so that both correct explanations and typical error directions are collected. A contrastive loss pulls together unbiased rationales with the same golden label and biased rationales under the same error direction, while pushing apart biased and unbiased rationales for the same sample. At inference, the supervisor scores a new rationale by the difference between its maximum similarity to the stored biased pool and its maximum similarity to the stored unbiased pool for the predicted relation; a positive difference flags the prediction as biased and triggers retrieval of feedback demonstrations from labeled samples in the most similar biased cluster. The LLM is then prompted again with those demonstrations, and the loop repeats until the rationale passes verification, which the paper reports converges in about four feedback rounds.

Load-bearing premise

The framework assumes that asking the LLM to explain the golden relation label produces a genuinely unbiased rationale about the evidence, rather than a post-hoc rationalization of the supplied label; if that assumption fails, the supervisor learns label-conditioned wording instead of true relation bias, and the feedback loop no longer corrects the underlying error.

Editorial extensions

If this is right

  • Because only samples flagged as biased are regenerated, the correction loop is selective; the paper reports that SRVF is the second-most efficient method in its comparison while giving the largest F1 gains.
  • The framework acts as a prompt-level plug-in rather than a fine-tune, and the paper reports gains on Llama-2-chat, Meta-Llama-3-Instruct, and GPT-3.5-turbo, so the correction mechanism is not tied to one model family.
  • The same verification-feedback procedure improves document-level relation extraction on DocRED and Re-DocRED and transfers to event detection on ACE05, indicating the supervision is not limited to sentence-level relation labels.
  • Convergence is fast: one correction round yields up to 11 points of absolute micro-F1, and performance saturates after about four rounds, so the number of extra LLM calls is bounded in practice.
  • Ablation results show that removing the label-guided intervention, the diversified intervention, the contrastive training, the feedback retrieval, or the regeneration step each lowers performance, so every component of the loop contributes to the reported gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit consequence is that the supervisor's representation could serve as a general bias probe, flagging ambiguous or low-confidence predictions for human review even when corrective demonstrations are unavailable.
  • A clean way to isolate the mechanism would be to strip the rationale out of the feedback prompt and keep only the retrieved demonstrations; if the F1 gain persists, the correction is coming from demonstration selection rather than from rationale verification.
  • The supervised signal is generated by the very LLM it corrects, so the quality of the 'unbiased' rationales is bounded by that model's ability to justify a supplied label; a weaker generator should narrow the representation gap between the two pools, and the paper does not directly measure this dependence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SRVF, an automated feedback framework for relation extraction (RE) with large language models. The method trains a BERT-based 'rationale supervisor' by contrastive learning on rationales collected via two prompt interventions: 'unbiased' rationales are induced by supplying the golden relation label to the LLM, and 'biased' rationales are obtained by providing in-context demonstrations whose labels differ from the golden label. At inference, the supervisor scores the LLM's generated rationale; if it is judged biased, the supervisor retrieves feedback demonstrations from the labeled set and the LLM regenerates the prediction, iterating until the rationale is deemed unbiased. Experiments cover three sentence-level RE datasets (SemEval, TACRED, Re-TACRED) with three demonstration-selection backbones and four few-shot settings, five LLM backbones, two document-level datasets, and one event-detection dataset, with ablations and sensitivity analyses.

Significance. If the reported gains are reliable, the paper fills a genuine gap: bringing automated verification-and-feedback correction to relation extraction. The design is thoughtful—using the supervisor both to detect bias and to retrieve corrective demonstrations is a novel combination, and the paper includes a broad experimental matrix, ablations, and sensitivity studies. The code is released. The main empirical claim is large and consistent across settings (e.g., up to 10.65 absolute micro-F1 improvement over random ICL). However, the absence of statistical validation and the lack of direct evidence that the supervisor's bias detection generalizes to test-time rationales are significant caveats.

major comments (4)
  1. [Rationale Verification and Feedback, Eq. (5)] The contrastive training separates rationales produced under two artificial interventions: with the golden label explicitly provided (do(Y=yi)) and with demonstrations whose labels differ from yi (do(I=Ddii)). At inference, the initial rationale is generated by standard ICL without either intervention, so it lies outside both training conditions. The paper never measures whether the supervisor's decision pb>0 actually matches ground-truth bias on real test-time rationales, nor whether the retrieved feedback demonstrations share the true relation with the test example. The end-task F1 gains are consistent with a supervisor that acts as a noisy gate, and the 'w/o FDR' ablation does not isolate verification accuracy. Please add a direct evaluation of the supervisor on held-out test rationales (e.g., rationales from the ICL baseline on the test set), report detection precision/recall, and show that correction success is concentrated in examples flagged as biased rather than in a random subset.
  2. [Table 1 and Experiments] All reported micro-F1 numbers are point estimates; no standard deviations, confidence intervals, or significance tests are given. The abstract claims that the method 'significantly outperforms' existing methods, but without repeated runs (e.g., different random demonstration samples for the Random backbone, or different seeds) it is impossible to assess whether the gains over Self-Consistency and GRACE are statistically reliable. Please provide mean±std over multiple seeds for at least the main comparisons and run a paired significance test (e.g., paired bootstrap).
  3. [Induce Unbiased Rationale and Appendix Quality Analysis] The label-guided procedure guarantees only that the generated rationale is sufficient for the golden label under the LLM (the do(R)->Y consistency check). It does not ensure that the rationale is free of the bias that occurs at test time; a rationale could simply restate or be conditioned on the supplied label and still pass step 2. The quality scores in Table 12 measure plausibility, not absence of bias. This matters because the 'unbiased' rationales are used as positive anchors in Eq. (4)–(5). Please provide a bias-specific analysis, for example by comparing induced 'unbiased' rationales with actual test-time correct rationales, or by showing that the supervisor's Su anchors are not identifiable solely through surface cues such as the presence of the golden label in the prompt.
  4. [Impact of the Hyper-parameter τ (Appendix)] The paper states that the temperature τ is set to 0.2 for all experiments, but the sensitivity analysis in Table 13 shows that τ=0.50 yields a higher average micro-F1 (41.49 vs. 40.68). This inconsistency needs an explanation: if the main results were obtained with a suboptimal τ, the hyperparameter selection procedure should be reported (e.g., chosen on a development set rather than on the test set). Otherwise, please confirm that the main conclusions are unchanged when τ=0.50 is used.
minor comments (5)
  1. [Experiments (typo)] The section header 'Evaluation Protocal' should be 'Evaluation Protocol'.
  2. [Figure 2 and Method] The SCM diagrams are used to motivate do(Y=yi) and do(I=...), but these operations are implemented as prompt modifications, not true causal interventions that cut all back-door paths (e.g., B can still influence R through X→B→R). The causal language should be tempered to avoid overclaiming.
  3. [Prompt examples] The paper uses both 'Reasoning Explanations' (e.g., Table 6) and 'rationale' (in the method text) to refer to the same output. Please unify the terminology.
  4. [Appendix, GRACE adaptation] The description of the GRACE baseline adaptation is terse. Please specify what the discriminator's positive and negative training samples are in the RE setting and how the 'best rationale' is selected during inference, so that the comparison is reproducible.
  5. [Figure 4] The error matrices show only the six non-Other relation types; please clarify whether predictions of 'Other' are omitted or merged, and whether the row sums are consistent with the total number of test examples.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SRVF's F1 gains are measured on held-out test sets, and its supervisor is a supervised calibration model trained on observable rationales, not a self-referential derivation.

full rationale

The paper's central claim is empirical: Table 1 reports micro-F1 scores on held-out SemEval, TACRED, and Re-TACRED test sets, and the reported improvements are measured outcomes of the full inference procedure, not quantities derived from the training assumptions. The rationale supervisor is trained on observable generation outputs—unbiased rationales induced via label-guided intervention and consistency-filtered by do(R)->Y, and biased rationales induced via conflicting demonstrations—under the contrastive loss in Eq. 1. At inference, Eqs. 5-7 use this trained supervisor to verify and retrieve feedback demonstrations. No test label or final F1 value appears among the supervisor's training inputs or as a fitted parameter. The definition of an 'unbiased rationale' (generated under do(Y=yi) and passing do(R)->Y=yi) sets the supervisor's training label, but the paper's reported prediction is the LLM's relation label on test examples, measured against external annotations. The only self-citation (Li et al. 2023b) appears in Related Work as an example of prior step-aware verifiers and negative-rationale collection; it is not invoked to justify SRVF's design, to supply a uniqueness theorem, or to forbid alternatives, so it is not load-bearing. The skeptic's concern that the biased/unbiased contrast may encode intervention artifacts (e.g., golden-label presence versus conflicting demonstrations) is a generalization and validity risk rather than an equation-level circularity: the mapping from Eq. 1's learned representation to Eq. 5's verification score is a model application, not an identity. No specific circular step can be exhibited, so this is a non-finding.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on three domain assumptions: that label-conditioned rationales are genuinely unbiased, that biased rationales cluster by predicted relation in the supervisor's embedding space, and that ICL with retrieved demonstrations corrects the LLM. These assumptions are not derived from first principles but are supported indirectly by the ablations and error-matrix analysis. The only tuned numeric inputs are the contrastive temperature, the number of feedback demonstrations, and the iteration count.

free parameters (3)
  • Contrastive temperature tau = 0.2 (main); 0.5 (best in Appendix Table 13)
    Temperature in Eq. (1) controls focus on hard negative pairs; the appendix explores values and shows sensitivity, with best average at 0.50, but the main experiments use 0.2.
  • Number of feedback demonstrations k = 5 (SemEval), 4 (TACRED, Re-TACRED)
    Set based on experiments on the SemEval test set (Appendix Fig. 9); performance drops when k exceeds 7, so this is a tuned hyperparameter affecting the central results.
  • Feedback iteration count m = 5 (for fairness with baselines)
    The paper sets 5 iterations for SRVF, Self-Consistency, and GRACE; the iteration analysis (Fig. 7) shows performance saturates around 3-4 rounds, so 5 is a safe cap.
assumptions (4)
  • domain assumption Label-conditioned rationales are genuinely unbiased.
    Invoked in 'Induce Unbiased Rationale'; if the LLM rationalizes the given label instead of explaining true evidence, the positive training set is contaminated.
  • domain assumption Biased rationales cluster by predicted relation in embedding space.
    Eqs. (5)-(7) rely on max-similarity to S_b and S_u; without this clustering, verification and retrieval anchors fail.
  • domain assumption The LLM corrects its prediction when given demonstrations from similar bias situations.
    The correction step re-runs ICL with retrieved demonstrations; the paper's ablations support it empirically but no mechanism analysis is given.
  • standard math Dot-product similarity in the supervised embedding space is a valid measure for comparing rationales.
    Eqs. (2) and (5) use dot-product similarity without alternative comparison methods; this is a standard but unproved choice in this context.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Relation Extraction via Supervised Rationale Verification and Feedback." pith.science (2026). https://pith.science/paper/SBEPEMIM

@misc{pith2026241207289,
  author       = {Pith},
  title        = {Pith review of: Enhancing Relation Extraction via Supervised Rationale Verification and Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SBEPEMIM}},
  note         = {Machine review of arXiv:2412.07289}
}
read the original abstract

Despite the rapid progress that existing automated feedback methods have made in correcting the output of large language models (LLMs), these methods cannot be well applied to the relation extraction (RE) task due to their designated feedback objectives and correction manner. To address this problem, we propose a novel automated feedback framework for RE, which presents a rationale supervisor to verify the rationale and provides re-selected demonstrations as feedback to correct the initial prediction. Specifically, we first design a causal intervention and observation method to collect biased/unbiased rationales for contrastive training the rationale supervisor. Then, we present a verification-feedback-correction procedure to iteratively enhance LLMs' capability of handling the RE task. Extensive experiments prove that our proposed framework significantly outperforms existing methods.

Figures

Figures reproduced from arXiv: 2412.07289 by the authors.

Figure 1
Figure 1. Comparison between current automated feedback [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The structure causal model for illustrating the pro [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An example of correcting the initial biased prediction of LLMs via the proposed SRVF framework in the inference [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Efficiency comparison of different methods on the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Illustration of the quality evaluation procedure for [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 8
Figure 8. Figure 8: Analysis of prompt sensitivity. Results (micro-F1 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 7
Figure 7. Figure 7: Results (micro-F1 scores) after [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 9
Figure 9. Figure 9: As shown in [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 9
Figure 9. Figure 9: Impact of the number of feedback demonstrations [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 23 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    AI@Meta. 2024. Llama 3 Model Card

    2024

  4. [4]

    Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; DasSarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; Joseph, N.; Kadavath, S.; Kernion, J.; Conerly, T.; El-Showk, S.; Elhage, N.; Hatfield-Dodds, Z.; Hernandez, D.; Hume, T.; Johnston, S.; Kravec, S.; Lovitt, L.; Nanda, N.; Olsson, C.; Amodei, D.; Brown, T.; Clark, J.; McCandlish, S.; Olah, ...

    arXiv 2022

  5. [5]

    Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020. A Simple Framework for Contrastive Learning of Visual Representations. In III, H. D.; and Singh, A., eds., Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, 1597--1607. PMLR

    2020

  6. [6]

    Chen, X.; Zhang, N.; Xie, X.; Deng, S.; Yao, Y.; Tan, C.; Huang, F.; Si, L.; and Chen, H. 2022. Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In Proceedings of the ACM Web conference 2022, 2778--2788

    work page 2022

  7. [7]

    Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    arXiv 2018

  8. [8]

    Gao, T.; Yao, X.; and Chen, D. 2021. S im CSE : Simple Contrastive Learning of Sentence Embeddings. In Moens, M.-F.; Huang, X.; Specia, L.; and Yih, S. W.-t., eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 6894--6910. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics

    2021

Show all 45 references
  1. [9]

    Golovneva, O.; Chen, M.; Poff, S.; Corredor, M.; Zettlemoyer, L.; Fazel-Zarandi, M.; and Celikyilmaz, A. 2023. ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning. arXiv:2212.07919

    2023 arXiv

  2. [10]

    Gou, Z.; Shao, Z.; Gong, Y.; Shen, Y.; Yang, Y.; Duan, N.; and Chen, W. 2023. CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. arXiv:2305.11738

    2023 arXiv

  3. [11]

    N.; Kozareva, Z.; Nakov, P.; \'O S \'e aghdha, D.; Pad \'o , S.; Pennacchiotti, M.; Romano, L.; and Szpakowicz, S

    Hendrickx, I.; Kim, S. N.; Kozareva, Z.; Nakov, P.; \'O S \'e aghdha, D.; Pad \'o , S.; Pennacchiotti, M.; Romano, L.; and Szpakowicz, S. 2010. S em E val-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals. In Proceedings of the 5th Internati...

    2010

  4. [12]

    Kamoi, R.; Zhang, Y.; Zhang, N.; Han, J.; and Zhang, R. 2024. When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs. arXiv preprint arXiv:2406.01297

    2024 arXiv

  5. [13]

    Khalifa, M.; Logeswaran, L.; Lee, M.; Lee, H.; and Wang, L. 2023. GRACE : Discriminator-Guided Chain-of-Thought Reasoning. In Bouamor, H.; Pino, J.; and Bali, K., eds., Findings of the Association for Computational Linguistics: EMNLP 2023, 15299--15328. Singapore: Association ...

    2023

  6. [14]

    H.; Gonzalez, J

    Kwon, W.; Li, Z.; Zhuang, S.; Sheng, Y.; Zheng, L.; Yu, C. H.; Gonzalez, J. E.; Zhang, H.; and Stoica, I. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles

    2023

  7. [15]

    LDC. 2005. ACE (Automatic Content Extraction) English Annotation Guidelines for Events, 5.4.3 2005.07.01 edition

    2005

  8. [16]

    Li, B.; Fang, G.; Yang, Y.; Wang, Q.; Ye, W.; Zhao, W.; and Zhang, S. 2023 a . Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness. arXiv preprint arXiv:2304.11633

    2023 arXiv

  9. [17]

    Li, G.; Wang, P.; and Ke, W. 2023. Revisiting Large Language Models as Zero-shot Relation Extractors. In Bouamor, H.; Pino, J.; and Bali, K., eds., Findings of the Association for Computational Linguistics: EMNLP 2023, 6877--6892. Singapore: Association for Computational Linguistics

    2023

  10. [18]

    Li, J.; Jia, Z.; and Zheng, Z. 2023. Semi-automatic Data Enhancement for Document-Level Relation Extraction with Distant Supervision from Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 5495--5505

    2023

  11. [19]

    Li, Y.; Lin, Z.; Zhang, S.; Fu, Q.; Chen, B.; Lou, J.-G.; and Chen, W. 2023 b . Making Language Models Better Reasoners with Step-Aware Verifier. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Proceedings of the 61st Annual Meeting of the Association for Computational ...

    2023

  12. [20]

    Liu, Y.; Iter, D.; Xu, Y.; Wang, S.; Xu, R.; and Zhu, C. 2023. G -Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2511--2522. Sin...

    2023

  13. [21]

    Ma, X.; Li, J.; and Zhang, M. 2023. Chain of Thought with Explicit Evidence Reasoning for Few-shot Relation Extraction. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  14. [22]

    C.; and Sun, A

    Ma, Y.; Cao, Y.; Hong, Y. C.; and Sun, A. 2023. Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples! In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  15. [23]

    Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651

    2023 arXiv

  16. [24]

    Nathani, D.; Wang, D.; Pan, L.; and Wang, W. 2023. MAF : Multi-Aspect Feedback for Improving Reasoning in Large Language Models. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 6591--6616. Si...

    2023

  17. [25]

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. Training language model...

    2022 arXiv

  18. [26]

    Ozyurt, Y.; Feuerriegel, S.; and Zhang, C. 2024. Document-Level In-Context Few-Shot Relation Extraction via Pre-Trained Language Models

    2024

  19. [27]

    Pan, L.; Saxon, M.; Xu, W.; Nathani, D.; Wang, X.; and Wang, W. Y. 2023. Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies. arXiv:2308.03188

    2023 arXiv

  20. [28]

    Pang, C.; Cao, Y.; Ding, Q.; and Luo, P. 2023. Guideline Learning for In-Context Information Extraction. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  21. [29]

    Paul, D.; Ismayilzada, M.; Peyrard, M.; Borges, B.; Bosselut, A.; West, R.; and Faltings, B. 2023. REFINER: Reasoning Feedback on Intermediate Representations. arXiv:2304.01904

    2023 arXiv

  22. [30]

    Pearl, J.; et al. 2000. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress, 19(2): 3

    2000

  23. [31]

    A.; and P \'o czos, B

    Stoica, G.; Platanios, E. A.; and P \'o czos, B. 2021. Re-tacred: Addressing shortcomings of the tacred dataset. In Proceedings of the AAAI conference on artificial intelligence, volume 35, 13843--13850

    2021

  24. [32]

    Sun, Q.; Huang, K.; Yang, X.; Tong, R.; Zhang, K.; and Poria, S. 2024. Consistency guided knowledge retrieval and denoising in llms for zero-shot document-level relation triplet extraction. In Proceedings of the ACM on Web Conference 2024, 4407--4416

    2024

  25. [33]

    T.; and Aljunied, S

    Tan, Q.; Xu, L.; Bing, L.; Ng, H. T.; and Aljunied, S. M. 2022. Revisiting DocRED-Addressing the False Negative Problem in Relation Extraction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 8472--8487

    2022

  26. [34]

    Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  27. [35]

    Wadhwa, S.; Amir, S.; and Wallace, B. 2023. Revisiting Relation Extraction in the era of Large Language Models. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

    2023

  28. [36]

    Wan, Z.; Cheng, F.; Mao, Z.; Liu, Q.; Song, H.; Li, J.; and Kurohashi, S. 2023. GPT - RE : In-context Learning for Relation Extraction using Large Language Models. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural ...

    2023

  29. [37]

    Wang, F.; Mo, W.; Wang, Y.; Zhou, W.; and Chen, M. 2023 a . A Causal View of Entity Bias in (Large) Language Models. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  30. [38]

    V.; Chi, E

    Wang, X.; Wei, J.; Schuurmans, D.; Le, Q. V.; Chi, E. H.; Narang, S.; Chowdhery, A.; and Zhou, D. 2023 b . Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations

    2023

  31. [39]

    Wei, X.; Cui, X.; Cheng, N.; Wang, X.; Zhang, X.; Huang, S.; Xie, P.; Xu, J.; Chen, Y.; Zhang, M.; et al. 2023. Zero-shot information extraction via chatting with chatgpt. arXiv preprint arXiv:2302.10205

    2023 arXiv

  32. [40]

    Wu, S.; and He, Y. 2019. Enriching pre-trained language model with entity information for relation classification. In Proceedings of the 28th ACM international conference on information and knowledge management, 2361--2364

    2019

  33. [41]

    Xu, D.; Chen, W.; Peng, W.; Zhang, C.; Xu, T.; Zhao, X.; Wu, X.; Zheng, Y.; and Chen, E. 2023. Large Language Models for Generative Information Extraction: A Survey. arXiv:2312.17617

    2023 arXiv

  34. [42]

    Yao, Y.; Ye, D.; Li, P.; Han, X.; Lin, Y.; Liu, Z.; Liu, Z.; Huang, L.; Zhou, J.; and Sun, M. 2019. DocRED: A Large-Scale Document-Level Relation Extraction Dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 764--777

    2019

  35. [43]

    J.; and Su, Y

    Zhang, K.; Guti \'e rrez, B. J.; and Su, Y. 2023. Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors. In Findings of ACL

    2023

  36. [44]

    Y.; and Li, L

    Zhang, K.; Wang, D.; Xia, J.; Wang, W. Y.; and Li, L. 2023. ALGO: Synthesizing Algorithmic Programs with Generated Oracle Verifiers. arXiv preprint arXiv:2305.14591

    2023 arXiv

  37. [45]

    Zhang, Y.; Zhong, V.; Chen, D.; Angeli, G.; and Manning, C. D. 2017. Position-aware Attention and Supervised Data Improve Slot Filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 35--45. Copenhagen, Denmark: Association for Compu...

    2017

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.