{"id":"ea7d2dd6-c38a-47d3-aa2e-b416534a7ab5","arxiv_id":"2411.18157","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey of 2020-2023 ACL-family papers on relation extraction finds BERT-based models dominate, while large language models like T5 show promise mainly in few-shot settings.","lead":"This paper reviews 65 recent papers on relation extraction, a task that identifies connections between entities in text, and finds that BERT-based models dominate current methods. A generalist might read it to understand the current toolbox and benchmark datasets for extracting structured facts from unstructured text.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Table 7 contradicts its claim that T5 excels in few-shot RE: T5 ranks 4th on FewRel behind BERT and GPT-2/BART systems, so the abstract's central claim overreaches.","rationale":"The reader's formal weakest_assumption is corpus representativeness, but their rationale also flags the few-shot overstatement. I agree that corpus scope is a genuine limitation; however, the more decisive and immediately checkable problem is internal: the paper's own Tables 5-7 contradict the abstract's few-shot claim. T5 appears in Table 7 only as OffMML-G(+negs) with F1 61.3, placing fourth behind BERT-based and GPT-2/BART systems, and no T5 model leads any of the three benchmark tables. The paper even acknowledges in Section 7.4 that LLMs do not play a central role in state-of-the-art RE. This internal inconsistency means the few-shot conclusion is not merely vulnerable to selection bias; it is unsupported by the evidence the authors themselves present. The BERT-dominance portion of the central claim is much better supported, so a conditional accept remains appropriate with mandatory revision of the few-shot claim and related abstract wording. I therefore do not move the reader's verdict, but I would emphasize the internal Table 7 contradiction rather than corpus representativeness as the primary reason for requiring correction.","tokens_in":45833,"tokens_out":4101,"duration_ms":38179,"concrete_test":"Re-extract Section 6 from the manuscript and independently sort Tables 5-7 by F1 descending, labeling each row's model family as BERT/RoBERTa, T5, GPT-family, or GLM. Then verify whether any T5-based system ranks first in any table and compute T5's best rank in Table 7. If Table 7 ranks OffMML-G(+negs) fourth behind ESC-ZSRE, RelationPrompt, and IDL, the abstract's sentence about T5 excelling in few-shot RE is unsupported. To rule out a simple corpus artifact, additionally query the FewRel leaderboard and EMNLP/ICLR/NeurIPS 2022-2023 papers for zero-shot and few-shot RE; if no T5-based method surpasses BERT-based ESC-ZSRE or GPT-2-based RelationPrompt, the claim should be removed from the abstract and Section 6.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim has two conjuncts: BERT-based models dominate RE, and LLMs like T5 excel in few-shot RE, especially on unseen relations. The first conjunct is supported by Tables 5-7, where BERT/RoBERTa occupy 11 of 15 top-five slots. The second is not supported by the paper's own evidence. In Table 7 (FewRel zero-shot RE), the T5-based method OffMML-G(+negs) ranks fourth with F1 61.3, behind ESC-ZSRE (BERT, 81.68), RelationPrompt (GPT2+BART, 79.96), and IDL (BERT, 62.61). In Table 5 (TACRED), the T5-based GEN-PT scores 75.3, below DeepStruct (GLM, 76.8) and two RoBERTa models. In Table 6 (DocRED), no LLM appears at all; all five entries are RoBERTa. Section 6 states that LLMs 'excel in few-shot learning scenarios' and the abstract repeats this as a key finding, but the tabulated evidence shows no T5 system achieving a top-one result on any of the three benchmarks, and on the few-shot benchmark T5 is not even in the top two. Section 7.4 then concedes that LLMs 'do not play a central role in advancing state-of-the-art performance,' which is inconsistent with the abstract's stronger few-shot claim. Since the few-shot excellence claim is part of the central conclusion and is contradicted by the paper's own tables, the finding as stated is unreliable even before considering corpus selection. The dominance conclusion may survive, but the LLM few-shot claim should be corrected or removed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript surveys relation extraction (RE) techniques based on language models, focusing on papers from ACL, NAACL, AACL, and EACL between 2020 and 2023. The authors report screening 81 conference papers, including 65 after exclusion criteria, and analyzing 56 datasets; the abstract and conclusion state that 137 papers were analyzed, combining dataset papers and research papers. The survey categorizes RE work by task (sentence-level, document-level, few-shot, distant supervision, open RE, multilingual/multimodal), tabulates the most frequent models and datasets, and compares state-of-the-art results on TACRED, DocRED, and FewRel. The central claims are that BERT-based methods dominate state-of-the-art RE results and that LLMs such as T5 show promise in few-shot RE, especially for unseen relations.","tokens_in":46162,"tokens_out":1389,"duration_ms":14011,"significance":"If the survey's conclusions are correct, the paper provides a useful consolidation of RE research at ACL venues over a focused four-year window, with detailed tables of models, datasets, and benchmark results. The compilation of 65 papers and 56 datasets is a potentially valuable reference for practitioners, and the observation that BERT/RoBERTa occupy most top benchmark slots (Tables 5-7) is a falsifiable, clearly stated claim. The normalization-by-release-year analysis in Section 7.2 is a thoughtful attempt to address the confounding factor that BERT is older than T5 or GPT. However, because the survey's corpus is restricted to four ACL venues and the inclusion criteria are subjective, the generalizability of the dominance claim to the broader RE field is not established. The paper's most serious weakness is that its own tables contradict the few-shot LLM claim in the abstract, as detailed below.","major_comments":[{"comment":"The abstract's claim that LLMs like T5 'excel in few-shot relation extraction scenarios where they excel in identifying previously unseen relations' is not supported by the paper's own evidence. In Table 7, the T5-based OffMML-G(+negs) ranks fourth on FewRel zero-shot RE with F1=61.3, behind BERT-based ESC-ZSRE (81.68), GPT2+BART-based RelationPrompt (79.96), and BERT-based IDL (62.61). Section 7.4 also concedes that LLMs 'do not play a central role in advancing state-of-the-art performance in extraction tasks.' The few-shot excellence claim should be removed or substantially weakened to match the data.","section":"Abstract and Section 7.4"},{"comment":"The paper reports inconsistent corpus sizes: the abstract and conclusion state '137 papers,' while Section 2 reports 'we examined 81 papers' and 'the final set comprised 65 papers.' Table 1 clarifies that 137 = 56 dataset papers + 81 research papers, but this arithmetic is not explained in the text and the abstract's 'analyzing 137 papers' conflates dataset papers with the survey's core analysis. The methodology should state explicitly that 81 research papers were screened, 65 were included, and 56 datasets were catalogued separately.","section":"Section 2, Table 1, and Conclusion"},{"comment":"The inclusion criteria are applied subjectively ('papers that presented novel approaches or significant advancements,' exclusion of temporal RE, NER-focused, non-LM, and non-adopted encoding methods), and the survey does not validate the representativeness of the resulting corpus against any external source. Since the central dominance conclusion in Section 6 is based only on the top-five entries per benchmark from this filtered set, the conclusion may reflect venue and selection bias. I recommend adding a sensitivity analysis or at least an explicit discussion of how the exclusion criteria could affect the BERT-dominance and LLM findings.","section":"Section 2 and Section 6"},{"comment":"The comparison in Section 6 is based on only three benchmarks (TACRED, DocRED, FewRel) and only the top five systems per benchmark, with no pooling of runs or statistical tests. The statement that 'BERT and RoBERTa collectively represent a substantial 75% of the primary outcomes' is arithmetically unclear: in the 15 total slots across Tables 5-7, BERT/RoBERTa appear in 11 slots (73%), and the 'primary outcomes' are not defined. The dominance conclusion would be more robust if the authors reported how many of the 65 included papers used BERT/RoBERTa versus T5/GPT in the actual benchmark evaluations, rather than only the top-five lists.","section":"Tables 5-7 and Section 6"}],"minor_comments":[{"comment":"The sentence 'To narrow down the papers, we searched for pieces that contained the phrase “Relation Extraction” in either the title or abstract' should specify whether the search was case-sensitive and whether 'relation extraction' as a phrase or as separate words was used, since this affects reproducibility.","section":"Section 2"},{"comment":"There is a typo: 'the entity pair‘ ‘Steve Jobs”' should read 'the entity pair “Steve Jobs” and “Apple Inc.”'.","section":"Section 3.1"},{"comment":"The paragraph on [163] states 'the model attained state-of-the-art results on benchmark datasets, namely TACRED and SemEval 2010,' but the reference list shows this is an AACL 2022 paper; the acronym 'FPC' in the same paragraph is not defined at first use.","section":"Section 5.1"},{"comment":"The sentence 'Large language models like RoBERTa and BERT are widely used' is misleading because RoBERTa and BERT are not typically classified as large language models in the sense of GPT or T5; consider revising to 'encoder-only language models like RoBERTa and BERT.'","section":"Section 6"},{"comment":"In Table 8, the count for BERT base is listed as 45, but the reference list contains 46 entries; please double-check the count and the alignment of references.","section":"Section 7.2"},{"comment":"The phrase 'they excel in identifying previously unseen relations' in the conclusion repeats the unsupported few-shot claim; see the major comment above.","section":"Section 8"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The survey's main result — that BERT and RoBERTa dominate relation extraction state of the art in ACL-family venues from 2020 to 2023 — actually holds up. The tables are the real value here: they give a structured map of models, tasks, and datasets that someone entering the field or picking baselines would find handy. Extending Bassignana and Plank's earlier survey to the 2020-2023 window and adding NAACL, AACL, and EACL is a legitimate incremental update, and the paper does a decent job of categorizing a messy literature.\n\nThe soft spots are real but not fatal. First, the count inconsistency: the abstract says 137 papers, the methodology says 81 examined and 65 included, and Table 1 reports all three numbers without reconciliation. That's sloppy and needs fixing. Second, the inclusion criteria are subjective — dropping temporal RE, NER-focused work, and non-adopted encodings — and the paper never validates the sample against an external corpus. So the BERT-dominance claim is credible but not bulletproof; it's a claim about this curated sample, not necessarily the whole field.\n\nThe bigger problem is the abstract's second conjunct: that LLMs like T5 \"excel\" in few-shot RE, especially on unseen relations. The paper's own Table 7 shows the T5-based OffMML-G(+negs) at fourth place on FewRel, behind two BERT systems and a GPT-2/BART system. T5 never tops any of the three benchmark tables. And Section 7.4 walks it back, admitting LLMs \"do not play a central role in advancing state-of-the-art performance.\" The abstract overreaches, and the conclusion should be rewritten to match the evidence.\n\nThe internal contradiction is the main thing I'd want a referee to push on. It's not fatal to the survey's overall utility — the BERT dominance finding is well-supported — but it means the central claim as stated is unreliable. With a corrected abstract, reconciled counts, and a brief justification of the selection criteria, this would be a solid reference survey. As it stands, it's a useful draft that needs revision, not a desk reject.\n\nWho gets value from this: graduate students and researchers wanting a quick inventory of RE methods and benchmarks over the period. I'd bring it to a reading group and I'd probably cite it once it's cleaned up. My recommendation: send it to peer review, with the expectation of major revision on the counting and the few-shot claim.","headline":"Useful survey of 2020-2023 ACL-family RE work, but the abstract's few-shot T5 claim is contradicted by the paper's own Table 7 and should be cut or heavily qualified before publication.","tokens_in":46689,"tokens_out":1434,"would_cite":true,"duration_ms":14731,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BERT-based models dominate state-of-the-art relation extraction in the recent ACL-conference literature, while large language models like T5 show promise mainly in few-shot scenarios.","keywords":["relation extraction","language models","BERT","RoBERTa","large language models","few-shot relation extraction","TACRED","DocRED"],"falsifier":"A concrete check would be to rebuild the three top-five tables (TACRED, DocRED, FewRel) with systems published through 2024 at all major NLP and ML venues, not just the four ACL-family conferences, and count what fraction of top results are BERT/RoBERTa versus LLM-based; if LLMs take a majority of the top entries, the 75% split is an artifact of corpus scope rather than a property of the field.","tokens_in":45592,"feed_emoji":"🔗","tokens_out":8505,"duration_ms":69505,"temperature":0.7,"pith_summary":"This survey tries to establish where relation extraction (RE) actually stands after four years of language-model-driven research. Reviewing 137 papers from the ACL family of conferences (2020–2023), with 65 meeting its inclusion criteria as research contributions, it argues that BERT-based encoders are the dominant recipe: BERT appears in nearly 55% of the papers, and BERT with RoBERTa supplies 75% of the top results in the three headline benchmark tables it compares (TACRED, DocRED, FewRel). The paper also claims that large language models such as T5 and GPT are currently peripheral in most RE settings, but lead in few-shot and zero-shot scenarios, where they identify previously unseen relations. A sympathetic reader would care because the survey maps a scattered literature into a picture of which models, tasks, and datasets are actually driving progress.","feed_headline":"BERT and RoBERTa take 75% of top relation-extraction results","feed_subtitle":"Across TACRED, DocRED and FewRel, LLMs lead only in few-shot and unseen-relation settings.","key_machinery":"The machine that carries the argument is the survey's curated corpus and its three comparative benchmark tables. The corpus is assembled by searching ACL, NAACL, AACL, and EACL proceedings for “Relation Extraction” in title or abstract and then applying inclusion and exclusion criteria (dropping temporal RE, NER-focused work, non-LM methods, and non-adopted encodings), yielding 65 research papers; the benchmark tables then rank the top five systems on TACRED, DocRED, and FewRel. These tables are what make the 75% BERT/RoBERTa claim concrete, and the paper's model-usage table links that performance dominance to adoption rates across papers.","core_discovery":"The central claim, stated in the abstract and supported in Section 6, is that BERT-based methods remain the state of the art for relation extraction. In the survey's comparison of the top five systems on TACRED (sentence-level), DocRED (document-level), and FewRel (few-shot), BERT and RoBERTa models hold 75% of the primary outcomes; RoBERTa-large is the encoder behind every top DocRED system, which the authors explain through RoBERTa's larger pretraining corpus, exclusive masked-language-model objective, and longer sequence handling. Large language models such as T5 and GPT contribute roughly 25% of the top outcomes and are especially effective in few-shot and zero-shot settings, where they can handle relations unseen in training. The paper also claims these LLMs are underused in the field, appearing in only about 8.5% of the papers, and attributes this to BERT's architectural alignment with RE rather than to model accessibility.","pith_inferences":["If the same benchmark comparison were widened to EMNLP, ICLR, and NeurIPS papers from 2023–2025, the 75/25 BERT-to-LLM split could narrow, since the survey window ends before large instruction-tuned models were routinely evaluated on RE; the dominance claim should be read as true for the ACL-family corpus it sampled.","The paper's final remark on relation “domain size” suggests a testable design rule: match model memory or generalization capacity to the cardinality of the relation type, which could be evaluated with per-relation F1 breakdowns on TACRED and FewRel.","The low 8.5% adoption of LLMs may be a lagging indicator rather than a ceiling; if the few-shot advantage on unseen relations compounds with longer context windows, document-level RE is the most likely place for LLMs to displace BERT-style encoders next.","A direct controlled comparison that keeps the same encoder backbone and varies only the pretraining objective (MLM versus text-to-text) would separate the architecture's contribution from BERT's head start in adoption."],"forward_implications":["Practitioners seeking top accuracy on standard RE benchmarks can still start from a fine-tuned BERT or RoBERTa encoder rather than a large generative model.","TACRED, DocRED, and FewRel function as the field's de facto evaluation triad, so new RE systems should report on all three to be directly comparable.","For document-level RE, the winning recipe is RoBERTa-large-style pretraining with long sequences and no next-sentence prediction, pointing future work toward context-window extensions.","Large language models should be targeted at few-shot and zero-shot settings, where their demonstrated advantage is generalizing to unseen relations.","The overall trajectory is toward transformer-based methods, with CNN, LSTM, and static word-embedding baselines fading out of the surveyed literature after 2021."],"supporting_citations":[{"why":"Introduces BERT, the pretrained encoder whose dominance the survey documents.","marker":"[23]"},{"why":"Introduces RoBERTa, the variant behind the top document-level RE results in the survey's tables.","marker":"[69]"},{"why":"Prior review of large language models in relation extraction that this survey extends and compares against.","marker":"[117]"},{"why":"Introduces DocRED, the document-level benchmark used in the survey's top-five comparison.","marker":"[140]"},{"why":"Introduces TACRED, the sentence-level benchmark used in the survey's top-five comparison.","marker":"[153]"},{"why":"Introduces FewRel, the few-shot benchmark used in the survey's top-five comparison.","marker":"[39]"},{"why":"Introduces T5, the text-to-text LLM whose few-shot capabilities the survey highlights.","marker":"[94]"},{"why":"The open-source model library that made pretrained LMs broadly accessible, used to argue adoption is not gated by access.","marker":"[124]"}],"fun_headline_variants":["BERT and RoBERTa dominate relation extraction in new survey","75% of top RE results use BERT, LLMs excel in few-shot","Survey: BERT still leads relation extraction, LLMs for few-shot","LLMs underused in relation extraction despite few-shot wins","Relation extraction: BERT prevails, T5 shines for new relations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole BERT-dominance conclusion rests on the assumption that the 65 papers selected from ACL, NAACL, AACL, and EACL between 2020 and 2023, together with the authors' inclusion and exclusion criteria, fairly represent the broader field of relation-extraction research.","fun_headline_variants_meta":{"raw":{"variants":["BERT and RoBERTa dominate relation extraction in new survey","75% of top RE results use BERT, LLMs excel in few-shot","Survey: BERT still leads relation extraction, LLMs for few-shot","LLMs underused in relation extraction despite few-shot wins","Relation extraction: BERT prevails, T5 shines for new relations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1313,"prompt_tokens":845,"completion_tokens":468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":377}},"tokens_in":461,"tokens_out":468,"duration_ms":4278,"temperature":1.0,"reasoning_tokens":377,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:26:51.611055+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be to rebuild the three top-five tables (TACRED, DocRED, FewRel) with systems published through 2024 at all major NLP and ML venues, not just the four ACL-family conferences, and count what fraction of top results are BERT/RoBERTa versus LLM-based; if LLMs take a majority of the top entries, the 75% split is an artifact of corpus scope rather than a property of the field.","supporting_citations":[],"review_version":1}