{"id":"0b759783-947d-4615-bfe1-91c1a411fd65","arxiv_id":"2507.22926","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A BERT input format that appends the head and tail entity names after the document is claimed to beat all prior document-level relation extraction systems, but the reported gains rest on misaligned evaluation protocols.","lead":"This paper proposes adding the two entity names as separate segments after a document when feeding it into BERT, then classifying the relationship from the first output token. It reports much higher F1 scores than previous systems on three relation extraction datasets, but the evaluation setup appears inconsistent with standard benchmark protocols.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported SOTA gains rest on non-standard evaluation: DocRED 'test' in Table V is the dev split, and the Re-DocRED model was trained on DocRED distant; official-split results are required before the central claim can be assessed.","rationale":"The reader's weakest assumption identifies exactly the load-bearing weakness: the experimental protocol is not the same as that of the cited systems. I verified this directly in the manuscript. Table II labels its test column as the DocRED dev set, so the 88.91% headline F1 in Table V is not an official DocRED test result. Table III labels its training data as DocRED train distant and its test column as the Re-DocRED test set, so the Re-DocRED comparison mixes training domains. These are not subtle scoring issues; they change the meaning of every comparative number in the paper. The additional single-label versus multi-label mismatch is also visible in Algorithm 1's softmax and argmax, and it independently undermines comparability with multi-label baselines. I looked for independent support that might rescue the claim: there is no code, no official test submission, no same-backbone controlled comparison, and no machine-checked verification, so the unusually large improvement over DocRE-CLiP rests entirely on the non-standard protocol. Because the reader already reached REJECT for these reasons, my stress-test does not move the verdict; it confirms it. If a future revision submits to the official DocRED test server and retrains on Re-DocRED with a multi-label output, the claim would need to be re-evaluated on those new numbers.","tokens_in":12030,"tokens_out":2444,"duration_ms":30336,"concrete_test":"Run the described pipeline (or its released implementation, if provided) on the official DocRED train_annotated or train_distant split, submit predictions to the DocRED test server, and record the reported test F1; then train the same model from scratch on the Re-DocRED training split and evaluate on Re-DocRED dev and test. If the official DocRED test F1 exceeds 66.31% and the Re-DocRED test F1 exceeds 67.53% under those exact splits, the central SOTA claim is supported; if the numbers fall into the 57-65% range typical of published BERT-based systems, the claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that its input encoding (document concatenated with head and tail entity text, classified from [CLS]) exceeds state-of-the-art results on DocRED, Re-DocRED, and REBEL. The evidence for this claim is Tables V and VI, which compare against published systems. However, the paper's own table headers invalidate the comparison. Table II is headed \"Test: DocRED (Dev set)\", so the 88.91% \"test\" F1 used in Table V is actually computed on the DocRED development set, not on DocRED's official test set (which is held out and evaluated through the DocRED CodaLab server). Likewise, Table III is headed \"Train & Val: DocRED train distant Test: Re-DocRED (Test set)\", meaning the reported Re-DocRED test F1 of 67.19% was produced by a model trained on DocRED distant supervision, whereas the cited Re-DocRED systems train on Re-DocRED's own training split. Comparing a model trained on different data and evaluated on a dev set against published test-set numbers is not a valid SOTA comparison. A second, independent issue reinforces the concern: Algorithm 1 lines 11-12 apply softmax and then argmax over relations, making the model single-label, while DocRED and Re-DocRED are multi-label datasets where an entity pair can have multiple valid relations. Even if the splits were corrected, a single-label output layer cannot reproduce the multi-label evaluation setting of the baselines without a thresholding or multi-label head, so the reported F1 numbers are not measuring the same task. The paper also provides no code or official test-server submission, so the unusually large gap (88.91% vs. 66.31% for DocRE-CLiP on DocRED) has no independent confirmation. The central claim is therefore unsupported by the experiments as reported; it could survive only if the same protocol were run on the official splits and with a multi-label output.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a document-level relation extraction method built on BERT. The input sequence concatenates the full document text with the head entity and tail entity texts as separate segments, and the [CLS] representation is passed through a fully connected layer followed by softmax to predict the relation for a given entity pair. The authors claim state-of-the-art or better F1 results on DocRED, Re-DocRED, and REBEL, reporting large margins over existing systems in Tables V and VI. The contribution is primarily empirical: a new input encoding for global context, with experiments on three datasets and comparisons to published baselines.","tokens_in":12355,"tokens_out":3419,"duration_ms":37111,"significance":"If the reported results were valid, the proposed encoding would be a surprisingly simple and effective way to inject global context into BERT for document-level relation extraction, and the cross-dataset results on DocRED, Re-DocRED, and REBEL would be noteworthy. The paper does not release code, and the empirical evaluation is the entire basis of the contribution, so the correctness of the evaluation protocol is load-bearing. Several visible protocol problems, however, invalidate the central comparison, and the single-label classification head is mismatched with the multi-label nature of the benchmarks. The paper's own tables and text contain internal contradictions about whether the method surpasses or merely approaches baselines on Re-DocRED. I therefore do not think the central claim is currently supported.","major_comments":[{"comment":"The reported DocRED test F1 of 88.91% is not computed on the official DocRED test set. Table II is headed \"Train & Val: DocRED train distant Test: DocRED (Dev set)\", and Table V labels the 88.91 value as \"Test F1\". DocRED's official test set is held out and evaluated through the CodaLab server; evaluating on the dev set and calling it test F1 is a different protocol. The comparison against published test-set numbers in Table V is therefore invalid, and the claimed 22.60-point improvement over DocRE-CLiP is not supported.","section":"IV-C, Tables II and V"},{"comment":"The Re-DocRED result is also obtained under a non-comparable protocol. Table III is headed \"Train & Val: DocRED train distant Test: Re-DocRED (Test set)\", meaning the model was trained on DocRED distant supervision rather than on Re-DocRED's own training split. The baselines in Table VI train on Re-DocRED's training data, so the reported 67.19 test F1 cannot be compared with them. Additionally, even under this mismatched protocol, 67.19 is not higher than the DREEAM test F1 of 67.53 reported in the same table, which contradicts the text claiming that the proposed method surpasses other approaches by a significant margin.","section":"IV-C, Tables III and VI"},{"comment":"The classification layer applies softmax followed by argmax over relations, which selects exactly one relation per entity pair. DocRED and Re-DocRED are multi-label datasets: an entity pair can hold several valid relations simultaneously. A single-label output head cannot reproduce the multi-label evaluation setting of the baselines without a thresholding or multi-label classification head. This mismatch means the reported precision, recall, and F1 values are not measuring the same task as the cited systems, independent of the data-split issues.","section":"III, Algorithm 1, lines 11-12"},{"comment":"The REBEL dataset is not adequately identified and is not evaluated against any baseline. The text describes REBEL as a BART-based seq2seq relation-extraction dataset, citing reference [28], but the cited NeurIPS 2024 paper is about reinforcement learning via regressing relative rewards, not the REBEL relation-extraction model/dataset. Table IV reports roughly 93% F1 on REBEL with no comparison to prior work, so it cannot support the conclusion that the proposed method outperforms state-of-the-art methods on this dataset.","section":"IV-A, REBEL paragraph and Table IV"},{"comment":"The conclusion contains an internal contradiction. The introduction and abstract state that the method achieves higher precision, recall, and F1 on all considered datasets, but the concluding paragraph says \"its performance on the Re-DocRED dataset still requires enhancement\" and lists false negatives on Re-DocRED as future work. Since Table VI shows a Re-DocRED test F1 below the best baseline, the blanket claim of superiority on all datasets is not consistent with the paper's own reported numbers.","section":"V, Conclusions"}],"minor_comments":[{"comment":"The experimental setup states that 80% of the data was used for training and the remainder for testing and validation, but DocRED and Re-DocRED have fixed official train/dev/test splits. The relationship between this custom split and the official splits used in Tables II and III is unclear and should be stated precisely.","section":"IV-B"},{"comment":"Figure 4 is captioned \"Training and Validation Loss-10 Epochs\", but the text says the optimal number of epochs is three and all reported tables show only three epochs. Please clarify whether training was run for 10 epochs and only three are shown, or whether the caption is incorrect.","section":"IV-C, Figure 4"},{"comment":"The sentence \"indicating a 24.61% increase on the validation set and comparable performance on the test set\" is confusing: 84.00 vs. 67.41 is a 24.6% relative increase, but this is not an absolute percentage-point gap, and the test values are not comparable because of the protocol mismatch.","section":"IV-C, Re-DocRED comparison paragraph"},{"comment":"There are numerous typographical and formatting errors, including \"avish.p@iiitdmj.ac.in\" with a period before @, \"DRN [31]„\" with a stray comma, \"BER T\" in Tables V and VI, and an incomplete parenthesis in Section II in the sentence about multi-label classification. These should be corrected.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The paper's central empirical claim is invalidated by evaluation-protocol mismatches that are visible in the authors' own table headers. The DocRED test numbers are dev-set numbers, the Re-DocRED model is trained on a different dataset, and the single-label head is mismatched with multi-label benchmarks. These are not presentation issues; they make the reported state-of-the-art comparison unsupported. I do not see signs of intentional misconduct, but the manuscript would need a completely new evaluation on official splits and a multi-label output layer before the claim could be assessed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does one simple thing: it appends the head and tail entity texts after the document, feeds the whole sequence through BERT, and classifies the [CLS] token. That is a reasonable and clean baseline idea, and the writing is readable. The experiments, however, do not support the central claim of large gains over SOTA. Table II is headed \"Test: DocRED (Dev set)\", so the 88.91% \"test\" F1 in Table V is actually computed on the dev set. Table III shows the Re-DocRED result was obtained by training on DocRED train distant, not Re-DocRED's own train split. Comparing those numbers against published test-set F1 scores is not a valid SOTA comparison; the gap disappears once the protocol is aligned. On top of that, Algorithm 1 applies softmax then argmax, making the model single-label, while DocRED and Re-DocRED are multi-label. Even with correct splits, a single-label head cannot reproduce the baselines' evaluation. The novelty, as the reader notes, is a small extension of sentence-pair BERT; the paper does not include a same-backbone comparison against, e.g., simply concatenating entities into the input without global context, so the contribution of the proposed encoding is not isolated. I checked the REBEL reference: [28] cites a reinforcement learning paper, not the relation extraction dataset, which is a concrete citation error. There is no code or test-server submission, so the unusually large numbers have no independent confirmation. The circularity burden is low, as the paper trains on external benchmarks, but the invalid benchmarking is a load-bearing flaw. Who is this paper for? A reader who wants a quick sanity check on a straightforward entity-pair input encoding might skim the idea, but the evaluation needs to be redone on official splits and with a multi-label output before the claimed results are credible. I would not cite the numbers, and I would not send this to a serious venue in its current form. If the authors rerun on the correct protocol and report a controlled ablation, the idea could be worth a workshop-level look. For now, the central claim is unsupported, and the paper needs major revision.","headline":"The input encoding is a plausible minor variant, but the reported SOTA results are invalid because the paper compares dev-set scores and a different training split against published test-set numbers, so the central claim collapses until the evaluation is redone on official splits.","tokens_in":12952,"tokens_out":576,"would_cite":false,"duration_ms":7962,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Appending entity text to the document yields state-of-the-art relation extraction on DocRED.","keywords":["document-level relation extraction","global context","input encoding","BERT","multi-sentence reasoning","DocRED","Re-DocRED","REBEL"],"falsifier":"Run the proposed method on the official DocRED test split (the 1,000-document held-out set) and on Re-DocRED using its own training split; if the reported 88.91% and 67.19% F1 numbers do not reproduce, the central claim of state-of-the-art performance collapses.","tokens_in":11822,"feed_emoji":"🔗","tokens_out":5951,"duration_ms":58173,"temperature":0.7,"pith_summary":"The paper proposes a document-level relation extraction method in which the input to BERT is the whole document followed by the head and tail entity texts, each placed in a separate segment. The [CLS] token representation is then classified into relation types, so the model must aggregate evidence from the entire document rather than only co-occurrence sentences. The authors report that this input encoding outperforms graph-based and transformer-based systems on DocRED and Re-DocRED, and also does well on the REBEL dataset, using a stock BERT encoder and a single classification layer. The significance would be that a simple representational change can capture multi-sentence reasoning without task-specific architecture.","feed_headline":"Entity-tagged documents hit 88.9% F1 in DocRED","feed_subtitle":"A simple input encoding appends entity text to the document and classifies the [CLS] token, reporting gains over prior systems.","key_machinery":"The input encoding strategy: the document tokens are placed after [CLS] with segment A, and after a [SEP] marker the head entity tokens and tail entity tokens are appended in segment B with their own [SEP] markers. Positional embeddings run across the whole sequence, so the transformer can relate any document token to the entity pair. The classification layer takes the pooled [CLS] output through a tanh, dropout, and a linear-softmax head, and the model is trained with a cross-entropy loss over relation types. This mechanism works by forcing the representation to be conditioned on the entity pair while still attending to the full document, so evidence from distant sentences can contribute to the prediction.","core_discovery":"The central claim is that representing an entity pair as appended text segments to the document sequence gives a pretrained transformer all the global context it needs for document-level relation extraction. The paper shows that the [CLS] token, after encoding the concatenated sequence, can be mapped through a softmax to predict the relation between the head and tail entities, and that this works across three datasets. The authors state that their method achieved higher precision, recall, and F1 than existing methods on the validation and test sets of all considered datasets, with the largest reported gap on DocRED (88.91% test F1 versus 66.31% for the previous best).","pith_inferences":["Correcting the evaluation protocol—using the official DocRED test split and Re-DocRED's own training split—could substantially reduce the reported margins; a fair comparison would likely show the method competitive rather than state-of-the-art.","The encoding is complementary to other DocRE components such as graph networks or adaptive thresholding, so combining them might yield further gains.","Because the model treats each entity pair independently, it ignores inter-pair dependencies that some recent systems exploit; a shared representation across pairs could improve consistency.","The unusually high REBEL score suggests the benchmark or the split may not be directly comparable to prior REBEL evaluations; a sanity check on the dataset statistics would clarify."],"forward_implications":["On DocRED, the reported test F1 of 88.91% would place the method above all systems listed in the comparison table, including DocRE-CLiP's 66.31%.","The approach classifies each entity pair individually, so it can handle any number of relation types without changing the architecture, as the classification layer is adapted to the dataset's relation set.","The same input encoding transfers to a sentence-level relation extraction dataset (REBEL) without modification, suggesting it is not tied to document-level specifics.","If the reported numbers are taken at face value, the method offers a simpler alternative to graph-based and contrastive-learning DocRE systems while performing at least as well."],"supporting_citations":[{"why":"Introduces the DocRED dataset, the primary benchmark whose test F1 is the main reported result.","marker":"[13]"},{"why":"Presents ATLOP, a key baseline with adaptive thresholding and localized context pooling that the proposed method compares against.","marker":"[14]"},{"why":"Introduces the Re-DocRED dataset, used for the second evaluation, and describes its correction of DocRED's false negatives.","marker":"[27]"},{"why":"Cited as the source of the REBEL dataset, used for the sentence-level evaluation.","marker":"[28]"},{"why":"Reports DocRE-CLiP, the strongest baseline in the DocRED comparison, which the proposed method claims to surpass by a large margin.","marker":"[42]"}],"fun_headline_variants":["Entity segments lift DocRED F1 to 88.9%","Global context via entity tags: 88.9% F1 on DocRED","Entity-as-segment encoding hits 88.9% F1 on DocRED","Append entities, not spans: 88.9% F1 on DocRED"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported advantage over existing systems assumes that the F1 numbers in the comparison tables are measured under the same protocol as prior work; the paper labels the DocRED dev set as 'test' and trains on DocRED's distantly supervised split for the Re-DocRED experiment, so the comparison may not be apples-to-apples.","fun_headline_variants_meta":{"raw":{"variants":["Entity segments lift DocRED F1 to 88.9%","Global context via entity tags: 88.9% F1 on DocRED","Entity-as-segment encoding hits 88.9% F1 on DocRED","Append entities, not spans: 88.9% F1 on DocRED"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000734,"raw_usage":{"total_tokens":3254,"prompt_tokens":888,"completion_tokens":2366,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":2281}},"tokens_in":504,"tokens_out":2366,"duration_ms":21531,"temperature":1.0,"reasoning_tokens":2281,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:46:44.644516+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the proposed method on the official DocRED test split (the 1,000-document held-out set) and on Re-DocRED using its own training split; if the reported 88.91% and 67.19% F1 numbers do not reproduce, the central claim of state-of-the-art performance collapses.","supporting_citations":[{"cited_title":"DocRED: A Large-Scale Document-Level Relation Extraction Dataset","cited_arxiv_id":"1906.06127","evidence_quote":"Introduces the DocRED dataset, the primary benchmark whose test F1 is the main reported result."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents ATLOP, a key baseline with adaptive thresholding and localized context pooling that the proposed method compares against."},{"cited_title":"Revisiting DocRED -- Addressing the False Negative Problem in Relation Extraction","cited_arxiv_id":"2205.12696","evidence_quote":"Introduces the Re-DocRED dataset, used for the second evaluation, and describes its correction of DocRED's false negatives."},{"cited_title":"Rebel: Re- inforcement learning via regressing relative rewards,","cited_arxiv_id":null,"evidence_quote":"Cited as the source of the REBEL dataset, used for the sentence-level evaluation."},{"cited_title":"Revisit- ing document-level relation extraction with context-guided link prediction,","cited_arxiv_id":null,"evidence_quote":"Reports DocRE-CLiP, the strongest baseline in the DocRED comparison, which the proposed method claims to surpass by a large margin."}],"review_version":1}