{"id":"c8f981a9-97d2-497a-8bd1-fad18d5199f0","arxiv_id":"2507.09925","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"DepBERT adds dependency-tree and POS-tag signals to a BERT token classifier and reports improved cause/effect extraction across three datasets.","lead":"DepBERT adds a dependency-tree attention channel and POS-tag signals to a BERT-style token classifier for extracting cause and effect phrases. The authors report accuracy gains on three datasets, including a new 22,273-sentence LLM-generated medical corpus.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is not yet supported because the largest dataset, CAUSAL-GPT, is acknowledged to contain duplicated, templated LLM-generated sentences, and the paper gives no evidence that duplicate or near-duplicate sentences were excluded from the random train/test split; the observed gains…","rationale":"The strongest_claim is an empirical superiority claim, so the evaluation is the load-bearing structure. The reader's REJECT verdict is justified. Among all evaluation weaknesses, the CAUSAL-GPT duplication issue is the most specific and the most easily settled: it is self-admitted in Section 4.1 and directly affects the largest table. The proposed check is concrete: release the data, measure cross-split near-duplicates, and recompute. If the margins collapse, the main new-dataset evidence disappears; if the margins survive, this particular concern is refuted, though the omitted syntax-aware baselines and SemEval split contamination would still weaken the state-of-the-art claim. I partially align with the reader's weakest_assumption: I do not rest the attack on SpaCy parse errors, but on the unvalidated, templated, potentially duplicated LLM dataset. This is a fair and falsifiable concern, and no claim of fraud is made.","tokens_in":11950,"tokens_out":6347,"duration_ms":81934,"concrete_test":"Release CAUSAL-GPT with train/test instance IDs and token spans; apply MinHash/LSH over token n-grams to find exact and near-duplicate sentences between train and test after the stated preprocessing. Recompute Table 2 excluding every test instance with a near-duplicate in train, or after deduplicating the whole corpus before splitting. If the deduplicated exact-match margin of DEPBERT over BERT/LLaMA drops materially, the reported superiority on this dataset is a template or duplication artifact rather than a genuine improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DEPBERT's headline claim is that it is 'better than various state-of-the-art supervised causality extraction methods' over three datasets (abstract; Tables 2-4). The most load-bearing leg is CAUSAL-GPT (22,273 instances, Table 2), created in Section 4.1 by an LLM and a program for active/passive sentence construction. The paper explicitly notes that 'the generated sentences may exhibit duplication' and require preprocessing, yet it does not state that deduplication was performed before the random 60/30/10 split. Duplicate or near-duplicate sentences appearing in both train and test would inflate exact-match accuracy by memorization. Moreover, Table 1 shows templated sentences in which the cause and effect terms appear verbatim ('X can lead to Y', 'X is a major risk factor for developing Y'), so a token tagger can succeed by lexical cues. Without releasing CAUSAL-GPT or providing a deduplication and human-validation analysis, the 85.8% exact-match accuracy of DEPBERT versus 81.1% for BERT on this dataset cannot be interpreted as evidence of causal extraction ability. The same concern is reinforced by the absence of comparisons to the syntax-aware transformers cited in Section 2 (Ahmad et al. 2021; Sachan et al. 2020), which are the closest state-of-the-art competitors.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DepBERT, a two-tower transformer model that combines a standard BERT encoder with a dependency-tree-aware attention tower, gated together to produce token representations for cause/effect phrase extraction. The authors introduce a new LLM-generated dataset, CAUSAL-GPT, with 22,273 instances, and report experiments on SemEval, SCITE, and CAUSAL-GPT showing that DepBERT outperforms baselines including BERT, LLaMA, and LSTM-based methods. The core claim is that integrating dependency structures and POS tags into the transformer architecture improves exact-match accuracy for causal phrase extraction.","tokens_in":12287,"tokens_out":4301,"duration_ms":48340,"significance":"If the reported gains are valid, the dependency-aware gated architecture is a plausible contribution to supervised causality extraction, and the new dataset could be a resource for the community. The paper includes useful ablations (BERT plus dependency, BERT plus POS tags) and consistent qualitative trends across three datasets, which lend initial plausibility to the mechanism. However, the evaluation protocol contains serious flaws—especially the merging of predefined train/test splits in SemEval and the acknowledged duplication in CAUSAL-GPT—that currently prevent the results from supporting the central claim.","major_comments":[{"comment":"The manuscript states that for SemEval, \"Train, test and validation partitions are then merged to concatenate into a single dataset,\" after which a new random 60/30/10 split is created. This merges the original test partition into the training pool, so original test instances can appear in the new training and validation subsets. Consequently, the numbers in Table 3 cannot be interpreted as a valid measure of generalization, and the claim that DepBERT outperforms baselines on SemEval is not supported. The authors must use the original predefined splits or provide a strong justification for why incorporating the original test set into training is acceptable.","section":"§4.1, SemEval data"},{"comment":"The paper acknowledges that \"the generated sentences may exhibit duplication\" and that preprocessing is needed, but it does not state whether duplicates or near-duplicates were removed before the random 60/30/10 split. Given the templated nature of the examples in Table 1 (e.g., \"Diabetes can lead to blindness if left uncontrolled\", \"Smoking is a major risk factor for developing lung cancer\"), lexical overlap between train and test can inflate exact-match accuracy through memorization. The authors must report a deduplication analysis (e.g., exact and near-duplicate rates between train and test) and either remove duplicates or use a split that prevents leakage.","section":"§4.1, CAUSAL-GPT dataset"},{"comment":"All results are reported from \"the initial stable runs\" with no multiple seeds or variance. Fine-tuning of transformer models is stochastic, and the margins in Table 3 are very small (e.g., DepBERT 0.842 vs. BERT plus dependency 0.841 exact accuracy). Without standard deviations or significance tests, the claim that DepBERT is \"better\" than baselines is not statistically supported. The authors should run multiple seeds and report mean and standard deviation, or at least show that the differences are consistent across seeds.","section":"§4.3, Experimental setup"},{"comment":"DepBERT has 227 million trainable parameters, while BERT has 109 million and the BERT-plus-dependency baseline is reported as 110 million. The comparison is therefore confounded by parameter count: the improvement attributed to the dependency-aware mechanism could simply reflect additional model capacity. The authors should ablate with a matched-capacity baseline (e.g., a larger BERT variant or a two-tower model without dependency edges) to isolate the contribution of the dependency structure.","section":"§4.3, Model parameters"},{"comment":"The paper cites syntax-augmented transformers (Ahmad et al., 2021; Sachan et al., 2020) as related work but does not compare against them in the experiments. Since these are the closest state-of-the-art competitors for syntax-aware models, the claim of outperforming \"various state-of-the-art supervised causality extraction methods\" is not fully established. The authors should add such comparisons or explain why they are not applicable.","section":"§2 and §4.2, Missing comparisons"}],"minor_comments":[{"comment":"The sentence \"for LLaMA, BERT, BERT plus Dependency, BERT plus POS tags, Sentence-BERT, the number of trainable parameters stands at 524 million, 109 million, 110 million, and 110 million, respectively\" lists five methods but only four parameter counts, and the LLaMA value (524M) is inconsistent with the earlier statement that LLaMA has 7B-65B parameters. This should be clarified.","section":"§4.3, Parameter count list"},{"comment":"In the attention equations, the symbol V is used both for the vertex set of the dependency graph and for the set of tokens connected to a given token via edges, which is confusing. Also, the softmax in the attention equation is taken over the set V but the denominator is written as a sum over j in [1,|V|] without explicitly defining the correspondence; please align the notation.","section":"§3.3, Notation"},{"comment":"The description of \"BERT plus dependency\" says it differs from BERT in that it is \"partially pretrained\" but does not specify whether it uses the same two-tower architecture as DepBERT. Clarify whether this baseline includes the dependency-attention tower without POS tags, and why its parameter count is only 110M when DepBERT has 227M.","section":"§4.2.4, BERT plus dependency baseline"},{"comment":"The definition of exact accuracy states that a model must predict all causal and effect tokens correctly, but it is not stated whether a sentence with multiple cause-effect pairs is scored per pair or per sentence. Please clarify the exact scoring unit.","section":"§4.4, Exact-match accuracy"}],"recommendation":"major_revision","confidential_remarks":"The SemEval split merging and CAUSAL-GPT duplication are serious evaluation flaws that, if not addressed, would justify rejection. However, they are fixable with a re-run of experiments under proper protocols. The parameter-count confound and missing state-of-the-art baselines also need attention. Given the plausibility of the core idea, I recommend a major revision rather than outright rejection, but the authors must provide a substantially revised evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper to know about: DepBERT, a two-tower transformer that injects dependency-tree attention and POS tags into a BERT-like encoder for cause-effect phrase extraction. The exact gated combination is new, and the authors deserve credit for a readable model description, including ablations that separate dependency and POS contributions. The CAUSAL-GPT dataset is a real artifact, though currently unreleased.\n\nWhat the paper does well: the model design is clear, the gating mechanism is explained, and the ablations show both dependency and POS tags matter. The writing is clean. If the evaluation were sound, the gains over BERT would be a useful increment for a contained NLP task.\n\nWhere it falls apart: three issues, roughly in severity. First, the SemEval evaluation is contaminated. They merge the original train and test partitions and then randomly re-split 60/30/10. Original test instances can appear in training, so the SemEval numbers in Table 3 are not interpretable. That is a real protocol violation, not a quibble. Second, the CAUSAL-GPT dataset is acknowledged to contain duplicated, templated sentences, and the paper does not state that deduplication or near-duplicate removal was done before the random split. With examples like \"X can lead to Y\", a token tagger can ride lexical cues. Without a leakage analysis or human validation, the 85.8% exact-match accuracy on that dataset cannot support the headline. Third, they do not compare against the syntax-augmented transformers they cite (Ahmad et al. 2021, Sachan et al. 2020), which are the closest existing approaches; that omission makes \"state-of-the-art\" hard to assess. Also, DepBERT has roughly double BERT's parameter count, and there are no error bars or multiple seeds, so it is unclear whether the gains come from syntax or capacity.\n\nNone of this requires the architecture to be wrong. The idea is plausible, and the ablation pattern is consistent across datasets. But as written, the central claim is not yet supported.\n\nThis paper is for people working on syntax-aware transformers for relation extraction, and for anyone who cares about evaluation hygiene in NLP. It deserves a serious referee, but only with a clear expectation of heavy revision: fix the SemEval split, release CAUSAL-GPT with a deduplication statement, add the missing syntax-aware baselines, and report variance. I would engage with it after those changes, not before.\n\nRecommendation: send to peer review for a revise-and-resubmit judgement, but the current version should not be accepted as-is.","headline":"DepBERT is a plausible architecture but the evaluation has a load-bearing split-contamination problem and an unvalidated LLM-generated dataset, so the headline result does not yet hold.","tokens_in":12785,"tokens_out":4375,"would_cite":false,"duration_ms":47107,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A gated two-tower transformer that adds dependency-tree attention beats BERT at cause-effect extraction.","keywords":["cause-effect extraction","dependency tree","transformer","token classification","BERT","gated attention","POS tags","causality extraction"],"falsifier":"Rewire the dependency edges in the right tower randomly while preserving each token's degree, retrain, and check whether exact-match accuracy falls substantially; if it does not, the specific dependency structure is not carrying the reported gain. A second check is to have human annotators verify a random sample of CAUSAL-GPT sentences and rerun the comparison on that verified subset: if DepBERT's margin over BERT largely disappears, the headline numbers are an artifact of the generated data's repetitive templates.","tokens_in":11763,"feed_emoji":"🧠","tokens_out":9884,"duration_ms":88086,"temperature":0.7,"pith_summary":"The paper sets out to show that a transformer can extract cause and effect phrases more accurately when its attention is made aware of the sentence's dependency tree. To that end, the authors build DepBERT, a two-tower model whose second tower restricts attention to neighboring tokens in the spaCy dependency graph and whose gate fuses that syntactic signal with ordinary BERT contextual embeddings. They evaluate the model as a token classifier on the SemEval and SCITE benchmarks plus a new 22,273-sentence medical dataset generated by an LLM, reporting higher exact-match accuracy and F1 than BERT, Sentence-BERT, LLaMA, and LSTM-based baselines on all three. The significance, if the results hold, is that grammatical structure can be injected into a pretrained transformer through an additional attention channel rather than through retraining or prompting.","feed_headline":"Dependency-aware transformer tops BERT at cause-effect extraction","feed_subtitle":"Adding syntax-tree edges to attention lifts exact-match pair accuracy on three causality benchmarks.","key_machinery":"The load-bearing machinery is the gated two-tower encoder. The left tower is the standard BERT encoder; the right tower is a graph-attention encoder over the dependency tree, where an affinity score between connected tokens is $a_{ik} = (v_i W_1)(v_k W_2)^T$, attention weights are softmax over the neighbor set, and the output is a weighted sum of value vectors passed through a residual normalization and a GELU feed-forward layer. POS-tag embeddings are added to the token, positional, and input embeddings in both towers. The outputs of the two towers, $\\mathbf{e}_i^b$ and $\\mathbf{e}_i^t$, are fused by $\\mathbf{e}_i^s = \\sigma(\\mathbf{e}_i^b W_6 + c)$ and $\\mathbf{e}_i = \\mathbf{e}_i^s \\odot \\mathbf{e}_i^b + (1-\\mathbf{e}_i^s) \\odot \\mathbf{e}_i^t$, which is a learned per-token gate. This gate lets the model decide how much of the final token representation comes from the dependency structure versus the sequential context.","core_discovery":"The paper's central claim is that incorporating a sentence's dependency tree into a transformer's attention mechanism improves extraction of cause-effect pairs. DepBERT runs two encoders in parallel: a standard BERT tower that computes full self-attention over all tokens, and a dependency tower in which each token's attention is restricted to its neighbors in the dependency graph, with query, key, and value matrices and a softmax over that neighbor set. Each token's input is the sum of its input, positional, and part-of-speech embeddings. The two towers' outputs are merged with a learned gate, so the final representation can weigh sequential co-occurrence evidence against syntactic evidence per token. Trained on token-level labels for cause, effect, special, and other tokens, the model is reported to outperform all compared baselines in exact-match accuracy and F1 across SemEval, SCITE, and CAUSAL-GPT.","pith_inferences":["We infer that the gated fusion could generalize to other relation-extraction tasks, such as drug-drug interaction or protein-protein interaction, where dependency edges carry much of the semantic signal; the paper does not test these tasks.","We infer that the model's accuracy will depend on parser quality; a testable extension would replace spaCy with gold parses or another parser to measure how much of the gain is parser-driven.","We infer that the CAUSAL-GPT template structure could inflate the absolute numbers: if the LLM repeats a narrow set of sentence frames, a model can memorize surface patterns, so human-validated diversity checks would make the benchmark's numbers more interpretable.","We infer that the gate's per-token values could be used diagnostically: tokens where the gate favors the dependency tower may correspond to the causal verb and its arguments, which could be analyzed in a follow-up study."],"forward_implications":["If DepBERT's reported gains hold, dependency-edge attention is a viable way to inject syntax into transformer token classification without changing the pretraining objective.","On all three datasets, the exact-match accuracy of DepBERT is the highest among the compared models, so the dependency channel plus POS tags yields a practical accuracy increase in cause-effect phrase extraction.","The CAUSAL-GPT dataset of 22,273 sentences gives the field a large training resource, though its automatic generation means labels should be inspected before reuse.","Because the two ablation baselines (BERT plus dependency, BERT plus POS tags) both trail the full model, the paper concludes that combining dependency structure and POS information contributes more than either alone."],"supporting_citations":[{"why":"Supplies the pretrained BERT base that initializes the left tower and serves as a primary baseline.","marker":"(Devlin et al., 2018)"},{"why":"Produces the dependency trees whose edges define the right tower's attention neighborhoods.","marker":"(Honnibal and Montani, 2020)"},{"why":"Provides the SemEval 2007 Task 4 cause-effect instances used in the combined SemEval benchmark.","marker":"(Girju et al., 2007)"},{"why":"Provides the SemEval 2010 Task 8 relation instances used in the combined benchmark.","marker":"(Hendrickx et al., 2010)"},{"why":"Supplies the SCITE dataset and the self-attentive Bi-LSTM-CRF baseline.","marker":"(Li et al., 2021)"},{"why":"Defines the token-classification framing and the linguistically informed Bi-LSTM baseline.","marker":"(Dasgupta et al., 2018)"},{"why":"LLaMA serves as a large pretrained-transformer baseline for comparison.","marker":"(Touvron et al., 2023)"},{"why":"The graph attention mechanism that the dependency tower adapts for attending over edges.","marker":"(Veličković et al., 2017)"}],"fun_headline_variants":["DepBERT: syntax-aware attention for cause-effect pairs","Syntax-aware transformer tops causality benchmarks","Dependency tree inside transformer improves cause-effect extraction","DepBERT: syntax edges boost cause-effect pair accuracy","Syntax-aware attention trumps BERT on cause-effect pairs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation's validity assumes the spaCy dependency parses and the automatically generated CAUSAL-GPT cause/effect labels are accurate enough that the measured improvements reflect genuine causality extraction rather than artifacts of the parser or the text generator.","fun_headline_variants_meta":{"raw":{"variants":["DepBERT: syntax-aware attention for cause-effect pairs","Syntax-aware transformer tops causality benchmarks","Dependency tree inside transformer improves cause-effect extraction","DepBERT: syntax edges boost cause-effect pair accuracy","Syntax-aware attention trumps BERT on cause-effect pairs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000537,"raw_usage":{"total_tokens":2556,"prompt_tokens":902,"completion_tokens":1654,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1582}},"tokens_in":518,"tokens_out":1654,"duration_ms":15139,"temperature":1.0,"reasoning_tokens":1582,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:43:43.522451+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rewire the dependency edges in the right tower randomly while preserving each token's degree, retrain, and check whether exact-match accuracy falls substantially; if it does not, the specific dependency structure is not carrying the reported gain. A second check is to have human annotators verify a random sample of CAUSAL-GPT sentences and rerun the comparison on that verified subset: if DepBERT's margin over BERT largely disappears, the headline numbers are an artifact of the generated data's repetitive templates.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained BERT base that initializes the left tower and serves as a primary baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Produces the dependency trees whose edges define the right tower's attention neighborhoods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SemEval 2007 Task 4 cause-effect instances used in the combined SemEval benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SemEval 2010 Task 8 relation instances used in the combined benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the token-classification framing and the linguistically informed Bi-LSTM baseline."}],"review_version":1}