{"id":"1c37d81a-e847-4679-834b-150fb7563de6","arxiv_id":"2501.01158","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A BioBERT-plus-GCN model using the full dependency adjacency matrix gives mixed results across BioNLP event extraction tasks, beating some baselines on Genia11 but losing on Genia13.","lead":"The paper tests whether feeding a full dependency parse graph into a graph neural network, on top of BioBERT embeddings, improves biomedical event extraction. It reports gains on some BioNLP datasets but its state-of-the-art claim is not consistently supported by its own tables.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"§3.2 ablation conflates dependency graph with added GCN capacity: BioBert-BEE has no GCN while BioBert-GNN-BEE adds a two-layer GCN, so the gains in Tables 3–4 do not isolate syntactic edges.","rationale":"The paper's central assertion has two parts: dependency graph information improves BEE, and the model slightly outperforms SOTA. The second is already questionable because Table 1 shows OneEE beats the model on Genia13. The first rests entirely on Tables 3–4, but those rows compare a model without a GCN to a model with a GCN; the graph is not the only thing changed. GCNs can improve token representations by smoothing and by added parameters even with uninformative edges, so the ablation cannot prove that dependency syntax is responsible. The reader's weakest assumption concerned fair comparison with published baselines; that is a real but different issue. I partially agree with that assessment, but the internal ablation confound is more load-bearing because it undercuts the paper's own controlled comparison, not just the comparison to external numbers. The appropriate remedy is a major revision adding equal-capacity GCN controls, multi-seed significance tests, and a corrected SOTA claim. Since this is fixable and matches the spirit of the reader's conditional verdict, I keep the verdict as CONDITIONAL/UNCHANGED rather than escalating to reject.","tokens_in":6208,"tokens_out":8686,"duration_ms":91095,"concrete_test":"Run BioBert-GNN-BEE with identical hyperparameters, seeds, and preprocessing on Genia11 and Genia13, replacing the normalized dependency adjacency matrix with (i) an identity matrix (self-loops only) and (ii) a complete graph. If either control retains most of the Table 3 gain over BioBert-BEE, the dependency edges are not the causal factor; the ablation should be redesigned with equal-capacity GCN controls and the SOTA statement rechecked against the official macro totals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main evidence for the central claim is the ablation in §3.2, but the two systems are not controlled: Algorithm 1/BioBert-BEE is BioBERT plus MLP sequence/argument classifiers without graph convolution, while Algorithm 2/BioBert-GNN-BEE adds a two-layer GCN over the normalized dependency adjacency matrix plus extra MLP head/dependent blocks. The large improvements (e.g., Genia11 total F1 from 60.97 to 69.81 in Table 3) could come from the extra GCN parameters and feature propagation rather than from dependency edges. A proper isolation would compare GCN models with identical capacity whose only difference is the edge set (true adjacency vs identity/self-loops or randomized adjacency). Separately, the abstract's 'slightly outperforms state-of-the-art' claim is not supported by Table 1 on Genia13, where OneEE's total 78.05 exceeds the model's 77.48. Thus neither the causal attribution to syntax nor the SOTA claim is currently established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two biomedical event extraction (BEE) models, BioBert-BEE and BioBert-GNN-BEE. The first uses a BioBERT encoder followed by sequence-labeling and argument-role classifiers. The second augments this architecture with a two-layer graph convolutional network over the full normalized dependency-parsing adjacency matrix of each sentence, with separate MLP 'head' and 'dependent' networks for trigger-argument scoring. The authors evaluate on Genia11, Genia13, pathway curation (pc), and cancer genetics (cg), reporting trigger identification/classification and argument identification/classification F1 scores. Their ablation study compares the two models to claim that dependency-graph information improves BEE, and they state that the proposed model slightly outperforms state-of-the-art models across datasets.","tokens_in":6462,"tokens_out":3807,"duration_ms":38662,"significance":"If the empirical claims were fully supported, the contribution would be practically useful: it replaces shortest-dependency-path pruning with the full dependency graph via a GCN on top of contextual BioBERT representations, and it provides a simple two-MLP mechanism to counter GCN oversmoothing. The authors are to be credited for testing across four datasets and for making the no-graph vs. graph comparison the central experiment. However, the current evidence is not yet convincing: the ablation does not isolate the syntactic edge information from added GCN capacity, the state-of-the-art claim is contradicted by the paper's own Table 1 on Genia13, and there are no statistical tests or shared evaluation details. The central empirical conclusion therefore needs substantial additional work before the paper can be accepted.","major_comments":[{"comment":"The ablation does not isolate the effect of dependency-graph information. BioBert-BEE has no GCN layers, while BioBert-GNN-BEE adds a two-layer GCN plus the MLP head/dependent blocks, so the large improvements (for example Genia11 total F1 from 60.97 to 69.81) could stem from additional parameters, feature propagation, or randomness rather than from syntactic edges. To support the claim that 'dependency parsing graph improves the performance,' the comparison must hold model capacity and propagation fixed, for example by comparing a GCN with the true adjacency matrix against a GCN with identity/self-loop adjacency or a randomly rewired adjacency. Without such a control, the causal attribution to syntactic information is not established.","section":"§3.2, Tables 3–4"},{"comment":"The abstract's claim that the model 'slightly outperforms state-of-the-art models on BEE over different datasets' is not supported by the paper's own numbers. On Genia13, OneEE's total F1 is 78.05 while the proposed model's total is 77.48, so the proposed model is worse on that dataset. The claim should be restricted to the datasets where the improvement actually holds (e.g., Genia11, pc, cg), or the discrepancy must be explained. In addition, the word 'significant' in the abstract is used without any statistical significance testing; the paper reports no standard deviations, no multiple runs, and no paired hypothesis tests, so apparent differences may not be reliable.","section":"§3.1, Table 1; Abstract"},{"comment":"The comparison against published state-of-the-art numbers is not controlled. Published baselines use their own preprocessing, tokenization, data splits, and metric definitions, so the 'total' scores may not be computed in the same way as the authors' macro-averaged total (which appears to average the four subtask F1 scores). For a credible SOTA comparison, the authors should either rerun the strongest baselines under their own evaluation script or clearly state that the comparison is against published numbers with potentially different metrics, and they should justify why such a comparison is valid. At minimum, the exact formula for 'total' and the metric-computation details must be specified.","section":"§3.1, Tables 1–2"},{"comment":"The manuscript omits essential implementation details needed to reproduce the central experiments. There is no specification of hyperparameters (learning rate, batch size, number of epochs, dropout, GCN hidden dimensions), no definition of how the adjacency matrix is normalized beyond the phrase 'normalise the full adjacency matrix,' and no description of the initialization of the GCN feature vectors or the random seed. The paper also does not provide a code repository or data-access details beyond saying the datasets are on Hugging Face. These omissions are load-bearing for an empirical paper whose main evidence is a new training configuration.","section":"§2.2 and §3.1"}],"minor_comments":[{"comment":"The model name is written inconsistently as 'BioBert', 'BioBERT', 'bioBert', and 'BioBertGNNBEE'; please standardize the notation.","section":"Throughout"},{"comment":"There are formatting errors around Equation (3): 'where|inEquation3 istheconcatenation' is missing spaces and the upright '|' is unclear; please rewrite this sentence and display the concatenation operator clearly.","section":"§2.2, Equation (3)"},{"comment":"The tables contain incomplete rows and inconsistent dashes (for example, GEANet and Extended TEES have only total scores), and the label 'ours-RO1-withGraph' is redundant. Please make the table headers and row labels consistent and add a note explaining how missing values are handled.","section":"Tables 1–2"},{"comment":"Figures 3 and 4 appear to be reproduced from another paper; if so, permission or explicit acknowledgment is required, and the source should be clearly credited in the caption.","section":"Figures 3–4"},{"comment":"Reference [13] lacks a venue and page numbers, and reference [20] lacks a publisher and location; some entries also have inconsistent author-name formatting. Please unify the bibliography style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be an extended abstract rather than a fully developed journal-length article. The main experiment—the ablation—is confounded, and the SOTA claim is internally contradicted by Table 1. Both issues are fixable within the scope of the manuscript if the authors add a capacity-controlled graph ablation, rerun or clearly reinterpret the baseline comparison, and provide statistical and implementation details. I would encourage the editor to request a major revision rather than reject, since the core idea is reasonable and the datasets are publicly available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read after going through it. The core idea is reasonable: instead of pruning the dependency tree to the shortest path, feed the full adjacency matrix into a GCN on top of BioBERT for biomedical event extraction. That is a legitimate, if modest, extension of existing GCN-over-syntax work, and the motivating example about losing a negation word on the pruned path is the right argument to make. The two-MLP head/dependent block to reduce oversmoothing is also a sensible touch. Credit where due: the paper attempts an ablation on all four datasets, and the gains from adding the graph look large (e.g., Genia11 total 60.97 to 69.81).\n\nBut the soft spots are real and one is load-bearing. The ablation does not isolate the dependency edges. BioBert-BEE has no GCN at all; BioBert-GNN-BEE adds a two-layer GCN plus extra MLP blocks. So the improvement in Tables 3 and 4 could come from extra parameters and feature propagation, not from syntactic structure. A proper control would hold capacity fixed and vary only the adjacency (true graph vs. identity or randomized edges). The reader's stress-test note is right on this, and the paper does not address it.\n\nThe SOTA claim is also shaky. On Genia13, the model's total F1 (77.48) is below OneEE's 78.05, so the abstract's \"slightly outperforms state-of-the-art\" is false on that dataset. The comparison to published numbers is done without shared preprocessing or a common evaluation script, and no statistical tests are reported. On top of that, Zhang et al.'s syntax-grounded GCN for joint entity and event extraction (reference [16]) is a close uncompared neighbor; omitting it weakens the novelty claim.\n\nImplementation details are thin: no code, no hyperparameters, no mention of repeated runs. For a paper whose whole argument is empirical, that matters.\n\nWho is this for? People working on biomedical event extraction who want a quick tour of a full-graph GCN variant. The paper is not a waste of time—the idea is worth testing—but as submitted it does not establish its central claims. I would send it to peer review only because the experimental flaw is fixable and the motivating question is legitimate; a good referee would ask for the capacity-matched ablation, code, and honest SOTA numbers. If the authors cannot provide those, the paper should not be accepted.","headline":"Plausible full-dependency-graph idea and a clean motivating problem, but the ablation doesn't isolate the graph and the SOTA claim is contradicted on Genia13; worth one serious referee round, not more.","tokens_in":6938,"tokens_out":1516,"would_cite":false,"duration_ms":17308,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using the full dependency parse graph, not just the shortest path between trigger and argument, improves biomedical event extraction when combined with BioBERT and a graph convolutional network.","keywords":["biomedical event extraction","graph convolutional network","dependency parsing","BioBERT","syntactic information","argument classification","nested events","BioNLP shared tasks"],"falsifier":"Re-running BioBert-BEE and BioBert-GNN-BEE on the same splits with a shared evaluation script and identical preprocessing would settle the claim. If a properly tuned no-graph baseline matches the graph model's F1 on Genia11 and Genia13, then the dependency graph is not the source of the gain; likewise, if published systems re-implemented under the same conditions beat the graph model, the SOTA comparison fails.","tokens_in":6068,"feed_emoji":"🧬","tokens_out":9048,"duration_ms":69469,"temperature":0.7,"pith_summary":"The paper's central claim is that biomedical event extraction improves when token representations are enriched by the entire dependency parse of a sentence rather than by the shortest path between a trigger and its argument. The authors build two models with identical structure, one with and one without a graph convolutional network over the full dependency adjacency matrix on top of BioBERT, and an ablation shows the graph version wins on all four datasets. The paper also reports that the graph model slightly outperforms published state-of-the-art systems on the Genia11, Genia13, cancer genetics, and pathway curation datasets. The authors argue the full graph avoids the fragility of shortest-path representations, where dropping a single word such as 'not' can flip a prediction.","feed_headline":"The whole parse tree, not just the path, boosts event extraction","feed_subtitle":"BioBERT plus a GCN over full dependency syntax edges out published systems on four biomedical event tasks.","key_machinery":"The load-bearing object is the normalized full adjacency matrix of the dependency parse tree, obtained from an off-the-shelf parser and used as the graph the GCN convolves over. Node features are BioBERT token encodings, so the GCN re-embeds each word with its syntactic neighborhood. For argument role classification, the model concatenates a head representation computed by one MLP for the trigger token with a dependent representation computed by a second MLP for the argument token, then passes the concatenation through a linear layer with softmax. The two separate MLPs are the mechanism intended to counteract oversmoothing and oversquashing, by letting distant nodes interact in the classifier without adding more GCN layers.","core_discovery":"The paper aims to establish that feeding the full adjacency matrix of the dependency parsing graph through a two-layer graph convolutional network initialized with BioBERT's contextual token representations gives a practically useful token embedding for biomedical event extraction. Trigger and argument classification are performed by separate MLP networks over head and dependent tokens, introduced to reduce oversmoothing, oversquashing, and missing positional encoding. The ablation between the no-graph BioBert-BEE and the full-graph BioBert-GNN-BEE is the main evidence: total F1 rises from 60.97 to 69.81 on Genia11 and from 69.76 to 77.48 on Genia13. Against published systems, the model reports total F1 values of 69.81 on Genia11, 77.48 on Genia13, 70.29 on cancer genetics, and 75.70 on pathway curation, characterized as a slight improvement over prior state of the art.","pith_inferences":["A natural extension would be to test whether the same full-graph GCN layer helps other structured prediction tasks with nested outputs, such as general relation extraction or abstract meaning representation parsing, where shortest-path heuristics are common.","The comparisons are against published numbers rather than re-implemented baselines; a shared-benchmark study could show whether the slight gains persist under identical evaluation.","Because the dependency graph comes from an off-the-shelf parser, corrupting the parse graph with random edge deletions could directly test whether full-graph completeness or something else drives the improvement.","The approach injects syntax without extra training data, so it could cheaply be grafted onto other BioBERT-based pipelines, potentially benefiting tasks where negation and long-range arguments matter."],"forward_implications":["Shortest-path and pruned-path syntax encodings should be replaced with full dependency graph encodings for event extraction, if the reported gains are real.","The ablation numbers imply syntactic structure contributes substantially over plain contextual embeddings, with Genia11 total F1 rising from 60.97 to 69.81.","A shallow GCN with separate head and dependent MLPs can handle long-range syntactic relations without the depth that usually causes oversmoothing.","The design is practical: a pretrained biomedical language model plus an off-the-shelf dependency parser and a two-layer GCN yields several F1 points with no additional supervision."],"supporting_citations":[{"why":"Supplies the contextualized-GCN baseline with path-centric pruning that the paper contrasts with its full-graph approach; the 'missing word' example motivates the design.","marker":"[17]"},{"why":"Uses shortest dependency path subpaths for relation extraction; the paper argues this representation is lossy for event extraction.","marker":"[11]"},{"why":"GEANet, a published biomedical event extraction system with hierarchical knowledge graphs, used as a state-of-the-art baseline and source of the trigger-argument combination idea.","marker":"[7]"},{"why":"Provides the graph-based dependency parsing formulation whose head-dependent interaction is adapted for the argument role classifier.","marker":"[8]"},{"why":"Syntax-grounded GCN for joint entity and event extraction; the paper borrows its sequence-tagging style with previous tag embeddings.","marker":"[16]"},{"why":"Introduces the biaffine head-dependent interaction that the paper simplifies into concatenation with an MLP.","marker":"[4]"},{"why":"OneEE, a one-stage overlapping and nested event extraction system, serves as a state-of-the-art baseline on Genia11 and Genia13.","marker":"[3]"},{"why":"CasEE, a cascade decoding event extraction system, serves as another state-of-the-art baseline for comparison.","marker":"[13]"}],"fun_headline_variants":["Full dependency graph beats shortest path for event extraction","Whole parse tree, not just one path, yields better event extraction","BioBERT plus GCN over full dependency syntax edges out SOTA","Using full dependency graph via GCN improves event extraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported advantage over published systems assumes the numbers being compared were produced under the same data splits, preprocessing, tokenization, and metric computation; if they were not, the slight state-of-the-art edge could be an artifact of evaluation differences rather than the graph.","fun_headline_variants_meta":{"raw":{"variants":["Full dependency graph beats shortest path for event extraction","Whole parse tree, not just one path, yields better event extraction","BioBERT plus GCN over full dependency syntax edges out SOTA","Using full dependency graph via GCN improves event extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000593,"raw_usage":{"total_tokens":2725,"prompt_tokens":839,"completion_tokens":1886,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":1818}},"tokens_in":455,"tokens_out":1886,"duration_ms":11863,"temperature":1.0,"reasoning_tokens":1818,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:33:11.976303+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running BioBert-BEE and BioBert-GNN-BEE on the same splits with a shared evaluation script and identical preprocessing would settle the claim. If a properly tuned no-graph baseline matches the graph model's F1 on Genia11 and Genia13, then the dependency graph is not the source of the gain; likewise, if published systems re-implemented under the same conditions beat the graph model, the SOTA comparison fails.","supporting_citations":[{"cited_title":"In: Riloff, E., Chiang, D., Hockenmaier, J., Tsujii, J","cited_arxiv_id":null,"evidence_quote":"Supplies the contextualized-GCN baseline with path-centric pruning that the paper contrasts with its full-graph approach; the 'missing word' example motivates the design."},{"cited_title":"In: Erk, K., Smith, N.A","cited_arxiv_id":null,"evidence_quote":"Uses shortest dependency path subpaths for relation extraction; the paper argues this representation is lossy for event extraction."},{"cited_title":"In: Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics","cited_arxiv_id":null,"evidence_quote":"Provides the graph-based dependency parsing formulation whose head-dependent interaction is adapted for the argument role classifier."},{"cited_title":"Neurocomputing422, 118–128 (01 2021)","cited_arxiv_id":null,"evidence_quote":"Syntax-grounded GCN for joint entity and event extraction; the paper borrows its sequence-tagging style with previous tag embeddings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the biaffine head-dependent interaction that the paper simplifies into concatenation with an MLP."},{"cited_title":"In: Pro- ceedings of the 29th International Conference on Computational Linguistics","cited_arxiv_id":null,"evidence_quote":"OneEE, a one-stage overlapping and nested event extraction system, serves as a state-of-the-art baseline on Genia11 and Genia13."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CasEE, a cascade decoding event extraction system, serves as another state-of-the-art baseline for comparison."}],"review_version":1}