{"id":"169f69e4-1cb1-4c8d-8dd5-04821325e7d0","arxiv_id":"2412.14688","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A graph-based event-event relation extraction model that injects logical constraints into event and event-pair nodes and reports state-of-the-art F1 on TRE and SRE benchmarks.","lead":"LogicERE is a neural network for extracting temporal and subevent relations between events in news documents, building a graph that encodes coreference, symmetry, and conjunction constraints. It reports state-of-the-art F1 on four relation-extraction benchmarks, though the evaluation includes nonstandard label modifications and possible use of gold coreference as input.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim for MAVEN-ERE is not supported because LogicERE is evaluated on a manually relabeled task with extra converse labels and oracle coreference edges that the published baselines do not have; a matched re-run is needed.","rationale":"I read the paper as making two coupled claims: a methodological claim that encoding coreference/symmetry/conjunction constraints in a logic-induced graph plus joint losses improves ERE, and an empirical claim of SOTA on MATRES/TCR/HiEve/MAVEN-ERE. The architecture is coherent and the ablations are internally consistent, so I do not object to the method as such. The load-bearing weakness is the evaluation protocol for MAVEN-ERE. The gold-coreference leak identified by the reader is real, and the Cee edges built from test-time annotations give LogicERE information the baselines lack. However, the paper's own 'w/o coreference' ablation costs only 0.3 F1, so that leak is probably not the main source of the reported gains. The more serious problem is that the MAVEN-ERE label set is manually altered: AFTER/VAGUE/SUPEREVENT are added, and the published baselines were not trained or evaluated on those labels. Since the symmetry loss is designed to produce exactly these converse/reflexive labels, the comparison in Table 5 is not apples-to-apples. The conjunction-loss derivation in the appendix is also incorrect: from the product t-norm residuum the penalty should be max(0, log a + log b - log c), not the absolute difference in Eqs. 17-19; this is a separate correctness issue for the 'joint logic learning' component, but it is secondary to the evaluation mismatch for the SOTA claim. Both would need to be fixed before the central claim can be accepted.","tokens_in":15228,"tokens_out":11577,"duration_ms":106094,"concrete_test":"Retrain and evaluate GraphEREjoint and RoBERTajoint on the exact modified MAVEN-ERE label scheme used by the authors (BEFORE/AFTER/SIMULTANEOUS/VAGUE for TRE and SUBEVENT/SUPEREVENT for SRE), with the same splits, event pairs, and evaluation script as LogicERE; if either baseline reaches or approaches LogicERE's 55.0 TRE or 30.3 SRE F1, the reported advantage is an artifact of the label redefinition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is state-of-the-art performance on benchmark datasets (Tables 2-5). For MAVEN-ERE, the paper changes the benchmark: in the Datasets and Metrics section it says it keeps only BEFORE and SIMULTANEOUS for TRE and manually adds AFTER and VAGUE, and for SRE it adds SUPEREVENT as a converse of SUBEVENT. The baselines in Table 5 are inherited from prior work on the unmodified MAVEN-ERE, so they are scored on a different label set. The added labels are exactly what the symmetry constraint produces, so LogicERE gets those positives for free from its own objective, while a baseline never trained to output AFTER/SUPEREVENT will be marked wrong on every added example. This can inflate the reported TRE/SRE F1 independently of any reasoning ability; it alone may explain the +3.0 SRE gain. Separately, the paper states that HiEve and MAVEN-ERE provide ground-truth coreference and that Cee edges are added from those annotations, giving the model oracle event grouping that the compared systems lack; the 'w/o coreference' ablation suggests this is not the main driver (only -0.3 F1), but it still breaks the 'no external tools' claim. A fair evaluation must use predicted coreference or omit Cee at test time.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LogicERE, a neural model for temporal event relation extraction (TRE) and subevent relation extraction (SRE). The core idea is to build a logic constraint induced graph (LCG) with two node types (events and event pairs) and three edge types that encode coreference, symmetry, and conjunction constraints. The model performs high-order reasoning on this graph using a relational graph transformer, and adds joint logic losses that softly enforce the symmetry and conjunction constraints. Experiments are reported on MATRES, TCR, HiEve, and MAVEN-ERE, claiming state-of-the-art performance on all four. The main contributions are the LCG construction and the joint logic learning objectives for event-event relation reasoning.","tokens_in":15493,"tokens_out":5350,"duration_ms":41226,"significance":"If the empirical claims were sound, the paper would make a useful contribution to event relation extraction by embedding logical constraints directly into a graph architecture, potentially removing the need for external parse trees or ontologies. The design of the LCG, with event-pair nodes enabling high-order interactions, is interesting and the ablation study is internally consistent. However, the evaluation contains two serious flaws that undermine the central SOTA claims: the MAVEN-ERE benchmark is modified by manually adding extra relation labels, and ground-truth coreference annotations are used as graph edges on datasets where coreference is a target relation. Because these flaws directly affect the headline comparisons, the contribution cannot be considered validated without a matched, benchmark-faithful evaluation.","major_comments":[{"comment":"The MAVEN-ERE evaluation is not performed on the unmodified benchmark. The paper states that for TRE it 'only consider[s] type BEFORE and SIMULTANEOUS' and then 'manually annotate[s] reflexive relationships AFTER and VAGUE', and for SRE it 'manually annotate[s] corresponding reflexive relationships SUPEREVENT'. The baselines in Table 5 are inherited from prior work that was trained and evaluated on the original MAVEN-ERE label set and instance distribution. Since the label sets and the test instances differ, the reported F1 numbers are not comparable. In particular, the added labels AFTER and SUPEREVENT are precisely the converse relations that the symmetry constraint produces; a baseline that never outputs these labels receives no credit on every manually added instance, while LogicERE can score them via its symmetry objective. This alone may explain the reported +3.0 SRE gain over GraphEREjoint, independently of any reasoning ability. A fair comparison requires either (a) training all baselines on the same modified label set as LogicERE, or (b) evaluating LogicERE on the original MAVEN-ERE label set without the manually added labels. Without this, the SOTA claim for MAVEN-ERE is unsupported.","section":"Datasets and Metrics"},{"comment":"The use of ground-truth coreference annotations to construct Cee event-event edges is a label-leakage problem for HiEve. In the model definition, RSub includes COREF as a target relation, and the paper says 'HiEve and MAVEN-ERE provide ground-truth event coreference annotations' and that Cee edges are added from those annotations. Thus, during both training and testing, the model is given oracle knowledge of a relation that it is also expected to predict. Although Table 4 reports only PARENT-CHILD and CHILD-PARENT micro-averages, Cee edges connect coreferent events and can propagate information across the graph that affects PC/CP predictions, so the reported comparison against baselines that do not receive gold coreference is unfair. The ablation 'w/o coreference' in Table 6 is insufficient because it is performed only on MAVEN-ERE, where COREF is not a target relation; it does not measure the impact of this leak on HiEve. The paper should either remove gold coreference edges at test time, use predicted coreference, or provide a HiEve ablation that isolates this effect.","section":"Logic Constraint Induced Graph"},{"comment":"The claim in the Joint Learning Evaluation that LogicERE 'improves by 3.0% in SRE' over GraphEREjoint and 'surpasses all baselines' is not meaningful under the current protocol, because GraphEREjoint was trained on the original MAVEN-ERE, which does not contain the manually added SUPEREVENT label. As a result, the comparison mixes a different label space with a different evaluation set. This is a load-bearing issue for the paper's central claim of state-of-the-art performance on benchmark datasets. The authors need to rerun all baselines on exactly the instances and label set used for LogicERE, or present results on the original MAVEN-ERE evaluation so that the comparison is apples-to-apples.","section":"Comparison"}],"minor_comments":[{"comment":"There are typos in the abstract: 'uniffed' should be 'unified' and 'Speciffcally' should be 'Specifically'.","section":"Abstract"},{"comment":"The definition of the label sets R_Temp and R_Sub is written as a comma-separated run-on; use formal set notation and define the elements clearly, e.g., R_Temp = {BEFORE, AFTER, EQUAL, VAGUE}.","section":"Model"},{"comment":"The cross-entropy loss in Eq. (12) is written as if each relation label is an independent binary prediction, but the model outputs a probability distribution via softmax. Please clarify how y_ei,ej and p_ei,ej are defined for a multi-class setting.","section":"Equation (12)"},{"comment":"The dynamic window mechanism is described only briefly; please specify how overlapping spans are merged for tokens other than [CLS] and <t>, and how this affects the contextualized representations of events that appear in multiple windows.","section":"Sequence Encoder"},{"comment":"In the ablation 'w/o coreference', the paper says it removes Cee and does not use ground-truth coreference annotations as training labels. Please clarify whether this applies at both training and test time, and state whether the model still predicts the COREF relation on HiEve.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"The evaluation flaws are substantial enough that I would not accept the paper in its current form. The MAVEN-ERE comparison is against baselines on a different task, and the gold-coreference leakage on HiEve is a serious methodological issue. However, the core modeling idea is plausible and the flaws are correctable with a carefully matched evaluation, so I recommend major revision rather than outright rejection. The authors should also be asked to clarify the exact test instances used in the MAVEN-ERE comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the core graph-design idea—event nodes plus event-pair nodes with edges for coreference, symmetry, and conjunction—is a legitimate contribution, and the ablation study is internally consistent. But the headline SOTA claims rest on an evaluation that is not comparable to the baselines, and the joint logic loss in the appendix doesn't match the t-norm derivation. You cannot take the MAVEN-ERE numbers at face value.\n\nWhat's genuinely new: fusing event-centered and event-pair-centered graphs into one relational graph transformer, with edge types that encode three logical constraints, is a neat move. The paper is upfront that it manually adds AFTER/VAGUE/SUPEREVENT to MAVEN-ERE, but it never confronts the consequence: every inherited baseline is trained on the original label set and can never output those labels. Any gold positive from the added converses automatically counts as a baseline error, and those converses are exactly what the symmetry objective produces for the model. That alone could explain the +3.0 SRE gain.\n\nSecond problem: for HiEve and MAVEN-ERE, the Cee edges come from ground-truth coreference annotations, and COREF is one of the SRE labels. So the graph is handed part of the answer at test time. The 'w/o coreference' ablation shows only a -0.3 F1 drop, which suggests this isn't the main driver, but it still violates the paper's 'no external tools' claim and makes the comparison unclean.\n\nThird, the conjunction loss derivation is wrong. The product t-norm residuum gives a max(0, ...) hinge after the negative log; the paper instead writes Lc1 = log p_i,j + log p_j,k - log p_i,k and takes an absolute value. That penalizes satisfied implications, which is the opposite of the intended behavior. The symmetry loss derives correctly, but the conjunction half is a genuine internal inconsistency.\n\nBottom line: the method is plausible and might be salvageable, but as reported the SOTA claim is not supported. I would send it to a serious referee only with the expectation that the authors re-run on matched label sets and predicted coreference. I wouldn't cite it in the next year.","headline":"The graph architecture is interesting and the ablations are clean, but the MAVEN-ERE evaluation is not comparable to baselines and the conjunction loss doesn't follow from the t-norm derivation; the SOTA claim is not supported without a matched re-run.","tokens_in":16045,"tokens_out":5447,"would_cite":false,"duration_ms":34948,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Encoding coreference, symmetry, and conjunction rules into a heterogeneous graph improves extraction of temporal and subevent event relations.","keywords":["event-event relation extraction","temporal relation extraction","subevent relation extraction","logic constraints","relational graph transformer","heterogeneous graph","coreference","joint learning"],"falsifier":"Run the HiEve and MAVEN-ERE SRE evaluations with coreference edges predicted by an independent coreference model (or with event-event edges removed) and check whether the F1 gains over SDLG and GraphERE persist; if the advantage collapses, the reported subevent improvements depend on gold coreference leakage rather than on the logic constraints.","tokens_in":14938,"feed_emoji":"🧩","tokens_out":5846,"duration_ms":36190,"temperature":0.7,"pith_summary":"The paper claims that event-event relation extraction—deciding whether two events are temporally ordered (before/after) or hierarchically linked (parent/child)—can be improved by making logical constraints part of the model's structure rather than only part of its training objective. The proposed LogicERE builds a Logic Constraint Induced Graph (LCG) with event nodes and event-pair nodes, wired by three edge types that respectively encode coreference, symmetry, and conjunction constraints, and reasons over this graph with a relational graph transformer. A joint logic learning module then converts the same constraints into differentiable losses. The authors report state-of-the-art F1 on MATRES, TCR, HiEve, and MAVEN-ERE, and argue the result is more coherent event evolution graphs without external parsers or ontologies.","feed_headline":"Logic constraints wired into a graph lift event-relation accuracy","feed_subtitle":"Encoding coreference, symmetry, and conjunction rules boosts temporal and subevent relation F1 across four benchmarks.","key_machinery":"The key object is the Logic Constraint Induced Graph (LCG), a heterogeneous graph with event nodes and event pair nodes. Three edge types encode the paper's three constraints: event-event edges link co-referenced events (coreference constraint), event pair-event pair edges link pairs sharing an event (symmetry and conjunction constraints), and event-event pair edges connect an event pair to its two events. A relational graph transformer performs high-order reasoning over this graph, using an edge-type scalar as an attention bias, and a joint logic learning module turns symmetry and conjunction into differentiable losses using product t-norm.","core_discovery":"The central discovery is that logic constraints can be enforced more strongly by feeding them into the graph itself. Co-referenced events are connected by event-event edges; event pairs that share an event are connected by event pair-event pair edges, which express symmetry and conjunction; and event-pair nodes are bridged to their two constituent events. Relational graph transformer layers propagate information over this heterogeneous graph, with an edge-type scalar modulating attention, producing enhanced event and event pair embeddings. Two additional losses derived from the same constraints via product t-norm push predictions toward symmetry and conjunctive consistency. The paper reports that this combination outperforms prior graph-based and logic-constrained baselines, including a gain of 5.4 F1 over the strongest previous coherence-constrained method on MATRES and 3.0 F1 in subevent relation extraction on MAVEN-ERE.","pith_inferences":["Because the SRE experiments on HiEve and MAVEN-ERE use ground-truth coreference annotations to build event-event edges, a fair end-to-end comparison would require predicted coreference; the paper does not report such a setting, so the SRE gains may not transfer to fully automatic pipelines.","The conjunction table effectively defines a transitivity system; a direct test would be whether the model's predictions on longer event chains respect the closure properties, which the paper does not measure.","A control experiment using random event-event edges instead of coreference edges could separate the benefit of the coreference signal itself from the benefit of extra graph connectivity.","The edge-type scalar attention bias is a minimal relational encoding; learning richer edge-type embeddings, or allowing multiple relations per event pair, is a natural extension the paper leaves open."],"forward_implications":["If correct, LogicERE shows that logical constraints can be implemented as graph structure, producing predictions that are more coherent than soft-loss regularization alone.","The method reaches state-of-the-art temporal relation scores on MATRES and TCR without dependency parsing or external ontologies, matching or beating systems that use them.","On MAVEN-ERE, joint training of temporal and subevent relations yields larger gains than split training, indicating the logic constraints transfer information across the two tasks.","The ablation results attribute the largest drops to removing event pair-event pair edges and to removing the joint logic learning objective, pinpointing where the graph reasoning carries the benefit."],"supporting_citations":[{"why":"Supplies the constrained-learning framework that LogicERE contrasts with and uses as a key baseline on MATRES and HiEve.","marker":"Wang et al. (2020)"},{"why":"Provides the logic-driven framework and product t-norm relaxation that the joint logic learning module adapts into differentiable losses.","marker":"Li et al. (2019)"},{"why":"Introduces the event-pair centered graph idea and the event-marking encoder trick that LogicERE builds on for its event pair nodes.","marker":"Chen et al. (2022)"},{"why":"Supplies the relational graph transformer architecture used for high-order reasoning over the heterogeneous LCG.","marker":"Bi et al. (2024)"},{"why":"A constraint-based baseline using probabilistic box embeddings that LogicERE compares against and outperforms.","marker":"Hwang et al. (2022)"},{"why":"A strong syntax-dependent graph baseline called SDLG that LogicERE beats without using external syntactic tools.","marker":"Zhuang, Fei, and Hu (2023b)"},{"why":"Defines the MAVEN-ERE dataset with joint temporal and subevent annotations, including the gold coreference used to build event-event edges.","marker":"Wang et al. (2022)"},{"why":"Provides the HiEve corpus used for subevent relation extraction evaluation.","marker":"Glavas et al. (2014)"}],"fun_headline_variants":["Graph-encoded logic constraints boost event relation accuracy","Reasoning with logic rules lifts event-relation extraction","High-order reasoning with logic-induced graphs improves ERE","Logic rules wired into graph raise event relation F1","Event graphs that enforce logic constraints top benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes it is legitimate to give the model ground-truth event coreference annotations as input edges when testing on HiEve and MAVEN-ERE; if those gold edges are not available at inference time, the reported subevent-relation gains shrink and the comparison with fully end-to-end baselines is no longer apples-to-apples.","fun_headline_variants_meta":{"raw":{"variants":["Graph-encoded logic constraints boost event relation accuracy","Reasoning with logic rules lifts event-relation extraction","High-order reasoning with logic-induced graphs improves ERE","Logic rules wired into graph raise event relation F1","Event graphs that enforce logic constraints top benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1346,"prompt_tokens":985,"completion_tokens":361,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":288}},"tokens_in":601,"tokens_out":361,"duration_ms":2958,"temperature":1.0,"reasoning_tokens":288,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:01:15.707993+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the HiEve and MAVEN-ERE SRE evaluations with coreference edges predicted by an independent coreference model (or with event-event edges removed) and check whether the F1 gains over SDLG and GraphERE persist; if the advantage collapses, the reported subevent improvements depend on gold coreference leakage rather than on the logic constraints.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the constrained-learning framework that LogicERE contrasts with and uses as a key baseline on MATRES and HiEve."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the event-pair centered graph idea and the event-marking encoder trick that LogicERE builds on for its event pair nodes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A constraint-based baseline using probabilistic box embeddings that LogicERE compares against and outperforms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the HiEve corpus used for subevent relation extraction evaluation."}],"review_version":1}