{"id":"39d9e90c-1eb4-48de-89fd-b51bbe211744","arxiv_id":"2411.10371","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A taxonomy and benchmark review of event causality identification, covering sentence-level, document-level, multilingual, and LLM-based methods.","lead":"This paper surveys how computers spot cause-and-effect links between events in text, sorting sentence-level and document-level methods into a taxonomy. It also collects or runs benchmark scores on four datasets so researchers can compare approaches.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's top rankings are built on non-comparable evaluations: Dr. ECIΔ and CPATTΔ used distinct data processing (App.","rationale":"The reader's weakest assumption identifies non-comparable evaluation protocols for Dr. ECI and CPATT as the critical weakness in the paper's quantitative comparison. My independent reading of Appendix A.1 and the result tables confirms this is the most load-bearing issue: the claimed contribution is a 'rigorous performance evaluation' of ECI methods on four benchmarks, and the two models marked as using distinct data processing are the ones presented as best on three of those benchmarks. No other concern is as consequential. The 'first comprehensive taxonomy' claim is a novelty assertion that could be debated, but it does not undermine the correctness of the survey's content; the dev-set-as-test choice for MAVEN-ERE is a comparability issue as well, but it affects a broader set of models and is explicitly stated, whereas the Δ discrepancy is concentrated on the top-ranked results and lacks any description of what differed. My recommendation is unchanged from the reader's CONDITIONAL verdict: the benchmark section needs revision or re-execution under a unified protocol before its rankings can be used as a shared performance baseline.","tokens_in":44471,"tokens_out":3189,"duration_ms":34034,"concrete_test":"Re-run Dr. ECI and CPATT on CTB and ESL using exactly the same candidate-pair construction, negative-sampling ratio, intra/inter-sentence split, and metric aggregation as the non-Δ baselines in Tables 8 and 9, then recompute the category averages. If Dr. ECIΔ's CTB F1 drops from 82.2 toward the next-best non-Δ value (75.4 or lower), or CPATTΔ's ESL intra-sentence F1 falls from 80.4 toward the 65-70 range, the headline benchmark ranking is an artifact of distinct preprocessing rather than model capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's quantitative comparison is one of its two headline contributions, but Appendix A.1 concedes that Dr. ECI and CPATT 'employed distinct data processing methods compared to other baselines' and marks them with Δ. These two models are precisely the ones highlighted as top performers: Dr. ECIΔ reaches 82.2 F1 on CTB versus 75.4 for the next best (DiffusECI), 84.1 on ESL intra-sentence versus 71.4, and 84.1 on MAVEN-ERE intra-sentence versus 64.2; CPATTΔ is second on CTB and ESL. The gap between Dr. ECI (13.0 F1 on CTB) and Dr. ECIΔ (82.2 F1) is far too large to be explained by prompting or reasoning differences alone; it implies a change in the candidate pair set, negative sampling, filtering, or test subset. Because the processing difference is not specified, the reported rankings conflate evaluation choices with model quality. The taxonomy may still be useful, but the assessment half of the paper's central claim is not supported for the top-scoring entries, and the Abstract's mention of 'rigorous performance evaluations' overstates what the appendix actually documents.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a survey of Event Causality Identification (ECI) in NLP. It formalizes ECI, reviews benchmark datasets and evaluation metrics, and proposes a taxonomy that separates Sentence-level ECI (SECI) from Document-level ECI (DECI), with subcategories for feature-pattern matching, machine-learning classification, deep semantic encoding, prompt-based fine-tuning, causal knowledge pre-training, data augmentation, graph reasoning, and LLM-based methods. It also contains a qualitative assessment in Tables 6 and 7, a quantitative comparison on CTB, ESL, MAVEN-ERE, and MECI in Appendix A, and a discussion of future directions and applications. The paper claims two headline contributions: the first comprehensive taxonomy for ECI and a rigorous quantitative performance evaluation of existing methods.","tokens_in":44834,"tokens_out":5012,"duration_ms":50136,"significance":"If the taxonomy is adopted, it gives the community a useful structured map of ECI methods, and the survey's coverage of roughly a decade of work, including recent LLM-based approaches, is a service to the field. The qualitative assessment in Section 7.1 and the detailed experimental appendix also provide a starting point for benchmarking, and the authors have made an organized dataset repository publicly available. Those strengths are real. However, the quantitative assessment is the part of the paper that is most exposed, and the benchmark's comparability problems are load-bearing for the 'rigorous performance evaluations' claim in the abstract. The taxonomy and qualitative discussion can stand, but the numerical rankings need to be repaired or substantially re-scoped.","major_comments":[{"comment":"The footnote states that Dr. ECI and CPATT 'employed distinct data processing methods compared to other baselines' and marks them with Δ. This is a direct admission that the top-performing entries in the benchmark were not evaluated under the same protocol as the rest. In Tables 8, 9, and 10, Dr. ECIΔ is reported as the best or second-best method on CTB, ESL, and MAVEN-ERE, and CPATTΔ is also highlighted in the CTB and ESL rankings. The gap between Dr. ECI (13.0 F1 on CTB) and Dr. ECIΔ (82.2 F1) is far too large to be explained by prompting or reasoning differences alone; it implies a change in candidate pairs, negative sampling, filtering, or test subset. Because the processing difference is not specified, the rankings in Section A.2 conflate evaluation choices with model quality, and the claim of 'rigorous performance evaluations' in the abstract is not supported for the top-scoring entries. The authors should either re-run Dr. ECI and CPATT under the exact common protocol used for other baselines, or explicitly present their results as non-comparable and remove them from the ranked headlines and from statements such as 'Dr. ECIΔ achieved the highest F1 score'.","section":"Appendix A.1, footnote 8"},{"comment":"The benchmark mixes numbers that were directly cited from original papers with numbers that the authors reproduced, using an asterisk marker for reproduced results. This is a reasonable practice in principle, but the tables do not carry a source column, so the reader cannot always tell which numbers come from which pipeline. More importantly, the footnote about Δ shows that at least two entries do not follow the common protocol, and the text does not specify what 'distinct data processing' means. Since the paper's comparative conclusion depends on all models being evaluated on the same candidate pair set and the same data splits, the authors should add a protocol/source column to every results table and specify the exact preprocessing used by Dr. ECI and CPATT, or exclude them from the comparison.","section":"Appendix A.1, Implementation Details"},{"comment":"All reported F1 values are point estimates without standard deviations, confidence intervals, or significance tests. For setups like 5-fold cross-validation on ESL and 10-fold cross-validation on CTB, differences of a few F1 points can easily be within run-to-run or fold-to-fold variance. The paper uses these point estimates to discuss which methods are 'top performers' and to compare broad categories of methods. Without variance information, the quantitative assessment is weaker than the text implies, and the reader cannot determine whether the reported ordering is stable. Adding per-model standard deviations or at least a statement about the number of runs would materially strengthen the benchmark.","section":"Section 7.2 and Tables 8–10"}],"minor_comments":[{"comment":"The opening sentence, 'Seem to models for SECI, prompt-based fine-tuning methods DECI methods use PLMs...', is ungrammatical and should be revised to something like 'Similar to SECI, prompt-based fine-tuning DECI methods use PLMs...'.","section":"Section 5.3"},{"comment":"The header 'ESC' appears to be a typo for 'ESL'; the surrounding text and earlier sections consistently refer to the Event StoryLine Corpus as ESL.","section":"Table 12"},{"comment":"The caption contains the misspelling 'corss MECI dataset' and should read 'cross MECI dataset'.","section":"Figure 12"},{"comment":"The word 'halluciantion' should be corrected to 'hallucination'.","section":"Section 8.7"},{"comment":"The reference list contains duplicate entries: [122] and [123] are both O'Gorman, Wright-Bettner, and Palmer 2016, and [194] and [195] are both Zhao et al. 2024. These should be consolidated.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The taxonomy and qualitative analysis are valuable and the survey is competently done, but the quantitative benchmark is currently not trustworthy for the specific claims it is used to support. The stress-test concern about Dr. ECIΔ and CPATTΔ is well founded. I would encourage the editor to ask for a revision in which the benchmark is either re-run under a uniform protocol or re-scoped so that non-comparable results are not presented as ranked performance comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you need a structured map of event causality identification. The SECI/DECI taxonomy is genuinely useful, and the qualitative assessment tables (implicit causality, annotation demands, generalization, external knowledge, false positives) are a good organizing device. The authors also release processed datasets and run a broad cross-dataset comparison. That is real work.\n\nThe soft spot is the benchmark. Appendix A.1 says Dr. ECI and CPATT used distinct data processing methods (the two Δ entries), and those are exactly the top-scoring systems. Dr. ECI's own reported score is 13.0 F1 on CTB; Dr. ECIΔ is 82.2. That gap is too large to come from prompting or reasoning alone — the candidate pair set or test subset must have changed, and the paper never says how. The rankings therefore conflate evaluation choices with model quality. The abstract's 'rigorous performance evaluations' overstates what the appendix documents. This is the main issue and it is fixable: rerun those systems under the common protocol, or clearly separate them from the comparable results and caveat the headline numbers.\n\nSmaller issues: single F1 values without variance, and reproduced (*) vs. cited numbers sit side by side without much discussion. These are minor compared to the Δ problem.\n\nThe 'first comprehensive taxonomy' claim is a bit strong; earlier surveys covered causality extraction and event relation extraction. But the SECI/DECI split and the method grouping are a legitimate synthesis, and the survey is more current than the alternatives.\n\nBottom line: the taxonomy and qualitative assessment carry the paper. The quantitative assessment, as reported, does not support the top rankings. The paper deserves peer review because the taxonomy is a reference contribution, but it needs a substantial revision of the benchmark section before publication. I'd cite the taxonomy; I would not cite the numbers.","headline":"Useful SECI/DECI taxonomy and qualitative assessment, but the benchmark's top rankings rest on two models evaluated under a different protocol.","tokens_in":45141,"tokens_out":1853,"would_cite":true,"duration_ms":17669,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes the first comprehensive taxonomy for event causality identification, splitting the field into sentence-level and document-level tasks, and benchmarks established model families on four datasets to show where each…","keywords":["event causality identification","taxonomy","survey","sentence-level ECI","document-level ECI","large language models","causal hallucination","information extraction"],"falsifier":"Re-run Dr. ECI and CPATT on CTB, ESL, MAVEN-ERE, and MECI under the exact preprocessing and data splits applied to the other reproduced baselines; if either model's F1 drops below the best non-flagged baseline, the survey's comparative ranking is refuted.","tokens_in":44320,"feed_emoji":"🧭","tokens_out":10680,"duration_ms":90674,"temperature":0.7,"pith_summary":"The paper is a survey, and its central claim is that the field of event causality identification (ECI) can be organized by a single taxonomy: sentence-level ECI (SECI) and document-level ECI (DECI), each with a small set of technical families. It also claims to give the first quantitative comparison of these families on four benchmark datasets, using precision, recall, and F1 for both intra- and inter-sentence causal pairs. If right, researchers gain a shared map of the design space and a reference ranking that shows which approaches balance precision and recall best. The main empirical finding is a trade-off: prompt-based fine-tuning and external-knowledge methods lead in balanced performance, while plain LLM prompting gets high recall but many false positives, which the paper calls causal hallucination.","feed_headline":"New taxonomy organizes event causality research and ranks its models","feed_subtitle":"Sentence- and document-level methods face off on four benchmarks; prompt-based and knowledge-enhanced systems lead.","key_machinery":"The central object is the taxonomy itself, and the mechanism that carries the argument is event-pair classification: for a text with event mentions, each model must label every pair of events as cause, caused_by, or none. The survey describes all reviewed methods as different ways of producing event-pair embeddings — through patterns, engineered features, PLM encoders, prompt templates, graph aggregation, or LLM reasoning — and then compares those embeddings with a classifier. The four-dataset evaluation protocol is the instrument that turns the taxonomy into a ranking, reporting F1 separately for sentence-internal and cross-sentence pairs.","core_discovery":"On its own terms, the paper's discovery is a map plus a measurement. Every ECI method reviewed is placed in a taxonomy with two branches — SECI and DECI — and within each branch a few technical families: feature pattern matching, machine-learning classification, deep semantic encoding, prompt-based fine-tuning, and causal knowledge pre-training for SECI; deep encoding, event-graph reasoning, prompt fine-tuning, LLM-based reasoning, and multi-lingual transfer for DECI. The measurement is a four-dataset comparison reporting precision, recall, and F1, separated into intra-sentence and inter-sentence causality, plus a multi-lingual benchmark. The consistent result is that prompt-based and knowledge-enhanced methods balance precision and recall, while simple LLM methods show high recall with low precision, a pattern attributed to causal hallucination.","pith_inferences":["If the comparability caveat is resolved and the ranking survives, the natural next step the authors do not take is a public leaderboard with a single, audited preprocessing pipeline; that would turn the one-time comparison into a living benchmark.","The taxonomy implies a recipe for hybrid models — combine event-graph reasoning with LLM consistency checking — that the survey itself does not test; a controlled comparison of such hybrids against the current top systems would be a direct test of the survey's map.","Because the four benchmark datasets are news- or encyclopedia-style, the ranking may not transfer to finance or healthcare; re-running the same protocol on domain corpora would show whether the taxonomy's families keep their relative advantages.","The survey's 'first comprehensive taxonomy' claim could be checked by applying the taxonomy to methods published after the survey's cutoff; a method that fits no family would mark the taxonomy as incomplete rather than wrong."],"forward_implications":["A shared taxonomy lets future ECI papers position new methods in one family and compare against a common set of baselines rather than launching new task formulations.","Prompt-based fine-tuning and knowledge-enhanced encoding should be treated as the strong defaults for SECI, since they lead in balanced F1 and in suppression of false positives.","Document-level causality remains the harder task, with consistently lower inter-sentence F1 across families, so graph reasoning and prompt-based methods that aggregate global context are the promising direction.","For LLM-based ECI, the practical implication is to add consistency checks, external knowledge, or decomposed reasoning before deployment, because plain prompting with a strong LLM produces too many false positives.","Multilingual ECI is not solved by English-centric methods: low-resource languages in the MECI benchmark lag even for strong LLMs, so cross-lingual transfer needs dedicated alignment."],"supporting_citations":[{"why":"Supplies the EventStoryLine corpus, one of the four benchmark datasets and the main inter-sentence causality test.","marker":"[21]"},{"why":"Supplies Causal-TimeBank, the compact intra-sentence benchmark used for the CTB evaluation.","marker":"[114]"},{"why":"Supplies MAVEN-ERE, the large-scale document benchmark used to compare recent SECI and DECI models.","marker":"[159]"},{"why":"Supplies MECI, the multilingual dataset used to assess cross-lingual and multi-lingual ECI.","marker":"[87]"},{"why":"Dr. ECI is the top-scoring LLM-enhanced system in the comparison and one of the two models flagged for non-standard data processing.","marker":"[17]"},{"why":"CPATT is the top-scoring prompt-based DECI system and the other model flagged for non-standard data processing, so the comparability caveat applies to the headline ranking.","marker":"[187]"}],"fun_headline_variants":["Survey maps event causality models, ranks them on four datasets","Taxonomy plus benchmarks: event causality methods face off","Event causality: new taxonomy, four-dataset model shootout","Causal event detection surveyed: taxonomy and head-to-head tests","How event causality models stack up: a taxonomy and benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark rankings assume every model was evaluated under the same data-processing and splitting protocol, but the appendix states that Dr. ECI and CPATT, two of the top scorers, used distinct data processing, so the headline ordering can reflect evaluation differences rather than model quality alone.","fun_headline_variants_meta":{"raw":{"variants":["Survey maps event causality models, ranks them on four datasets","Taxonomy plus benchmarks: event causality methods face off","Event causality: new taxonomy, four-dataset model shootout","Causal event detection surveyed: taxonomy and head-to-head tests","How event causality models stack up: a taxonomy and benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1241,"prompt_tokens":931,"completion_tokens":310,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":228}},"tokens_in":547,"tokens_out":310,"duration_ms":3477,"temperature":1.0,"reasoning_tokens":228,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:40:44.883966+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Dr. ECI and CPATT on CTB, ESL, MAVEN-ERE, and MECI under the exact preprocessing and data splits applied to the other reproduced baselines; if either model's F1 drops below the best non-flagged baseline, the survey's comparative ranking is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies MAVEN-ERE, the large-scale document benchmark used to compare recent SECI and DECI models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CPATT is the top-scoring prompt-based DECI system and the other model flagged for non-standard data processing, so the comparability caveat applies to the headline ranking."}],"review_version":1}