{"id":"86e0efff-969a-43d4-a49a-550c319185bf","arxiv_id":"2411.08708","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Triggers help document-level event extraction mainly when events are dense and localized, and prompt-based models benefit from having trigger slots in the prompt even when the triggers are random.","lead":"This paper asks whether explicit trigger spans are needed for document-level event extraction, testing four models and four trigger sources on three datasets. The answer is conditional: triggers help most when documents contain many localized events, and prompt-based models benefit from trigger slots even when the triggers are random.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central attribution of trigger benefit to event density is confounded with argument localization across the three datasets; the paper concedes the missing many-events/dispersed-arguments cell, so the headline claim is underdetermined.","rationale":"I agree with the reader's weakest_assumption. The paper handles the missing cell honestly—it is noted in Section 4.2 and Result #5—but honesty about a confound does not remove it. The central claim in the abstract and Section 7.2 requires identifying the role of event density separately from argument localization; the three datasets cannot do that. The proposed test fills the missing cell synthetically. If the test shows triggers help under many-events/dispersed-arguments, the claim survives; otherwise it should be rephrased as primarily about localization (plus perhaps event density as a joint condition). This does not undermine the released annotations or the empirical ablations, and it does not change the reader's CONDITIONAL verdict. A secondary observation: the abstract's 'even random triggers' ICL claim is contradicted on CMNEE in Table 2 (GPT-4O: no-trigger .635 vs random .533), so that sentence should also be qualified; it is not the primary load-bearing point.","tokens_in":17237,"tokens_out":6493,"duration_ms":60662,"concrete_test":"Construct a many-events/dispersed-arguments condition by concatenating MUC test documents (each with roughly one event and ~40-token argument spread) into multi-event documents, preserving gold event templates and trigger annotations. Rerun DEGREE e2e and TANL pipeline with human triggers versus no triggers using the same 5-fold protocol. If trigger-conditioned models now outperform no-trigger models, event density is a sufficient driver; if they do not, argument localization is the confounded factor and the paper's event-density attribution should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core explanatory claim is that trigger usefulness depends on the typical number of events per document (abstract; Result #4). This attribution is not identifiable from the evidence. In Table 1, WikiEvents combines the high-event-density condition with tightly localized arguments (1-2 tokens apart), while MUC and CMNEE combine low density with dispersed or only moderately localized arguments. The two proposed causal factors never vary independently in the direction that matters. Section 7.2 Result #5 explicitly concedes: 'We unfortunately do not have a dataset with documents with many events with dispersed arguments.' The paper's own CMNEE results suggest localization may be the driving factor even at low density: pipelined TANL on CMNEE benefits from human/LLM/keyword triggers (54-64 F1 vs 43.8 no-trigger) whereas on MUC, similarly low density but dispersed arguments, it does not. Because the experiment matrix lacks the high-density/dispersed-arguments cell, the abstract's 'depends on dataset characteristics (i.e., typical number of events)' cannot be distinguished from a localization-only or interaction explanation. This is a correctable limitation, but it bears directly on the headline conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether explicit trigger-span extraction is necessary for document-level event extraction (DEE). It evaluates four sequence-to-sequence models (TANL, GTT, DEGREE, GENIE) in both pipeline and end-to-end configurations, with no triggers and with triggers of four quality levels (human, LLM, keyword, random), across three DEE datasets (MUC, WikiEvents, CMNEE), and compares them against GPT-4 and GPT-4-MINI in-context prompting baselines. The central claims are: (i) triggers are not needed when documents contain few events; (ii) triggers matter when documents contain many localized events, with higher-quality triggers preferred; (iii) lower-quality triggers often substitute for human annotation; (iv) the mere presence of trigger-like spans, even random ones, helps in-context learning; and (v) providing additional event schema information makes models robust to degraded trigger quality. The paper also releases MUC human trigger annotations, all trigger sets, and code.","tokens_in":17482,"tokens_out":2793,"duration_ms":25517,"significance":"If the central claims hold, the paper provides the first systematic, controlled comparison of trigger-based and trigger-free systems for document-level event extraction, challenging the field's default assumption that triggers are indispensable. The practical implications are concrete: for low-event-density corpora, trigger annotation can be skipped without loss; for ICL, even arbitrary trigger placeholders help; and rich event schema information can compensate for noisy triggers. The public release of MUC trigger annotations and the experimental code is a durable contribution. However, the paper's causal attribution of trigger usefulness to event density is not identifiable from the chosen datasets, because event density and argument localization are confounded across the three corpora. The empirical results themselves are real and reproducible in structure, but the headline explanation needs re-framing or additional evidence.","major_comments":[{"comment":"The claim that trigger usefulness depends on the typical number of events per document is confounded with argument localization. In Table 1, WikiEvents combines high event density (15.7 events/doc) with tightly localized arguments (1.6 tokens between arguments), while MUC and CMNEE combine low density with dispersed or moderately localized arguments (38.3 and 8.8 tokens, respectively). The paper explicitly states in §7.2 that it lacks a dataset with many events and dispersed arguments. As a result, the abstract's 'depends on dataset characteristics (i.e. the typical number of events per document)' cannot be distinguished from a localization-only or an interaction explanation. The CMNEE pipelined TANL results (43.8 F1 without triggers vs. 54.3-64.0 with human/LLM/keyword triggers) suggest localization may be the driving factor even at low density. The paper should either re-frame the conclusion as a correlation, or add a dataset (or a controlled manipulation) that separates the two factors.","section":"§4.2, §7.2 (Result #5), Table 1"},{"comment":"The paper reports mean F1 scores with standard deviations across 5-fold cross-validation but performs no statistical significance tests. Many of the central contrasts appear to be within one standard deviation (e.g., DEGREE e2e no-trigger vs. human-trigger on MUC: 59.7±0.00 vs. 56.6±0.01; TANL e2e no-trigger vs. human-trigger on CMNEE: 43.8±0.00 vs. 41.0±0.00). Since the headline claims rest on such comparisons, paired significance tests (or confidence intervals with a stated pairing structure) are necessary. Without them, the 'triggers help' vs. 'triggers not needed' dichotomies are not fully supported.","section":"§6, Table 2"},{"comment":"For MUC, the paper selects the earliest human-annotated trigger span after taking the union, and notes that 'this typically produced better extraction scores than using the trigger span most centrally located' among the event's arguments. This is a heuristic chosen based on the evaluation data, and it is a free parameter of the study. Because the choice systematically favors the better-scoring option, it risks overfitting the reported results. The paper should pre-specify the selection rule or report results for both early and central trigger choices so readers can assess sensitivity.","section":"§4.3, MUC trigger selection"}],"minor_comments":[{"comment":"The MUC annotation agreement is moderate (Krippendorff alpha 0.530, F-kappa 0.453), and the paper then takes the union of all annotators' triggers. The effect of this aggregation choice on downstream extraction performance is not analyzed or discussed.","section":"§4.3, MUC annotation"},{"comment":"The caption states 'Datasets cannot be distinguished in the figure.' Adding colors or marker shapes for the three datasets would substantially improve readability and allow the reader to verify the exceptions discussed in §7.1.","section":"Figure 3"},{"comment":"The phrase 'This result is almost a negation of the previous claim' is imprecise. The relationship between Result #4 and Result #5 should be stated explicitly: the direction of trigger benefit reverses, but only under the additional conditions of many events, localized arguments, and higher-quality triggers.","section":"§7.2, Result #5"},{"comment":"The WikiEvents column is missing all GTT results without explanation. If GTT was not run on WikiEvents, that should be stated in the text or table footnote; if it was run but failed, the failure mode should be described.","section":"Table 2"},{"comment":"The sentence 'In Table 3, we show the additional annotation guidelines we curated to trigger in conjecture with Walker et al., 2005' should read 'curated to trigger annotation in conjunction with Walker et al., 2005' or similar; the current phrasing is ungrammatical.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid, useful empirical study, and the authors are transparent about the missing dataset cell. The main revision needed is to temper the causal attribution to event density and to add significance testing; both are addressable within the manuscript's scope. The dataset/code release is a strong point. I would not reject, because the core observation—triggers are not universally needed, and their benefit varies with corpus properties—is credible even if the specific factor cannot be isolated with the current three datasets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. This is the first systematic ablation of trigger usefulness for document-level event extraction, and it ships a genuinely useful new artifact: human trigger annotations for MUC. The empirical work is real and mostly careful. But the headline story—that trigger benefit depends on event density—is not actually identifiable from the three datasets used, because density and argument localization move together. The paper itself admits the missing cell. That's the thing to know.\n\nWhat's new and good. The paper compares human, LLM, keyword, and random triggers across TANL, GTT, DEGREE, GENIE, and GPT-4o prompting, in pipeline and end-to-end modes. That's a wide net, and the random-trigger control is a nice touch: it shows that in-context learning needs trigger-shaped spans even when the spans are meaningless, which is a real finding. The result that LLM/keyword triggers often match human triggers is practically valuable—it means annotation budgets can be cut. The robustness from templates and instructions (DEGREE) is also solid. The release of data and code is a plus.\n\nWhere it's soft. The central attribution is confounded. WikiEvents bundles many events per document with arguments 1–2 tokens apart; MUC and CMNEE bundle few events with dispersed or only moderately localized arguments. So Result #4 and #5 cannot separate \"many events\" from \"localized arguments.\" The paper's own Section 7.2 Result #5 concedes there is no many-events/dispersed-arguments dataset. Worse, the CMNEE data point cuts the other way: pipelined TANL benefits from triggers on CMNEE despite low event density, which suggests argument localization may be the real driver. The abstract overstates the case by saying \"typical number of events\"; the conclusion phrase \"many localized events\" is more honest. Second, there's no significance testing; with five-fold CV and overlapping standard deviations, several claims rest on differences of a couple of points. Third, the MUC trigger selection heuristic (choose the earliest annotated span) was chosen because it performed best, which introduces a mild selection-on-validation effect. Fourth, CMNEE is machine-translated and filtered to ~60% of documents, so those numbers carry translation noise. None of these are fatal; the first one just needs rephrasing, an explicit \"this is a hypothesis\" label, and ideally an extra dataset.\n\nWho it's for. Anyone building document-level extraction systems or deciding whether to annotate triggers. It's a measurement paper, not an architecture paper, and that's fine.\n\nRecommendation: send it to peer review. Require the authors to revise the abstract and conclusions to acknowledge the confound, add significance testing or at least cautious language, and label the density-specific claim as a hypothesis. With those changes this becomes a solid contribution.","headline":"Useful empirical study that shows triggers are sometimes optional for document-level event extraction, but the central causal claim about event density is underdetermined by the confounded dataset design; it deserves peer review with required revisions.","tokens_in":17975,"tokens_out":3222,"would_cite":true,"duration_ms":27988,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that explicit trigger spans are not required for document-level event extraction; their value depends on event density and argument localization.","keywords":["event extraction","document-level event extraction","event triggers","argument extraction","in-context learning","sequence-to-sequence models","LLM prompting","event schemas"],"falsifier":"Run the same trigger-vs-no-trigger comparison on a dataset with many events per document but arguments far apart from any trigger (for instance, by taking WikiEvents and artificially shuffling or relocating argument mentions away from their triggers). If trigger-free systems still match trigger-conditioned ones, the paper's density explanation fails and argument localization is the driver; if trigger-based systems prevail, the density explanation is supported.","tokens_in":17055,"feed_emoji":"🎯","tokens_out":6993,"duration_ms":55254,"temperature":0.7,"pith_summary":"Document-level event extraction asks a system to fill event templates from a whole text, where a document can describe several events with arguments scattered across sentences. This paper asks whether the sentence-level convention of first identifying a trigger span — a word or phrase that names the event — carries over to this harder setting. Across three datasets and several sequence-to-sequence systems, the authors show that triggers are beneficial only under specific conditions: documents with many events whose arguments sit close to the trigger. For documents with few, spread-out events, trigger-free systems match or beat trigger-conditioned pipelines, and supplying a natural-language event template makes performance resilient to trigger quality. Lower-cost triggers from LLMs or keyword lists can substitute for human annotation, and in-context LLM prompting is helped by the mere presence of trigger spans, even random ones.","feed_headline":"Triggers: optional for event extraction except in dense documents","feed_subtitle":"Skip human trigger annotation for sparse-event corpora; even random triggers help LLM prompts.","key_machinery":"The central object is the trigger span, defined (following Walker et al., 2005) as a verb, noun, or adjective that most clearly expresses an event's occurrence; the paper treats triggers as 'rationales' that can be supplied or predicted. The machinery is a comparative experimental design that varies trigger quality (human, LLM, keyword, random) and architecture (pipeline vs end-to-end, with and without triggers) across three datasets, plus a prompting baseline for GPT-4O and GPT-4O-Mini. This design isolates whether the trigger's content, its mere presence, or the surrounding schema information carries the performance difference.","core_discovery":"The central claim is that trigger spans are a conditional rather than integral component of document-level event extraction. On WikiEvents, where each document contains roughly 15-19 events and arguments are 1-2 tokens apart, human-annotated triggers yield clear gains over trigger-free systems, and pipeline models that can anchor on exact trigger positions do best. On MUC and CMNEE, which average about one or two events per document with arguments tens of tokens apart, removing triggers costs little or nothing: DEGREE performs best without triggers at all. The paper further claims that the event schema's natural-language description (template, type definition, example triggers) can substitute for trigger quality, and that in-context LLM prompting benefits from any trigger spans in demonstrations because the demonstrations instill the trigger concept even when the spans are random.","pith_inferences":["The paper's central attribution (event density drives trigger usefulness) is confounded with argument localization across its three datasets; building or synthesizing a dataset with many events and dispersed arguments would settle whether triggers help because events are numerous or because arguments are close to a lexical anchor.","The random-trigger prompting result suggests that even non-content marker spans (e.g., bracketed labels like [EVENT]) might be inserted into demonstrations to teach the trigger concept without any real trigger vocabulary, which could be tested directly.","The paper's design implies a practical decision rule for new datasets: measure events-per-document and argument dispersion first, then decide whether to budget for trigger annotation; this rule could be packaged as a simple diagnostic for dataset collections.","If the schema-information result transfers, then future work on low-resource event extraction could avoid trigger annotation entirely by investing instead in writing clear event templates, turning annotation cost from lexical spotting to schema writing."],"forward_implications":["Dataset creators for document-level event extraction can skip costly human trigger annotation when documents are event-sparse, provided they supply a natural-language event schema.","For event-dense corpora with localized arguments, human or generated triggers remain useful, but LLM- or keyword-produced triggers match human annotation in most settings.","Prompt-based LLM extraction should always include trigger spans in few-shot demonstrations, since random spans already improve $F_1$ by several points on WikiEvents and others.","End-to-end systems are the simpler default choice: they match pipeline performance except when argument localization makes pipeline anchoring helpful."],"supporting_citations":[{"why":"Supplies the MUC dataset, one of the three evaluation corpora, and defines the document-level template-filling task.","marker":"Sundheim, 1991"},{"why":"Supplies the WikiEvents dataset and the GENIE argument-extraction model used in the pipeline experiments.","marker":"Li et al., 2021"},{"why":"Supplies the CMNEE dataset, the Chinese military-news corpus used as a third evaluation setting.","marker":"Zhu et al., 2024"},{"why":"Provides TANL, the pipeline model that performs trigger detection followed by argument extraction.","marker":"Paolini et al., 2021"},{"why":"Provides GTT, the end-to-end template-filling model that does not rely on triggers by default.","marker":"Du et al., 2021"},{"why":"Provides DEGREE, the end-to-end model that takes event templates and natural-language instructions, showing robustness to trigger quality.","marker":"Hsu et al., 2022"},{"why":"Provides the template-matching alignment and scoring algorithm used to evaluate multi-event predictions.","marker":"Das et al., 2022"},{"why":"Supports the finding that in-context demonstrations can be effective even when the provided labels are wrong or random, which the paper cites for its random-trigger prompt result.","marker":"Min et al., 2022"}],"fun_headline_variants":["Triggers: only essential for dense event documents","Skip trigger annotation for sparse event corpora","Random triggers still help LLM event extraction","Trigger-free extraction works for sparse event docs","Event triggers: conditional, not integral"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper attributes the trigger benefit to the typical number of events per document, but across its three datasets event density and argument localization vary together, and there is no many-events-but-dispersed-arguments dataset to separate the two causes.","fun_headline_variants_meta":{"raw":{"variants":["Triggers: only essential for dense event documents","Skip trigger annotation for sparse event corpora","Random triggers still help LLM event extraction","Trigger-free extraction works for sparse event docs","Event triggers: conditional, not integral"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000824,"raw_usage":{"total_tokens":3581,"prompt_tokens":897,"completion_tokens":2684,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":2618}},"tokens_in":513,"tokens_out":2684,"duration_ms":20065,"temperature":1.0,"reasoning_tokens":2618,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:25:41.728979+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same trigger-vs-no-trigger comparison on a dataset with many events per document but arguments far apart from any trigger (for instance, by taking WikiEvents and artificially shuffling or relocating argument mentions away from their triggers). If trigger-free systems still match trigger-conditioned ones, the paper's density explanation fails and argument localization is the driver; if trigger-based systems prevail, the density explanation is supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CMNEE dataset, the Chinese military-news corpus used as a third evaluation setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides TANL, the pipeline model that performs trigger detection followed by argument extraction."}],"review_version":1}