{"id":"e4d28cf5-82d2-4cca-a552-8cbc56391840","arxiv_id":"2412.11814","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"EventSum provides the first large-scale Chinese multi-document summarization dataset focused on dynamic events, with manually annotated test summaries and structured sub-events, arguments, causal relations, and temporal relations.","lead":"This paper introduces EventSum, a large Chinese dataset of 5,100 news events with over 57,000 news articles for event-centric multi-document summarization. It also proposes four recall metrics based on natural language inference to measure whether generated summaries cover key event information.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temporal Recall's gold labels are auto-generated by glm-4-9b, not human-annotated, and its NLI model may be trained on the same automatic pipeline, weakening the metric-validation claim.","rationale":"The reader's weakest assumption was that the manually annotated test set is complete and accurate, including temporal relations. My concern is more specific: the paper's own construction description shows that temporal relations in the test set are not manually annotated at all; they are produced by glm-4-9b during automatic data construction and are not regenerated from the human-written reference summary. This is an internal inconsistency that directly affects the Temporal Recall metric. Additionally, the NLI model used to compute Temporal Recall is trained on the same automatically constructed EventSum data, creating a plausible route for label leakage if the train/test split is not carefully enforced. This does not invalidate the dataset itself, and the other three recall metrics rest on manual annotations, so the overall conditional verdict remains appropriate. However, the authors should verify the NLI training split and provide human-annotated temporal relations for a validation sample before the metric suite is adopted as a benchmark standard. The proposed concrete test would settle whether the Temporal Recall numbers are trustworthy.","tokens_in":22470,"tokens_out":8771,"duration_ms":84056,"concrete_test":"Download the released data and code; check the construction script for the Temporal Recall NLI training set. (1) If any EventSum test instances or their automatically annotated T relations appear in the NLI training data, retrain the Temporal Recall NLI model after removing all test-split instances and recompute Table 2's TR column and Table 3's Temporal Recall row. (2) Independently, take a random sample of 100 test events and have two annotators manually annotate temporal relations directly from the human-written reference summaries r', then compute agreement with the existing T labels. If the retrained TR scores shift materially or human agreement with T is low (e.g., below 80%), the Temporal Recall metric and the conclusion that the designed metrics are crucial need revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's second major claim is that the four designed recall metrics are crucial for evaluating event-centric summaries. The Temporal Recall component rests on two weak links. First, in 'Dataset Construction' the paper states that temporal relationships were annotated automatically by glm-4-9b during Automatic Data Construction, with only an 83% qualified rate; the subsequent 'Human Annotation' section describes manual annotation of sub-events, arguments, summaries, global information, and causal relations, but not temporal relations. The test instance i=[D, r', T, G] therefore keeps the automatically produced T, which was derived from the original Baidu Baike reference r, not from the human-written r'. Second, 'Designed Recall Metrics' says the NLI model for Temporal Recall was trained on the automatically constructed EventSum data with these same automatically annotated temporal relationships; if that training data includes any test-set instances or their T labels, the discriminator has already seen the gold labels it is later asked to verify. Because Temporal Recall is one of the four metrics used in Table 2 and Table 3 to support the conclusion that the recall metrics are reliable and important, this makes the metric-validation claim vulnerable: the high consistency in Table 3 could partly reflect label leakage or a mismatched gold standard. The dataset contribution itself is not undermined, but the benchmark's evaluation layer needs repair.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EventSum, a Chinese multi-document summarization dataset focused on dynamic events, containing 5,100 events and 57,984 news documents. The test set is manually annotated with sub-events, arguments, causal relations, and human-written summaries, while temporal relations are automatically generated by glm-4-9b. The authors propose four recall metrics (Event, Argument, Causal, and Temporal Recall) implemented with NLI models and evaluate ten long-context LLMs. The main empirical claims are that the event-centric summarization task is challenging for current LLMs and that the designed recall metrics are crucial for evaluating summary comprehensiveness.","tokens_in":22725,"tokens_out":5106,"duration_ms":48313,"significance":"The dataset contribution is potentially significant: EventSum is, to my knowledge, the first large-scale Chinese multi-document summarization dataset with event-centric annotations, and the manual annotation of the test set is a credible effort to mitigate data leakage. The NLI-based recall metrics are a sensible way to go beyond ROUGE and BERTScore when evaluating content coverage. However, the paper's second central claim—that all four recall metrics are reliable and important—rests on validation that has a serious gap for the Temporal Recall component. The dataset itself is not undermined, but the evaluation layer needs repair before the metric claims can be accepted.","major_comments":[{"comment":"The test instance i=[D, r', T, G] includes temporal relations T that were automatically annotated by glm-4-9b from the original Baidu Baike reference r, before r was replaced by the human-written r'. The Human Annotation section describes manual annotation of sub-events, arguments, summaries, global information, and causal relations, but not temporal relations. The automatic temporal annotation has only an 83% qualified rate. Therefore, the gold T in the test set may be inconsistent with both r' and D, and up to 17% of the temporal labels may be wrong. This affects the TR column in Table 2 and the Temporal Recall consistency in Table 3, so the claim that Temporal Recall is a reliable metric is not currently supported. The authors should manually verify or correct T on the test set, or clearly report the extent to which the auto-generated T matches r'.","section":"Dataset Construction (Temporal Relation Annotation; Human Annotation)"},{"comment":"The NLI model for Temporal Recall is trained on 'the automatically constructed data of EventSum' using the same automatically annotated temporal relationships. The paper does not state whether test-set instances and their T labels were excluded from this training data. If they were not excluded, the discriminator has already seen the gold labels it is later asked to verify, making the 92.9% consistency in Table 3 potentially inflated by label leakage. Even with a clean split, both the gold labels and the NLI training data are generated by the same glm-4-9b pipeline, so the high agreement may reflect shared systematic bias rather than genuine entailment ability. The authors must clarify the split and, ideally, re-validate Temporal Recall on a held-out set with human-annotated temporal labels.","section":"Evaluation Metrics (Designed Recall Metrics); Table 3"},{"comment":"The reliability assessment of the four recall metrics is based on 50 sampled instances per metric and on a single model (Claude-3-Opus). Given that the paper concludes the designed metrics are 'crucial' for evaluating comprehensiveness, this evidence is thin. The authors should provide a larger validation sample, compute confidence intervals, or include multiple models / annotators to show that the consistency scores are stable and not idiosyncratic to one setting.","section":"Metric Evaluation and Table 3"}],"minor_comments":[{"comment":"The text says 'Tabel 2' instead of 'Table 2'.","section":"Overall Results"},{"comment":"The model name 'InterLM2.5-7B-Chat-1M' should be 'InternLM2.5-7B-Chat-1M'.","section":"Tables 2, 5, and 6"},{"comment":"There is a typo 'T emporal Recall' in the paragraph introducing the designed recall metrics; it should read 'Temporal Recall'.","section":"Evaluation Metrics"},{"comment":"In the discussion of GameWikiSum, 'game vedio' should be 'game video'.","section":"Related Work"},{"comment":"The 'qualified rate is 83%' for temporal annotation is reported without the sample size used to compute it; please state how many temporal relationships were reviewed.","section":"Dataset Construction (Quality Control)"},{"comment":"The table note for word/character counts should clarify whether ROUGE scores are computed on Chinese characters or on tokenized words, since this affects comparability of the numbers reported in Table 2.","section":"Data Analysis / Table 1"}],"recommendation":"major_revision","confidential_remarks":"The dataset and its manual test set are likely a useful resource for the community. The main risk is not the dataset itself but the validation of the evaluation metrics, particularly Temporal Recall. If the authors can either manually annotate the test-set temporal relations or demonstrate that the auto-generated T is carefully aligned with the human-written reference summaries and exclude all test-set instances from NLI training, the claims can be supported. I would also encourage the authors to be more explicit about the sample size of the reliability check in Table 3, as the current 50-instance evaluation is a weak basis for the 'crucial metrics' conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The EventSum dataset is a genuine resource. It is the first large-scale Chinese multi-document summarization dataset, with 5,100 events, 57,984 news documents, and an average input length of 13,471 characters per instance, far beyond Multi-News and DUC/TAC. The multi-stage human annotation of the test set—sub-events, arguments, causal relations, global summaries—is described concretely, and quality-control pass rates are reported. The LLM experiments are solid and support the claim that this task is challenging. The designed recall metrics are a sensible response to the limits of ROUGE and BERTScore for event-centric evaluation, and the Event and Argument Recall components are reasonably constructed.\n\nThe soft spot is Temporal Recall, and it is load-bearing. Temporal relationships were automatically annotated by glm-4-9b during automatic data construction, with an 83% qualified rate, and the human annotation step did not re-annotate them. The test instance keeps that auto-generated T. Then the NLI model for Temporal Recall is trained on the same automatically constructed EventSum data with those same temporal annotations. In other words, the discriminator is trained on the very labels it is later asked to verify. The reported consistency between NLI and human judgment for Temporal Recall (92.9% on 50 instances) does not validate the gold labels—at best it says the trained NLI matches human judgment on a small sample, but the gold T is not human-verified. This directly weakens the second headline claim that the recall metrics are crucial, at least for the temporal component.\n\nOther issues are minor by comparison. The 'first Chinese MDS dataset' claim would be stronger with a broader survey of prior Chinese MDS work. The metric-validation sample of 50 instances is small and lacks confidence intervals. The NLI training for Event and Argument Recall overlaps with the authors' own CMNEE, but that is disclosed and not a serious problem.\n\nWho gets value: people building Chinese MDS resources, event understanding researchers, and anyone designing factuality-style recall metrics. The dataset contribution stands and deserves to be used, but the evaluation layer needs repair. I would send this to peer review, with a clear request to fix Temporal Recall—either re-annotate temporal relations on the test set with human annotators, or drop Temporal Recall from the headline claims until it is validated against human-verified labels.","headline":"Useful new Chinese MDS dataset with a real metric-validation problem: Temporal Recall rests on auto-generated labels and an NLI model trained on the same pipeline.","tokens_in":23296,"tokens_out":1938,"would_cite":true,"duration_ms":18110,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EventSum provides the first large-scale Chinese dataset for summarizing evolving news events, with 5,100 events and 57,984 news articles, and shows that even strong long-context LLMs miss key event facts.","keywords":["event-centric summarization","multi-document summarization","Chinese dataset","event understanding","summarization evaluation","recall metrics","natural language inference","long-context LLM"],"falsifier":"Independently re-annotate a random sample of the test set with a fresh team and compare the annotated key elements; if the overlap is substantially below the self-reported 93% pass rate, the recall metrics are partly measuring annotation noise. Alternatively, feed a summary that deliberately restates every argument but omits all causal relations, and check whether Causal Recall drops to zero while ROUGE stays high.","tokens_in":22234,"feed_emoji":"📰","tokens_out":4627,"duration_ms":36071,"temperature":0.7,"pith_summary":"This paper introduces the Event-Centric Multi-Document Summarization (ECS) task: given multiple news documents about one evolving event, produce a concise, chronological summary that captures the key sub-events, arguments, causal links, and temporal order. To support the task, the authors built EventSum, the first large-scale Chinese multi-document summarization dataset, with 5,100 events and 57,984 news articles (an average of 11.4 documents and 13,471 characters per event), using Baidu Baike entries as seeds and official news sites as input sources. Because standard metrics like ROUGE and BERTScore cannot tell whether a summary actually contains the event's key facts, the paper designs four recall metrics—Event, Argument, Causal, and Temporal Recall—computed by trained textual-entailment models. Experiments with ten long-context LLMs show the task remains hard: even the best models score low on recall, especially for sub-events, and the designed metrics expose gaps that lexical overlap misses.","feed_headline":"5,100 events test whether AI can summarize evolving news","feed_subtitle":"A new Chinese benchmark and recall metrics show long-context LLMs still miss key event facts.","key_machinery":"The methodological core is a set of recall metrics computed through natural language inference—models that judge whether a statement can be inferred from a text. For each annotated key element (a sub-event sentence, an argument, a causal relation, or a temporal relation), a trained NLI model decides whether the element is entailed by the generated summary, and recall is the fraction of elements entailed. The NLI models are trained on data built from existing event datasets (CMNEE for events and arguments, a translated MAVEN-ERE for causal relations, EventSum's own auto-annotated temporal data) plus automatically generated negative examples via remove, revise, and replace strategies. The multi-stage human annotation pipeline for the test set—sub-event and argument annotation, summary writing, global merging, and causal annotation—is the other load-bearing component, because the recall metrics assume the annotated elements are complete and correct.","core_discovery":"EventSum is the first large-scale Chinese multi-document summarization dataset centered on dynamic events, spanning 2000–2024 with 5,100 events and 57,984 news documents. The test set is manually annotated through a multi-stage pipeline: annotators mark sub-events and arguments (time, location, person, organization), write per-document summaries, merge these into a global chronological summary, and annotate causal relations between sub-events as cause or precondition; temporal relations are auto-annotated with an LLM. The authors claim this makes EventSum a demanding benchmark: long-context LLMs, including GPT-4o and Claude-3-Opus, perform far better on lexical/semantic similarity than on the four designed recall metrics, indicating that generated summaries often omit key event information even when they read fluently.","pith_inferences":["The recall metrics could be adapted to other languages and domains by retraining the NLI discriminators on analogous event datasets; the paper only demonstrates them for Chinese.","The human-annotated global summaries, keyed to sub-events and relations, could support supervised models for event graph generation, not just summarization.","Because the test set is manually written and the input documents are public news, the dataset may serve as a controlled probe for measuring how much event information models of different scale and context length actually retain."],"forward_implications":["EventSum provides a reusable benchmark for evaluating whether summarization models truly track event structure rather than surface similarity.","The four recall metrics give a concrete way to measure information completeness in event summaries, complementing ROUGE and BERTScore.","The dataset's size and temporal span (2000–2024, events up to 31+ days) make it a resource for timeline summarization and event-centric reasoning research.","The consistent finding that longer input contexts improve lexical metrics but not necessarily event recall points to a specific weakness in current long-context LLMs."],"supporting_citations":[{"why":"Multi-News is the primary existing large-scale multi-document summarization dataset that EventSum is compared against for scale and input length.","marker":"(Fabbri et al. 2019)"},{"why":"ACE 2005 provides the widely adopted definitions of events and arguments used in the annotation guidelines.","marker":"(Christopher et al. 2005)"},{"why":"MAVEN-ERE supplies the annotation style for causal relations and its translated version is used to build the training data for the Causal Recall NLI model.","marker":"(Wang et al. 2022)"},{"why":"CMNEE is the Chinese event extraction dataset used to construct the training data for the Event Recall and Argument Recall NLI models.","marker":"(Zhu et al. 2024)"},{"why":"chinese-roberta-wwm-ext is the base language model fine-tuned to build the NLI discriminators for all four recall metrics.","marker":"(Cui et al. 2019)"},{"why":"The Recognising Textual Entailment challenge defines the entailment relation that the recall metrics rely on to decide whether an annotated element is covered by a summary.","marker":"(Dagan, Glickman, and Magnini 2005)"},{"why":"ROUGE provides the standard lexical overlap metrics used as the common baseline in the experiments.","marker":"(Lin 2004)"},{"why":"BERTScore provides the standard semantic similarity metric used alongside ROUGE to compare against the designed recall metrics.","marker":"(Zhang et al. 2020)"}],"fun_headline_variants":["EventSum: LLMs omit key facts in event news summaries","5,100 events chart LLM blind spots in news summarization","AI missed key event details on EventSum news benchmark","Benchmark: LLMs recall only part of evolving event info","LLMs miss core event facts in Chinese multi-news summaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation rests on the assumption that the manually annotated test set is complete and accurate—that annotators caught every key sub-event, argument, and relation—so the recall metrics measure summary quality rather than annotation gaps.","fun_headline_variants_meta":{"raw":{"variants":["EventSum: LLMs omit key facts in event news summaries","5,100 events chart LLM blind spots in news summarization","AI missed key event details on EventSum news benchmark","Benchmark: LLMs recall only part of evolving event info","LLMs miss core event facts in Chinese multi-news summaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000925,"raw_usage":{"total_tokens":4004,"prompt_tokens":1024,"completion_tokens":2980,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":2895}},"tokens_in":640,"tokens_out":2980,"duration_ms":19320,"temperature":1.0,"reasoning_tokens":2895,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:32:34.726072+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently re-annotate a random sample of the test set with a fresh team and compare the annotated key elements; if the overlap is substantially below the self-reported 93% pass rate, the recall metrics are partly measuring annotation noise. Alternatively, feed a summary that deliberately restates every argument but omits all causal relations, and check whether Causal Recall drops to zero while ROUGE stays high.","supporting_citations":[],"review_version":1}