Pith. sign in

REVIEW 3 major objections 6 minor 25 references

EventSum: A Large-Scale Event-Centric Summarization Dataset for Chinese Multi-News Documents

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EventSum provides the first large-scale Chinese dataset for summarizing evolving news events, with 5,100 events and 57,984 news articles, and shows that even strong long-context LLMs miss key event facts.

desk verdict Useful new Chinese MDS dataset with a real metric-validation problem: Temporal Recall rests on auto-generated labels and an NLI model trained on the same pipeline. read the letter →

arxiv 2412.11814 v2 pith:FH4KVWFV submitted 2024-12-16 cs.CL

Mengna Zhu , Kaisheng Zeng , Mao Wang , Kaiming Xiao , Lei Hou , Hongbin Huang , Juanzi Li This is my paper · ORCID
classification cs.CL
keywords event-centricsummarizationmulti-documentChinesedataseteventunderstandingevaluationrecallmetricsnaturallanguageinferencelong-contextLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces the Event-Centric Multi-Document Summarization (ECS) task: given multiple news documents about one evolving event, produce a concise, chronological summary that captures the key sub-events, arguments, causal links, and temporal order. To support the task, the authors built EventSum, the first large-scale Chinese multi-document summarization dataset, with 5,100 events and 57,984 news articles (an average of 11.4 documents and 13,471 characters per event), using Baidu Baike entries as seeds and official news sites as input sources. Because standard metrics like ROUGE and BERTScore cannot tell whether a summary actually contains the event's key facts, the paper designs four recall metrics—Event, Argument, Causal, and Temporal Recall—computed by trained textual-entailment models. Experiments with ten long-context LLMs show the task remains hard: even the best models score low on recall, especially for sub-events, and the designed metrics expose gaps that lexical overlap misses.

What carries the argument

The methodological core is a set of recall metrics computed through natural language inference—models that judge whether a statement can be inferred from a text. For each annotated key element (a sub-event sentence, an argument, a causal relation, or a temporal relation), a trained NLI model decides whether the element is entailed by the generated summary, and recall is the fraction of elements entailed. The NLI models are trained on data built from existing event datasets (CMNEE for events and arguments, a translated MAVEN-ERE for causal relations, EventSum's own auto-annotated temporal data) plus automatically generated negative examples via remove, revise, and replace strategies. The multi-stage human annotation pipeline for the test set—sub-event and argument annotation, summary writing, global merging, and causal annotation—is the other load-bearing component, because the recall metrics assume the annotated elements are complete and correct.

What would settle it

Independently re-annotate a random sample of the test set with a fresh team and compare the annotated key elements; if the overlap is substantially below the self-reported 93% pass rate, the recall metrics are partly measuring annotation noise. Alternatively, feed a summary that deliberately restates every argument but omits all causal relations, and check whether Causal Recall drops to zero while ROUGE stays high.

Watch

Extended reading notes

Core claim

EventSum is the first large-scale Chinese multi-document summarization dataset centered on dynamic events, spanning 2000–2024 with 5,100 events and 57,984 news documents. The test set is manually annotated through a multi-stage pipeline: annotators mark sub-events and arguments (time, location, person, organization), write per-document summaries, merge these into a global chronological summary, and annotate causal relations between sub-events as cause or precondition; temporal relations are auto-annotated with an LLM. The authors claim this makes EventSum a demanding benchmark: long-context LLMs, including GPT-4o and Claude-3-Opus, perform far better on lexical/semantic similarity than on the four designed recall metrics, indicating that generated summaries often omit key event information even when they read fluently.

Load-bearing premise

The whole evaluation rests on the assumption that the manually annotated test set is complete and accurate—that annotators caught every key sub-event, argument, and relation—so the recall metrics measure summary quality rather than annotation gaps.

Editorial extensions

If this is right

  • EventSum provides a reusable benchmark for evaluating whether summarization models truly track event structure rather than surface similarity.
  • The four recall metrics give a concrete way to measure information completeness in event summaries, complementing ROUGE and BERTScore.
  • The dataset's size and temporal span (2000–2024, events up to 31+ days) make it a resource for timeline summarization and event-centric reasoning research.
  • The consistent finding that longer input contexts improve lexical metrics but not necessarily event recall points to a specific weakness in current long-context LLMs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The recall metrics could be adapted to other languages and domains by retraining the NLI discriminators on analogous event datasets; the paper only demonstrates them for Chinese.
  • The human-annotated global summaries, keyed to sub-events and relations, could support supervised models for event graph generation, not just summarization.
  • Because the test set is manually written and the input documents are public news, the dataset may serve as a controlled probe for measuring how much event information models of different scale and context length actually retain.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces EventSum, a Chinese multi-document summarization dataset focused on dynamic events, containing 5,100 events and 57,984 news documents. The test set is manually annotated with sub-events, arguments, causal relations, and human-written summaries, while temporal relations are automatically generated by glm-4-9b. The authors propose four recall metrics (Event, Argument, Causal, and Temporal Recall) implemented with NLI models and evaluate ten long-context LLMs. The main empirical claims are that the event-centric summarization task is challenging for current LLMs and that the designed recall metrics are crucial for evaluating summary comprehensiveness.

Significance. The dataset contribution is potentially significant: EventSum is, to my knowledge, the first large-scale Chinese multi-document summarization dataset with event-centric annotations, and the manual annotation of the test set is a credible effort to mitigate data leakage. The NLI-based recall metrics are a sensible way to go beyond ROUGE and BERTScore when evaluating content coverage. However, the paper's second central claim—that all four recall metrics are reliable and important—rests on validation that has a serious gap for the Temporal Recall component. The dataset itself is not undermined, but the evaluation layer needs repair before the metric claims can be accepted.

major comments (3)
  1. [Dataset Construction (Temporal Relation Annotation; Human Annotation)] The test instance i=[D, r', T, G] includes temporal relations T that were automatically annotated by glm-4-9b from the original Baidu Baike reference r, before r was replaced by the human-written r'. The Human Annotation section describes manual annotation of sub-events, arguments, summaries, global information, and causal relations, but not temporal relations. The automatic temporal annotation has only an 83% qualified rate. Therefore, the gold T in the test set may be inconsistent with both r' and D, and up to 17% of the temporal labels may be wrong. This affects the TR column in Table 2 and the Temporal Recall consistency in Table 3, so the claim that Temporal Recall is a reliable metric is not currently supported. The authors should manually verify or correct T on the test set, or clearly report the extent to which the auto-generated T matches r'.
  2. [Evaluation Metrics (Designed Recall Metrics); Table 3] The NLI model for Temporal Recall is trained on 'the automatically constructed data of EventSum' using the same automatically annotated temporal relationships. The paper does not state whether test-set instances and their T labels were excluded from this training data. If they were not excluded, the discriminator has already seen the gold labels it is later asked to verify, making the 92.9% consistency in Table 3 potentially inflated by label leakage. Even with a clean split, both the gold labels and the NLI training data are generated by the same glm-4-9b pipeline, so the high agreement may reflect shared systematic bias rather than genuine entailment ability. The authors must clarify the split and, ideally, re-validate Temporal Recall on a held-out set with human-annotated temporal labels.
  3. [Metric Evaluation and Table 3] The reliability assessment of the four recall metrics is based on 50 sampled instances per metric and on a single model (Claude-3-Opus). Given that the paper concludes the designed metrics are 'crucial' for evaluating comprehensiveness, this evidence is thin. The authors should provide a larger validation sample, compute confidence intervals, or include multiple models / annotators to show that the consistency scores are stable and not idiosyncratic to one setting.
minor comments (6)
  1. [Overall Results] The text says 'Tabel 2' instead of 'Table 2'.
  2. [Tables 2, 5, and 6] The model name 'InterLM2.5-7B-Chat-1M' should be 'InternLM2.5-7B-Chat-1M'.
  3. [Evaluation Metrics] There is a typo 'T emporal Recall' in the paragraph introducing the designed recall metrics; it should read 'Temporal Recall'.
  4. [Related Work] In the discussion of GameWikiSum, 'game vedio' should be 'game video'.
  5. [Dataset Construction (Quality Control)] The 'qualified rate is 83%' for temporal annotation is reported without the sample size used to compute it; please state how many temporal relationships were reviewed.
  6. [Data Analysis / Table 1] The table note for word/character counts should clarify whether ROUGE scores are computed on Chinese characters or on tokenized words, since this affects comparability of the numbers reported in Table 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the dataset and recall metrics are not defined in terms of their own outputs; the temporal-recall training overlap is a data-leakage caveat, not a by-construction reduction.

full rationale

The derivation chain is self-contained. The dataset is constructed from Baidu Baike entries and news retrieval, and the central claims—task difficulty and metric importance—are evaluated on held-out LLM outputs using Eq. (1)-(2), where e is taken from the annotated reference and s from the generated summary. The NLI discriminators are trained on external event datasets (CMNEE, translated MAVEN-ERE) plus EventSum's automatically constructed data; this is a standard train-a-classifier step, not a fitted parameter renamed as a prediction. The only self-referential element is Temporal Recall: the test instance keeps the automatically annotated T (derived from the original r, not the human r'), and the Temporal Recall NLI model was trained on "automatically constructed data of EventSum" with the same glm-4-9b temporal-relation annotation. That is a data-validity and potential-leakage caveat—the paper does not state that the NLI training data excludes the test instances or their T labels—but it is not a by-construction equivalence: T is an input premise to the entailment check, not the predicted output, and Table 3 checks the discriminators against human judgments. The 83% qualified rate for LLM temporal annotation is an acknowledged quality limitation, not circularity. Self-citations such as CMNEE provide external training data and are not invoked as proof of task novelty or metric necessity. No circular step is established.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The free parameters are design choices that shape dataset composition and the metric training data. The axioms are domain assumptions about the reliability of Baidu Baike as a source, the completeness of retrieved news, and the transferability of NLI models. No invented physical or formal entities are introduced.

free parameters (4)
  • document relevance similarity threshold = 0.5
    Used to filter retrieved news documents against Baidu Baike summaries during automatic data construction; chosen by hand and directly determines which documents enter each instance.
  • retrieved news articles per event = 20
    Bing News Search API is used to retrieve 20 articles published within a month of the event date; this cap bounds the input pool.
  • input document count bound = 5-20
    The number of input documents is controlled between 5 and 20 after cleaning, a design choice affecting the input length distribution.
  • NLI negative instance Remove threshold = 0.5
    Sentences with similarity above 0.5 to the key element are removed to construct negative instances for NLI training, shaping the discriminators.
assumptions (4)
  • domain assumption Baidu Baike entries provide reliable reference summaries and a sufficient source of event-related news articles for event-centric summarization.
    The entire dataset is built on Baidu Baike entries and their descriptions as pseudo-summaries; if these entries are not representative of dynamic events, the dataset is skewed.
  • domain assumption The automatically retrieved news documents, after similarity filtering, cover the key information of each event.
    Automatic data construction relies on Bing News Search and a 0.5 similarity threshold; this assumes retrieved documents are relevant and complete enough.
  • domain assumption NLI models trained on CMNEE, a translated MAVEN-ERE, and EventSum's automatic temporal annotations can reliably judge whether key event elements are entailed in generated summaries.
    The four recall metrics depend on these NLI discriminators; their transfer to the EventSum test distribution is validated only on 50 instances.
  • ad hoc to paper The three negative-instance generation strategies (Remove, Revise, Replace) produce valid non-entailment examples for training the NLI models.
    The negative data are synthesized by LLM-guided modifications; there is no external validation that they match true non-entailment in real summaries.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EventSum: A Large-Scale Event-Centric Summarization Dataset for Chinese Multi-News Documents." pith.science (2026). https://pith.science/paper/FH4KVWFV

@misc{pith2026241211814,
  author       = {Pith},
  title        = {Pith review of: EventSum: A Large-Scale Event-Centric Summarization Dataset for Chinese Multi-News Documents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FH4KVWFV}},
  note         = {Machine review of arXiv:2412.11814}
}
read the original abstract

In real life, many dynamic events, such as major disasters and large-scale sports events, evolve continuously over time. Obtaining an overview of these events can help people quickly understand the situation and respond more effectively. This is challenging because the key information of the event is often scattered across multiple documents, involving complex event knowledge understanding and reasoning, which is under-explored in previous work. Therefore, we proposed the Event-Centric Multi-Document Summarization (ECS) task, which aims to generate concise and comprehensive summaries of a given event based on multiple related news documents. Based on this, we constructed the EventSum dataset, which was constructed using Baidu Baike entries and underwent extensive human annotation, to facilitate relevant research. It is the first large scale Chinese multi-document summarization dataset, containing 5,100 events and a total of 57,984 news documents, with an average of 11.4 input news documents and 13,471 characters per event. To ensure data quality and mitigate potential data leakage, we adopted a multi-stage annotation approach for manually labeling the test set. Given the complexity of event-related information, existing metrics struggle to comprehensively assess the quality of generated summaries. We designed specific metrics including Event Recall, Argument Recall, Causal Recall, and Temporal Recall along with corresponding calculation methods for evaluation. We conducted comprehensive experiments on EventSum to evaluate the performance of advanced long-context Large Language Models (LLMs) on this task. Our experimental results indicate that: 1) The event-centric multi-document summarization task remains challenging for existing long-context LLMs; 2) The recall metrics we designed are crucial for evaluating the comprehensiveness of the summary information.

Figures

Figures reproduced from arXiv: 2412.11814 by the authors.

Figure 1
Figure 1. An example of the “2023 Hebei Heavy Rain”. The [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the construction process. It introduces the data construction process for the “2023 Hebei Heavy Rain” [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Analysis of Reference Summary. The distribution [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Analysis of the impact of various input documents [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Most of the events selected in the dataset occurred [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Prompt used for LLMs to rephrase the structured event information into coherent natural language sentences to obtain [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Prompt used for LLMs to revise key event information to obtain negative instances of NLI models. The translated [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Prompt for LLMs to generate summaries according to the input documents. The translated version is under the original [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Cases for generated summaries analysis. Each column represents a corresponding example of its specific type. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 20 canonical work pages

  1. [1]

    Argument Recall t2 represents the key event argument, and the label is used to indicate whether the argument is en- tailed in the input text t1

    The label was set to “0”. Argument Recall t2 represents the key event argument, and the label is used to indicate whether the argument is en- tailed in the input text t1. Considering that arguments may include abbreviations, pronouns, etc., to more accurately evaluate argument recall, positive instances consists of two parts: 1) For ordinary ar- guments n...

    work page

  2. [2]

    The provided demonstrations introduce a substantial amount of information unrelated to the target dynamic event, which poses a new challenge for the model to more carefully discern and extract valid information

    work page

  3. [3]

    ACM Comput

    Multi-document Summarization via Deep Learning Techniques: A Survey. ACM Comput. Surv., 55(5). Nenkova, A.; Passonneau, R.; and McKeown, K. 2007. The Pyramid Method: Incorporating human content selection variation in summarization evaluation. ACM Trans. Speech Lang. Process., 4(2): 4–es. Oliveira, A. S.; and Costa, A. H. R. 2021. PLSUM: Gen- erating PT-BR...

    arXiv 2007

  4. [4]

    要求摘要中能够包含其中的关键事件信息及相关 的时序、因果关系等信息。

    work page

  5. [5]

    要求输出只包含摘要信息，没有其他任何无关的 信息和提示的信息，输入输出的格式都只包含一个字符 串且应该根据输入文本生成纯中文摘要文本信息。请你 不要输出无关信息且输出不能是空字符串，一定要确保 有文本输出，如果实在无法生成摘要，请输出'无'，但字 符串一定不能为空。

    work page

  6. [6]

    I will provide you with several news reports related to a specific thematic event, and I would like you to generate a summary focusing on the thematic event

    输出摘要时尽可能使用输入文本中的表达。 You are a linguistics expert, proficient in text summarization tasks. I will provide you with several news reports related to a specific thematic event, and I would like you to generate a summary focusing on the thematic event. The specific requirements are as follows:

    work page

  7. [8]

    Yan, R.; Kong, L.; Huang, C.; Wan, X.; Li, X.; and Zhang, Y

    Online: Association for Computational Linguistics. Yan, R.; Kong, L.; Huang, C.; Wan, X.; Li, X.; and Zhang, Y . 2011. Timeline Generation through Evolutionary Trans- Temporal Summarization. In Barzilay, R.; and Johnson, M., eds., Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing , 433–443. Edin- burgh, Scotland, UK.: ...

    arXiv 2011

  8. [9]

    The significant increase in input length poses a greater challenge to the long-context processing capabilities of LLMs, which has been validated in Figure 5(a)

    work page

Show all 25 references
  1. [10]

    role": "system

    The summarized text exhibits greater stylistic diversity, and the inclusion of demonstrations partially constrains the style and content of the generated text. We plan to conduct a more in-depth and detailed analysis in our future work. E. Cases for Generated Summaries Analysi...

  2. [11]

    }, { "role

    <句子> ... 需要严格按照给定的格式输出。 下面我将给你三个例子，你的输出需要按照同样的格式。" }, { "role": "user", "content": "输入文本：周三早上，美国佛罗里达号核潜艇反应 堆附近起火，有四人受到轻伤，但没有造成重大损失。 火焰是在上午 10点多工作人员在核潜艇反应堆舱上方施工切割潜艇外壳时燃着的。官 方称起火大约10分钟后就被扑灭了，核潜艇上的设备未被损坏。受伤的 4人，包括一名佛罗里达号船员被送到朴茨茅斯海军医疗中心救治。起 火的原因正在调查之中。 佛罗里达号潜艇的反应堆已经关闭了2个多月。 该潜艇正在朴...

  3. [12]

    }, …… ] [ {

    周三早上，美国佛罗里达号核潜艇起火，有四人受到轻伤，但 没有造成重大损失。" }, …… ] [ { "role": "system", "content": "You are a linguistics expert specializing in text rewriting tasks. I will provide you with a piece of text along with the event information it contains, and I would like you to help me rewrite the...

  4. [13]

    role": "system

    On Wednesday morning, a fire broke out on the USS Florida nuclear submarine, causing minor injuries to four people, but no significant damage was reported." }, …… ] Figure 7: Prompt used for LLMs to rephrase the structured event information into coherent natural language sente...

  5. [14]

    }, { "role

    关注其中的时间、地点、人名、数量和其他数字信息，在文本 中随机修改，当有数值出现时，优先修改数值。2. 理解句子，对句子 进行扩写或者删除一些关键信息。具体的任务要求如下： A. 确保每条数据中都至少改动一处信息。 B. 请直接输出修改后的文本字符串，不要输出其他任何无关的信 息。 下面我将给你三个例子，你的输出需要严格按照同样的格式。" }, { "role": "user", "content": "输入文本：5月22日当地时间早上7:50左右，美国 华盛顿号航母发生火灾，导致1人一级烧伤。" }, { "role": "assistant",...

  6. [15]

    }, { "role

    Focus on altering the time, location, names, quantities, and other numerical information within the text. When numerical values appear, prioritize modifying them. 2. Understand the sentences and either expand them or delete some key information. The specific task requirements ...

  7. [16]

    输入的多段文本只有一个主题事件，找出主题事 件，围绕其中的主题事件为多段文本生成摘要。

  8. [20]

    The summary must be generated solely based on the provided documents

  9. [21]

    Identify the thematic event and generate a summary for the multiple documents centered around this thematic event

    The input consists of multiple documents with only one thematic event. Identify the thematic event and generate a summary for the multiple documents centered around this thematic event

  10. [22]

    The summary must be organized in chronological order

  11. [23]

    The summary should include key event information and related temporal and causal relationships

  12. [24]

    Both the input and output should be formatted as a single string, and the summary should be a pure Chinese text generated based on the input documents

    The output should contain only the summary information, without any unrelated information. Both the input and output should be formatted as a single string, and the summary should be a pure Chinese text generated based on the input documents. Please do not output unrelated inf...

  13. [25]

    Thurman猫一杯

    Use expressions from the input documents in the summary whenever possible. Figure 9: Prompt for LLMs to generate summaries according to the input documents. The translated version is under the original prompt in the figure. Issue Incomplete or missing information Over or Under...

  14. [2005]

    https: //catalog.ldc.upenn.edu/LDC2006T06

    ACE 2005 Multilingual Training Corpus. https: //catalog.ldc.upenn.edu/LDC2006T06. Chu, E.; and Liu, P. J. 2019a. MeanSum: A Neural Model for Unsupervised Multi-document Abstractive Summarization. arXiv:1810.05739. Chu, E.; and Liu, P. J. 2019b. MeanSum: A Neural Model for Unsu...

    2005 arXiv

  15. [2017]

    Large scale opinion mining for social, news and blog data. J. Syst. Softw., 127(C): 237–248. Urologin, S. 2018. Sentiment analysis, visualization and classification of summarized news articles: a novel ap- proach. International Journal of Advanced Computer Sci- ence and Applic...

    2018 arXiv

  16. [2020]

    1” indicates that t2 is entailed in t1, while a label of “0

    OpenReview.net. Zhang, Y .; Er, M. J.; and Zhao, R. 2015. Multi-document extractive summarization using window-based sentence rep- resentation. In 2015 IEEE Symposium Series on Computa- tional Intelligence, 404–410. IEEE. Zhang, Z.; Elfardy, H.; Dreyer, M.; Small, K.; Ji, H.; ...

    2015 arXiv

  17. [2022]

    Transactions of the As- sociation for Computational Linguistics, 10: 163–177

    SummaC: Re-visiting NLI-based models for incon- sistency detection in summarization. Transactions of the As- sociation for Computational Linguistics, 10: 163–177. Li, M.; Ma, T.; Yu, M.; Wu, L.; Gao, T.; Ji, H.; and McK- eown, K. 2021. Timeline Summarization based on Event Gra...

    2021 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.