Pith. sign in

REVIEW 3 major objections 5 minor 39 references

Providing event-organized structural cues measurably improves LLM performance on clustering and temporal QA in massive document streams, but current models still fail at tracking entity states and recency.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:57 UTC pith:LRVROURX

load-bearing objection StreamBench is a genuinely useful new benchmark, but the Temporal QA result is compromised by answer leakage and unfiltered future cues; the clustering and summarization findings are the solid parts. the 3 major comments →

arxiv 2603.19250 v2 pith:LRVROURX submitted 2026-02-25 cs.CL

Can Structural Cues Save LLMs? Evaluating Language Models in Massive Document Streams

Yukyung Lee , Yebin Lim , Woojun Jung , Wonjun Choi , Susik Yoon This is my paper
classification cs.CL
keywords Document Stream MiningLarge Language ModelEvaluationBenchmarkTemporal ReasoningStructural CuesTopic ClusteringStreaming QA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces StreamBench, a benchmark of 605 real-world news events and 15,354 documents, to test LLMs when many concurrent stories are mixed in one stream. It argues that a large part of LLM failure in this setting is organizational: models cannot separate which facts belong to which event or find the most recent update. To test this, the paper provides 'structural cues'—event cards listing People, Location, Result, and Event Attributes—and measures the difference in performance. Cues improve topic clustering by up to 4.37 B3 F1 points and temporal QA by up to 9.63 accuracy points, with smaller gains in summarization. The remaining gap shows that tracking current entity states and judging recency is still an unsolved temporal-reasoning problem, even with perfect organization.

Core claim

The central claim is that structural cues reorganize existing information by event and, by doing so, help LLMs locate relevant facts and separate distinct events in a mixed document stream. The paper reports that these cues lift clustering performance (reducing over-clustering) and temporal QA accuracy (especially for counting and temporal-ordering questions), while leaving summarization largely unchanged. Crucially, the gains are not uniform: questions that require comparing events over time—who currently holds a role, what is the latest update—show little or negative cue benefit. The authors conclude that organization is an addressable bottleneck, but temporal reasoning over located inform

What carries the argument

Structural cues: for each event, a small structured card (People, Location, Result, Event Attributes) extracted from the source articles and verified by humans, appended to the model's input. The paper measures their effect via Δorg = performance with cues minus performance without, and Δgap = remaining distance to ceiling. The cue ablation and the comparison with RAG and serialized facts show that the gain comes from event-level organization, not from fact access alone.

Load-bearing premise

The load-bearing premise is that the structural cues are faithful, complete, and available before the model reads the documents, and that the ground-truth answers in Temporal QA are independent of those cues; if the cues leak answer-bearing content or are unavailable in a true stream, then Δorg measures oracle access rather than a general organizational benefit.

What would settle it

Create a Temporal QA split where the correct answer is not present in any cue field (e.g., requires summing counts across events or inferring a state from raw documents only), run the same raw-vs-cued comparison, and check whether the Δorg gain disappears; alternatively, build cues incrementally from the stream itself rather than from the full event record and see if the gain remains. If either drops the gain to zero, the paper's attribution to 'organization' would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, organization is a measurable, addressable bottleneck: grouping documents by event before inference should raise clustering and QA accuracy, with the largest gains for small models and dense streams.
  • Retrieval alone is not sufficient: the paper's comparison shows RAG barely helped, while event-structured cues gave the largest gain, pointing to structure rather than selection as the active ingredient.
  • Summarization will need more than event cards: gains there are small and mostly in coverage and faithfulness, not coherence or relevance.
  • Temporal reasoning is separable: even with oracle organization, entity-state tracking and recency judgments stay error-prone, so progress on temporal awareness is a distinct requirement for streaming LLMs.
  • The benchmark itself, with 605 events and 15,354 documents, provides a reusable testbed for future streaming-evaluation work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's cues are constructed offline from the complete event record; a natural extension is to test whether incrementally built cues (from the stream only) preserve the gain, and whether a learned component that updates a knowledge graph or table as documents arrive can close part of the Δgap.
  • Because answers to the Temporal QA task are selected from the cue fields themselves, the measured Δorg may partly reflect oracle access; a stricter test with answers only in raw documents (or requiring multi-event synthesis) is needed to separate 'organization helps' from 'cues contain the answer.'
  • The pattern that small models benefit most from cues suggests a scaling story: as models grow, organization becomes less of a bottleneck; this implies the practical recipe 'structure the input' may be a cheap substitute for model capacity in streaming settings.
  • If the finding generalizes, a testable prediction is that any structured pre-grouping of a stream (by topic, by event, by entity timeline) should transfer beyond news, such as to social media or financial filings, wherever interleaved events occur.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces StreamBench, a news-stream benchmark with 605 events and 15,354 documents spanning 2016 and 2025, and evaluates seven instruction-tuned LLMs (1B–123B) on topic clustering, temporal QA, and multi-topic summarization. The central diagnostic is a comparison between a raw condition, where models see a sliding 7-day window of documents, and a cued condition, where the same window is supplemented by structural cues (People, Location, Result, Event Attributes) extracted per event. The main empirical claim is that structural cues improve topic clustering (up to +4.37 B3 F1) and temporal QA (up to +9.63 accuracy) by helping models locate relevant information and separate events, while summarization gains are small and temporal reasoning over located information remains difficult. The paper reports Wilcoxon tests, ablations over cue components and cue structure, and an LLM-as-judge summarization analysis.

Significance. If the central comparison is internally valid, the paper makes a useful contribution: it operationalizes two streaming-specific failure modes (intra-topic and inter-topic conflict), provides a reproducible benchmark, and shows that explicit event-level organization can partially offset these failures across model scales. The strengths include controlled document-volume sampling (k = 1,3,5,10), multiple model families and sizes, statistical testing, component ablations, and public release of code and data. The conclusion that organization is a measurable bottleneck but temporal reasoning remains hard is interesting and actionable. However, the temporal-QA result, which is one of the two headline gains, is currently confounded by two design issues: cues are not filtered by the question timestamp, and QA answers are selected from the very cue fields supplied in the treated condition. Until those are addressed, the headline claim about temporal QA should be treated as not yet established.

major comments (3)
  1. [§3.2.2 Eq. (5) vs. §3.3.1] The cued condition is not temporally aligned with the question timestamp. Eq. (5) restricts documents to t_{m,n} ≤ t_q, but the Cued Input definition in §3.3.1 provides cues {s_{m,n}} for all events in the window, with no corresponding t_q filter. Because W_j is a 7-day window and t_q falls inside it, cues for events occurring after t_q can be included. Since cues are extracted offline from the complete event record (§4.3), the cued condition may contain outcome information from the future relative to the question, which the raw condition cannot access. This makes Δorg for Temporal QA a measure of oracle access, not only of organization. The §6.2 ablation does not control for this, because RAG, Serialized Facts, and w/ cue all use the same oracle cues. Please re-measure with cues filtered to t ≤ t_q, or explicitly state and demonstrate that such filtering was already applied.
  2. [§A.2 and §4.4] QA answers are selected from the same cue fields that are supplied in the treatment. §A.2 states: 'Answers are selected from structured cue fields associated with each event (e.g., Result, People, Location).' For Result Recognition questions, the Result cue is, by construction, a summary of the event's main outcome, i.e., the answer. For Entity Tracking, the People/Location cue lists entities that can contain the correct answer, and distractors are not guaranteed to be equally present in the cue. Thus the cued input does not merely reorganize neutral information; it embeds answer-bearing facts into the treated context. This is a circularity risk for the central temporal-QA claim. Please construct a control in which cues exclude answer-bearing fields, or in which QA answers and distractors are generated independently of cue content, and report how much of the +5.35 to +9.63 accuracy gain
  3. [§4.4 (Summarization references)] Reference summaries are produced by concatenating human-written event descriptions and then consolidating them with GPT-4o. The cued condition provides event-level cue summaries of these same events, so cued outputs may be systematically closer to the reference format. The paper does not test whether the small ROUGE-L gains and larger METEOR/CheckEval gains are partly an artifact of this reference-construction procedure. This is not as central as the QA issues, but it should be discussed and, ideally, controlled by reporting a human-written or independently constructed reference set on a subset.
minor comments (5)
  1. [§4.4] The Temporal QA annotation paragraph appears twice, with the second occurrence adding details but also repeating the 1,087 / 1,483 counts and the 623 / 464 breakdown. Please merge the two paragraphs and remove the duplication.
  2. [Tables 8–10] The column headers 'wo/ cue' are inconsistent with the body text's 'w/o cue'. Also, in Table 9, the Gemma-2-9B row (23–31%) is explained in Appendix B as a formatting failure; consider marking this row with a footnote in the table itself so readers do not misinterpret it as a genuine accuracy result.
  3. [§5.3.2] The sentence 'Mistral-Large 123B : 4.3% error' is missing a predicate and context. It appears to refer to current_state errors with cues, but as written it is unclear whether 4.3% is an error rate, a gap, or an improvement. Please restate.
  4. [§5.1.2] When inputs exceed the context window, documents are 'uniformly truncated while maintaining proportions across events.' Since the cued condition adds cue tokens to the same window, the truncation may remove more document tokens in the cued condition than in the raw condition. Please state whether the total input length or the document-token budget was matched across conditions.
  5. [§2/Related Work] The related-work discussion would benefit from a brief comparison with benchmarks that also provide structured or timeline-based context, such as HoH and the DICE-style long-context event benchmarks, to clarify the novelty of the diagnostic comparison rather than only the benchmark itself.

Circularity Check

2 steps flagged

Temporal QA Δ_org reduces to answer-in-cue leakage: gold answers are drawn from the cue fields the cued input provides, and the time filter of Eq. (5) is not applied to cues.

specific steps
  1. self definitional [Section 4.4 / A.2 (QA generation) with Section 3.3.1 (Cued Input)]
    "Answers are selected from structured cue fields associated with each event (e.g., Result, People, Location) and must be supported by the referenced articles [A.2]. Cued Input: the model receives documents in W_j along with structural cues {s_{m,n}} for all events in the window [3.3.1]."

    The gold answer for Temporal QA is, by construction, an element of the cue tuple (People, Location, Result), and the cued condition places that tuple directly in the prompt. The raw condition must extract the same fact from article text. Thus Δ_org = M(Cued) − M(Raw) measures the effect of printing the answer field in the input, not the effect of organizational structure. The QA 'prediction' is forced by the annotation protocol, which defines the answer from the same cue field that the treated input contains.

  2. other [Section 3.2.2 Eq. (5) vs. Section 3.3.1 (Cued Input); acknowledged in Section 7]
    "The model receives only documents up to that timestamp: W_j^(q) = {d_{m,n,k} ∈ W_j | t_{m,n} ≤ t_q} [Eq. 5]. Cued Input: the model receives documents in W_j along with structural cues {s_{m,n}} for all events in the window [3.3.1]. Our structural cues were constructed offline from complete event information [Section 7]."

    Eq. (5) filters the raw documents by t_q, but the cued-input definition filters nothing: cues for all events in the 7-day window W_j are included, and W_j can contain events after t_q. Because cues are extracted offline from the complete event record, the cued prompt may contain post-timestamp facts and answer updates, while the raw prompt is time-filtered. Therefore the Temporal QA Δ_org conflates organization with oracle access to future information; the paper's own limitation concedes the cues are 'constructed offline from complete event information.'

full rationale

The paper contains no fitted parameter renamed as a prediction, so the extreme fitted-input circularity does not apply. The clustering and summarization results are independently defined: clustering uses topic labels, and summarization is compared against Wikipedia-based reference summaries, so those arms retain empirical content despite oracle cue construction. The circularity is concentrated in the Temporal QA arm, which is also the arm with the largest headline gain. There, the gold answer is selected from the cue fields (People/Location/Result) that the cued input provides verbatim, and the temporal filter of Eq. (5) is applied only to documents, not to the offline-extracted cues. Both properties make the QA Δ_org reduce by construction to answer-bearing oracle access rather than to the benefit of organization. The self-citation to CheckEval is a normal tool citation and is not load-bearing for the core derivation. Because two of three tasks retain independent content, the overall circularity score is 6.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The benchmark introduces no new physical or formal entities; structural cues are reorganizations of existing document facts. The main dependencies are dataset-construction choices and annotation assumptions, not fitted parameters in a derivation.

free parameters (4)
  • window size w and stride s = w=7 days, s=1 day
    Evaluation schedule chosen by hand; all results depend on this granularity and no sensitivity analysis is reported.
  • documents sampled per event k = k ∈ {1,3,5,10}
    Independent variable used to control stream volume; claims about conflicts are conditional on these values.
  • article-event similarity threshold = 0.6
    Hand-set cosine threshold used to filter collected news articles; no sensitivity analysis is reported.
  • RAG retrieval threshold = 0.5
    Hand-set threshold in the Section 6.2 ablation separating retrieval from organization.
axioms (6)
  • domain assumption Wikipedia section headings and W2E event definitions provide correct event/topic ground truth
    Used in Sections 4.1 and 4.4 to define events, topics, and clustering labels.
  • domain assumption NewsAPI relevance ranking and cosine-similarity filtering correctly associate each article with its event
    Section 4.1; incorrect document-event mapping would propagate to all three tasks.
  • domain assumption GPT-4o cue extraction is faithful and introduces no information beyond the source documents
    Section 4.3; only a 10% QA subset is human-verified (80.6–83.3% agreement), and cues feed the treatment condition.
  • domain assumption QA answers selected from cue fields are correct ground truth
    Appendix A.2; automatic filtering keeps 1,087 of 1,483 generated pairs, and human agreement is partial.
  • domain assumption 2025 stories lie beyond most evaluated models' knowledge cutoffs, so parametric knowledge does not drive results
    Table 7; cutoffs are approximate and the 2016 stories are inside training data, so this assumption is partially violated.
  • domain assumption Every event in a window has a structural cue available before the model reads the documents
    Sections 3.3 and 7; this is an offline oracle not available in a true streaming setting, acknowledged by the authors.

pith-pipeline@v1.3.0-alltime-deepseek · 20439 in / 11894 out tokens · 106124 ms · 2026-08-02T20:57:53.518918+00:00 · methodology

0 comments
read the original abstract

Evaluating language models in streaming environments is critical, yet underexplored. Existing benchmarks either focus on single complex events or provide curated inputs for each query, and do not evaluate models under the conflicts that arise when multiple concurrent events are mixed within the same document stream. We introduce StreamBench, a benchmark built from major news stories in 2016 and 2025, comprising 605 events and 15,354 documents across three tasks: Topic Clustering, Temporal Question Answering, and Summarization. To diagnose how models fail, we compare performance with and without structural cues, which organize key facts by event. We find that structural cues improve performance on clustering (up to +4.37%) and temporal QA (up to +9.63%), helping models locate relevant information and separate distinct events. While temporal reasoning remains an open challenge inherent to current LLMs, consistent gains across tasks show that structural cues are a promising direction for future work in massive document streams.

Figures

Figures reproduced from arXiv: 2603.19250 by Susik Yoon, Wonjun Choi, Woojun Jung, Yebin Lim, Yukyung Lee.

Figure 1
Figure 1. Figure 1: Two challenges in streaming environments. Intra [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Document stream volume over time. The x-axis [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance bottleneck analysis across tasks and model scales. Stacked bar charts show base performance and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Δorg across model scales and document sizes per event (𝑘) for each task. * indicates statistical significance (𝑝 < 0.05). but drops to 70.04 (𝑘=3), 68.03 (𝑘=5), and 67.96 (𝑘=10). In our setup, clustering is incremental–each arriving document is either assigned to an existing topic or used to create a new one. As 𝑘 increases, more documents from different topics are mixed together in the context, making cor… view at source ↗
Figure 5
Figure 5. Figure 5: CheckEval results for summarization, with [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Temporal Document distribution of StreamBench. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 4 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shya- mal Anadkat, et al. 2023. Gpt-4 technical report.arxivabs/2303.08774 (2023). https://arxiv.org/abs/2303.08774

    Pith/arXiv arXiv 2023

  2. [2]

    2002.Introduction to topic detection and tracking

    James Allan. 2002.Introduction to topic detection and tracking. Kluwer Academic Publishers, USA, 1–16

    2002

  3. [3]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. InProceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss (Eds.). Association for Computational ...

    2005

  4. [4]

    Hai Leong Chieu and Yoong Keok Lee. 2004. Query based event extraction along a timeline. InProceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval(Sheffield, United Kingdom) (SIGIR ’04). Association for Computing Machinery, New York, NY, USA, 425–432. doi:10.1145/1008992.1009065

    arXiv 2004

  5. [5]

    Hui Dai, Ryan Teehan, and Mengye Ren. 2025. Are LLMs Prescient? A Contin- uous Evaluation using Daily News as the Oracle. InForty-second International Conference on Machine Learning. https://openreview.net/forum?id=v2nV83Q849

    2025

  6. [6]

    Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Babu Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A Huerta, and Hao Peng. 2025. Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. InFindings of the Association for Computational Linguistics: EMNLP 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Carol...

    doi:10.18653/v1/2025.findings-emnlp.126 2025

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The LLaMA 3 Herd of Models.arxivabs/2407.21783 (2024). https: //arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  8. [8]

    Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev

    Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating Summa- rization Evaluation.Transactions of the Association for Computational Linguistics 9 (2021), 391–409. doi:10.1162/tacl_a_00373

    doi:10.1162/tacl_a_00373 2021

  9. [9]

    Cristiano Mesquita Garcia, Ramon Abilio, Alessandro Lameiras Koerich, Alceu de Souza Britto, and Jean Paul Barddal. 2025. Concept Drift Adaptation in Text Stream Mining Settings: A Systematic Review.ACM Trans. Intell. Syst. Technol. 16, 2, Article 27 (Feb. 2025), 67 pages. doi:10.1145/3704922

    doi:10.1145/3704922 2025

  10. [10]

    Heitor Murilo Gomes, Jesse Read, Albert Bifet, Jean Paul Barddal, and João Gama. 2019. Machine learning for streaming data: state of the art, challenges, and opportunities.SIGKDD Explor. Newsl.21, 2 (Nov. 2019), 6–22. doi:10.1145/ 3373464.3373470

    arXiv 2019

  11. [11]

    Tuan-Anh Hoang, Khoi Duy Vo, and Wolfgang Nejdl. 2018. W2E: A Worldwide- Event Benchmark Dataset for Topic Detection and Tracking. InProceedings of the 27th ACM International Conference on Information and Knowledge Management (Torino, Italy)(CIKM ’18). Association for Computing Machinery, New York, NY, USA, 1847–1850. doi:10.1145/3269206.3269309

    arXiv 2018

  12. [12]

    Qisheng Hu, Geonsik Moon, and Hwee Tou Ng. 2024. From Moments to Mile- stones: Incremental Timeline Summarization Leveraging Large Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar KDD ’26, August 09–13, 2026, Jeju Island, Repu...

    doi:10.18653/v1/2024.acl-long.390 2024

  13. [13]

    AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, Ddl Casas, F Bres- sand, G Lengyel, G Lample, L Saulnier, et al. 2024. Mistral 7B. arXiv 2023.arxiv abs/2310.06825 (2024). https://arxiv.org/abs/2310.06825

    Pith/arXiv arXiv 2024

  14. [14]

    Smith, Yejin Choi, and Kentaro Inui

    Jungo Kasai, Keisuke Sakaguchi, yoichi takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. 2023. RealTime QA: What’s the Answer Right Now?. InThirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=HfKOIPCvsv

    2023

  15. [15]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles(Koblenz, Germany)(SOSP ’23). Association for Computing Machinery, New York, N...

    arXiv 2023

  16. [16]

    Yukyung Lee, JoongHoon Kim, Jaehee Kim, Hyowon Cho, Jaewook Kang, Pil- sung Kang, and Najoung Kim. 2025. CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Chris- tos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, a...

    doi:10.18653/v1/2025.emnlp-main.796 2025

  17. [17]

    Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Lin...

    doi:10.18653/v1/2024.acl-long.818 2024

  18. [18]

    Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2025. Long- context LLMs Struggle with Long In-context Learning.Transactions on Machine Learning Research(2025). https://openreview.net/forum?id=Cw2xlg0e46

    2025

  19. [19]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. InText summarization branches out. 74–81

    2004

  20. [20]

    Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien De Masson D’Autume, Tim Scholtes, Manzil Za- heer, Susannah Young, Ellen Gilsenan-Mcmahon, Sophia Austin, Phil Blun- som, and Angeliki Lazaridou. 2022. StreamingQA: A Benchmark for Adap- tation to New Knowledge over Time in Question Answering Models. InPro- c...

    2022

  21. [21]

    Nishanth Nakshatri, Siyi Liu, Sihao Chen, Dan Roth, Dan Goldwasser, and Daniel Hopkins. 2023. Using LLM for Improving Key Event Discovery: Temporal-Guided News Stream Clustering with Event Summaries. InFindings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Ling...

    work page doi:10.18653/v1/2023.findings-emnlp.274 2023

  22. [22]

    Jie Ouyang, Tingyue Pan, Mingyue Cheng, Ruiran Yan, Yucong Luo, Jiaying Lin, and Qi Liu. 2025. HoH: A Dynamic Benchmark for Evaluating the Impact of Outdated Information on Retrieval-Augmented Generation. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterin...

    work page doi:10.18653/v1/2025.ac 2025

  23. [23]

    Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu

  24. [24]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318

    2002

  25. [25]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. InProceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing. Association for Computational Linguistics. https://arxiv.org/abs/1908.10084

    Pith/arXiv arXiv 2019

  26. [26]

    Jiayu Song, Mahmud Elahi Akhter, Dana Atzil-Slonim, and Maria Liakata. 2025. Temporal reasoning for timeline summarisation in social media. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Moham- mad Taher Pilehvar (Eds.). Association for ...

    work page doi:10.18653/v1/2025.acl-long.1362 2025

  27. [27]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al . 2025. Gemma 3 technical report.arxivabs/2503.19786 (2025). https://arxiv.org/abs/2503.19786

    Pith/arXiv arXiv 2025

  28. [28]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size.arxivabs/2408.00118 (2024). https://arxiv.org/abs/2408.00118

    Pith/arXiv arXiv 2024

  29. [29]

    Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2024. FreshLLMs: Re- freshing Large Language Models with Search Engine Augmentation. InFindings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association ...

    doi:10.18653/v1/2024.findings-acl.813 2024

  30. [30]

    Xue Wu and Kostas Tsioutsiouliklis. 2024. Thinking with knowledge graphs: En- hancing LLM reasoning through structured data.arXiv preprint arXiv:2412.10654 (2024)

    Pith/arXiv arXiv 2024

  31. [31]

    Runxin Xu, Tianyu Liu, Lei Li, and Baobao Chang. 2021. Document-level Event Extraction via Heterogeneous Graph-based Interaction Model with a Tracker. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), Chengqing Zong...

    work page doi:10.18653/v1/2021.acl-long.2 2021

  32. [32]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al . 2024. Qwen2.5 Technical Report.arxivabs/2412.15115 (2024). https://arxiv.org/abs/2412.15115

    Pith/arXiv arXiv 2024

  33. [33]

    Susik Yoon, Hou Pong Chan, and Jiawei Han. 2023. PDSum: Prototype-driven Con- tinuous Summarization of Evolving Multi-document Sets Stream. InProceedings of the ACM Web Conference 2023(Austin, TX, USA)(WWW ’23). Association for Com- puting Machinery, New York, NY, USA, 1650–1661. doi:10.1145/3543507.3583371

    arXiv 2023

  34. [34]

    Susik Yoon, Yu Meng, Dongha Lee, and Jiawei Han. 2023. SCStory: Self-supervised and Continual Online Story Discovery. InProceedings of the ACM Web Conference 2023(Austin, TX, USA)(WWW ’23). Association for Computing Machinery, New York, NY, USA, 1853–1864. doi:10.1145/3543507.3583507

    arXiv 2023

  35. [35]

    Michael Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating Extra- Linguistic Contexts into QA. InProceedings of the 2021 Conference on Empir- ical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Compu- tational Linguistics, Online and Punta Cana, Dominican Republ...

    doi:10.18653/v1/2021.emnlp-main.586 2021

  36. [36]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi

  37. [37]

    What was the result of [event]?

    Zhihan Zhang, Yixin Cao, Chenchen Ye, Yunshan Ma, Lizi Liao, and Tat-Seng Chua. 2024. Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek...

    arXiv 2024

  38. [2019]

    Bertscore: Evaluating text generation with bert.arXiv:1904.09675(2019)

    Pith/arXiv arXiv 1904

  39. [2024]

    IEEE Trans

    Unifying Large Language Models and Knowledge Graphs: A Roadmap. IEEE Trans. on Knowl. and Data Eng.36, 7 (July 2024), 3580–3599. doi:10.1109/ TKDE.2024.3352100

    arXiv 2024