{"id":"d1492105-e4b9-4ff3-8815-23118e89d096","arxiv_id":"2411.08324","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Daily Oracle shows that LLM forecasting accuracy degrades smoothly over time as pretraining data becomes outdated, and retrieval augmented generation only partially offsets the decline.","lead":"This paper introduces Daily Oracle, a benchmark that generates fresh true/false and multiple-choice forecasting questions from daily news to continuously test how well LLMs predict future events. It matters because the results show LLM accuracy steadily declines as training data ages, and adding recent news articles only slows the drop, not stops it.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temporal decline may reflect item difficulty rather than stale pretraining data: the gold-article control itself declines (Figure 5), and the key filtering criterion has near-zero human agreement (Kappa 0.02), so a controlled knowledge-cutoff experiment is needed before the causal claim.","rationale":"The reader identified the low inter-annotator agreement on 'Non-answerability' as the weakest assumption; this is a genuine validity threat, but it alone does not refute the observed degradation. The more load-bearing gap is that the paper's causal claim—'as pre-training data becomes outdated, performance degrades'—requires showing that the same model with an older knowledge cutoff declines faster than with a newer cutoff on the same benchmark. The paper only compares different models with different cutoffs, which is confounded by model quality, and it runs a gold-article control that still declines, hinting at a temporal difficulty trend. The proposed continued-pretraining checkpoints directly test the causal mechanism; the BM25 recovery check addresses the construct-validity concern. Thus the verdict remains CONDITIONAL, with the added requirement of these controls before the causal interpretation is accepted.","tokens_in":25639,"tokens_out":8979,"duration_ms":96389,"concrete_test":"Create two checkpoints of an open-weight model (e.g., Llama-3-8B) by continued pretraining on the Daily Oracle news corpus with training data cutoffs at 2022-06 and 2024-06. Evaluate both on the same 2020-2024 Daily Oracle questions in closed-book, RAG, and gold-article settings; if the 2024-cutoff checkpoint has a substantially flatter accuracy-over-time curve, the degradation is caused by knowledge staleness, whereas identical downward slopes indicate a benchmark difficulty artifact. As a secondary check, sample 200 filtered questions and use BM25 retrieval over the pre-publication corpus to measure how many answers are obtainable before the publication date, quantifying the non-forecasting share implied by Kappa 0.02.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The core claim that accuracy falls because pretraining data becomes outdated is supported only by a correlation between question date and model knowledge cutoff; the benchmark's temporal axis is confounded with possible trends in intrinsic item difficulty. The gold-article setting (Figure 5) is meant to control for missing knowledge, yet it shows the same downward trend for most models, suggesting that later articles/questions may simply be harder to comprehend or answer, independent of knowledge staleness. This confound is aggravated by the QA filtering step: the decisive criterion, 'Non-answerability Before the Publication Date,' has Fleiss Kappa 0.02 among human annotators (Table 4), so the dataset may contain many items that are not genuine forecasts but rather questions answerable from pre-publication information. If so, the smooth decline could be a property of the LLM-generated item distribution (e.g., GPT-4o-mini generating progressively harder questions) rather than of model knowledge freshness. Because the paper never manipulates the knowledge cutoff while holding the model and item set fixed, the observed degradation cannot be causally attributed to outdated pretraining data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Daily Oracle, a continuously updated forecasting benchmark in which QA pairs (True/False and Multiple Choice) are automatically generated from daily news articles and filtered by an LLM judge against seven quality principles. The authors use a static subset of 31,510 questions (16,783 TF, 14,727 MC) spanning January 2020 to December 2024, with publication date treated as the resolution date, and evaluate eight LLMs in three settings: closed-book, constrained open-book with BM25 retrieval under varying RAG cutoffs, and gold-article reading comprehension. The central empirical claim is that forecasting accuracy degrades as the gap between a model's knowledge cutoff and the question date grows, with average relative drops of 21.55% on TF and 11.33% on MC questions from the start to the end of the evaluation period; the degradation persists under retrieval and, surprisingly, when the gold article containing the answer is provided. The paper interprets the residual decline as evidence of outdated internal representations and argues for continuous model updating.","tokens_in":25885,"tokens_out":4642,"duration_ms":47840,"significance":"If the causal claim were established, the paper would make a meaningful contribution: Daily Oracle is one of the largest continuously generated forecasting benchmarks, it anchors answers to real news outcomes rather than synthetic labels, it evaluates multiple model families, and it includes retrieval and gold-article controls that go beyond prior temporal-generalization work. The public release of code and data is a concrete asset. The paper's honest reporting of human evaluation, including the very low agreement on the key filtering criterion, is also a strength. However, the central claim that accuracy falls because pretraining data becomes outdated is not yet supported: the temporal axis is confounded with possible changes in intrinsic item difficulty, and the paper's own gold-article control exhibits the same declining trend that the knowledge-staleness story is meant to explain.","major_comments":[{"comment":"The decisive filter for a 'genuine forecast' is 'Non-answerability Before the Publication Date', yet Table 4 reports Fleiss Kappa 0.02 for this principle among four human annotators, i.e., essentially no agreement. Because every question's status as a forecast depends on this criterion, the closed-book decline in Table 3 could be driven by a changing fraction of items that are actually answerable from pre-publication information, rather than by the staleness of model knowledge. Please provide an analysis restricted to items on which annotators (or a more reliable procedure) agree that the question is non-answerable before the resolution date, and report whether the temporal decline persists on that subset.","section":"Section 3.3 / Table 4"},{"comment":"The gold-article setting is introduced as a control that should demonstrate answerability and remove missing-knowledge effects, but Figure 5 shows the same downward trend for most models even when the answer is present in the prompt. The paper attributes this to 'outdated internal representations,' but an equally parsimonious explanation is that later-generated questions are intrinsically harder to read or answer, independent of knowledge cutoff. To support the causal interpretation, the authors should hold the item set fixed and compare models (or checkpoints with different knowledge cutoffs) on those same items, or alternatively regress out a per-item difficulty measure (e.g., a strong model's gold-article accuracy) and show that a significant temporal decline remains.","section":"Section 4.2 / Figure 5"},{"comment":"The paper reports yearly averages and year-over-year accuracy changes as point estimates without confidence intervals or significance tests. Several models show small average yearly declines (e.g., Gemma-2-2B TF at -1.04%), and Figure 3 displays single moving-average trajectories. Please provide bootstrap confidence intervals clustered by month, or formal trend tests (e.g., Mann-Kendall), and state for which models the decline is statistically distinguishable from noise. Without this, the smoothness and universality of the degradation is asserted rather than demonstrated.","section":"Section 4.1 / Table 3 / Figure 3"},{"comment":"The dataset generation and filtering pipeline changed from GPT-3.5/GPT-4 to GPT-4o-mini/GPT-4o after October 2024, while the evaluation window extends through December 2024. Because the final months' questions are produced by a different generator, the observed decline could partly reflect a shift in question difficulty rather than model knowledge staleness. Please analyze the effect of this generator change, for example by measuring question difficulty before and after the switch under a fixed reference model, or by repeating the main analysis on only the pre-switch period.","section":"Section 3.1 / Footnote 6"}],"minor_comments":[{"comment":"The 21.55% TF and 11.33% MC drops are relative declines (from 64.68% to 50.74%, and 58.30% to 51.69%, respectively); please state explicitly that these are relative percentages to avoid the reader interpreting them as absolute percentage-point changes.","section":"Abstract / Section 4.2"},{"comment":"The main-text open-book figure shows only Mixtral-8x7B (TF) and Llama-3-8B (MC); please state in the caption that other models appear in Appendix B.4 and list the specific RAG cutoff dates shown in the figure.","section":"Section 4.2 / Figure 4"},{"comment":"The human evaluation sample is only 60 QA pairs, and the average Fleiss Kappa of 0.26 conceals a wide range (0.02 to 0.66). The near-zero agreement on the most important principle deserves discussion in the main text rather than only in the appendix.","section":"Section 3.3"},{"comment":"Counting refusals as incorrect mechanically drives some models below the random baseline (e.g., Mistral-7B and Mixtral-8x7B on TF). Please present the refusal-excluded accuracy more prominently in the main text, since it is the more informative measure of forecasting ability.","section":"Appendix B.2 / Figure 9"},{"comment":"Several appendix figure captions and axis labels appear corrupted with 'gid' strings (e.g., Figures 9-11); the final version should ensure proper rendering of these labels.","section":"Appendix figures"},{"comment":"The Limitations paragraph states that the trend is 'weak within the time horizon available,' which is in tension with the conclusion's claim of a 'significant smooth decline.' Please reconcile these statements, ideally by adding the statistical analysis requested above.","section":"Section 4.3 / Limitations"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline major-revision case. The benchmark contribution is real and likely useful to the community, and the authors are transparent about their human evaluation. The load-bearing problem is the confound between question date and item difficulty, amplified by the near-zero annotator agreement on the non-answerability criterion and by the generator-model switch in late 2024. A controlled knowledge-cutoff experiment or a per-item difficulty adjustment could establish the central causal claim; without it, the paper remains a descriptive study of a new benchmark with an interesting but unproven interpretation. I would encourage the editor to send the revision back with a request for such an analysis rather than to reject, because the dataset itself is a contribution independent of the causal claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the benchmark is a genuine contribution and worth having; the paper's central causal claim—that accuracy drops because pretraining data goes stale—is not actually pinned down by the experiments. The gold-article control, which is meant to remove missing-knowledge effects, also declines, and the key filtering criterion has near-zero human agreement. So the smooth degradation could be item difficulty, not knowledge staleness. The authors need a different experiment to make that claim.\n\nWhat's new and good: Daily Oracle is the first continuously updated daily forecasting QA benchmark built automatically from news, with 31.5k questions over 2020–2024, code and data released. That is useful regardless of the interpretation. The degradation pattern is consistent across eight model families and persists in retrieval and gold-article settings, and the comparison to forecasting-market datasets in Appendix C is a thoughtful argument for why their dataset is better suited to trend analysis. The Anderson & Schooler-style analysis of word frequency is a nice touch. The human evaluation, even with its low kappa, is more than most benchmark papers do.\n\nSoft spots: (1) The \"non-answerability before publication\" principle is the load-bearing filter, and Fleiss kappa 0.02 means human annotators effectively disagree on what counts as a forecast. If many questions are answerable from pre-publication articles, the benchmark measures memory of recency, not forecasting. (2) The gold-article setting shows declining accuracy for most models, which directly undercuts the \"outdated knowledge\" explanation: with the answer in context, a knowledge-staleness story predicts flat accuracy. It could be that models process recent articles less well, but that is a different claim. (3) No confidence intervals or significance tests anywhere; the YoY changes in Table 3 have no uncertainty. (4) The generator models change from GPT-3.5/GPT-4 to GPT-4o-mini/GPT-4o in October 2024, which could shift question difficulty in the later period—a simple control would help. (5) The knowledge-cutoff comparisons are confounded with model identity and question date; a controlled experiment holding the model and item set fixed while varying the cutoff would be the clean test.\n\nBottom line: this deserves peer review. The benchmark is real and useful; the empirical pattern is worth reporting with the causal language dialed back to what the data support. I would send it to a good referee with instructions to focus on the filtering reliability and the gold-article confound. If the authors add a controlled knowledge-cutoff analysis and confidence intervals, it could be a solid paper.","headline":"Useful new continuous forecasting benchmark, but the headline claim about outdated pretraining data is confounded by item difficulty and weak filtering reliability.","tokens_in":26369,"tokens_out":2483,"would_cite":true,"duration_ms":25692,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs lose forecasting accuracy as their training data ages.","keywords":["LLM evaluation","temporal generalization","event forecasting","daily news benchmark","knowledge cutoff","retrieval-augmented generation","continuous pre-training"],"falsifier":"If a random sample of Daily Oracle questions were reviewed by independent human forecasters with forecasting expertise, and the accuracy-over-time curve computed only on the subset they unanimously classify as unanswerable before publication showed no decline, the paper's claim of smooth degradation would be falsified.","tokens_in":25462,"feed_emoji":"📉","tokens_out":12351,"duration_ms":102333,"temperature":0.7,"pith_summary":"The paper introduces Daily Oracle, a benchmark that generates true/false and multiple-choice forecasting questions from daily news, so a model's ability to predict near-future events can be tested on new questions every day. Using this benchmark on eight LLMs with questions dated between January 2020 and December 2024, the authors find that accuracy on forecasting questions falls steadily as the gap between a model's training data and the question date grows, with a relative drop of 21.55% on True/False questions and 11.33% on Multiple Choice questions from the start to the end of the period. The decline appears even when the model is given retrieved news articles in an open-book setting, and even when it receives the exact article the question was generated from. The authors' conclusion is that outdated pre-training data degrades forecasting ability over time, that retrieval is not enough to offset it, and that continuous model updating is needed.","feed_headline":"LLM accuracy on future events drops as training data ages","feed_subtitle":"Even with fresh news articles retrieved into context, models lose ground on forecasting questions over time.","key_machinery":"The load-bearing object is the Daily Oracle pipeline: each day, a set of news articles is summarized, and an LLM generates two True/False and two Multiple Choice questions per article, with the article's publication date treated as the resolution date, so each question is a genuine forecast only if it is not answerable from pre-publication information. A seven-principle filter scores every candidate question on correctness, non-answerability before publication, absence of leakage, objectivity, clear temporal element, public interest, and non-obviousness, keeping items that score at least 13 out of 14. The evaluation machinery is the temporal axis: each model's knowledge cutoff is compared against each question's resolution date, and accuracy is plotted over time; the constrained open-book setting adds a RAG cutoff that limits retrieval to articles before a chosen date, isolating the effect of information recency. A word-frequency recurrence analysis inspired by a classic memory study shows that the knowledge demands of the benchmark follow a stable temporal structure, which supports interpreting the accuracy decline as a temporal-generalization effect.","core_discovery":"The central discovery is that an LLM's accuracy on forecast-style questions about recent events decreases smoothly and substantially as the interval between the model's knowledge cutoff and the question's resolution date grows. In the closed-book setting, accuracy on True/False questions falls on average from 64.68% to 50.74%, and on Multiple Choice questions from 58.30% to 51.69% between 2020 and 2024; year-over-year declines accelerate after the knowledge cutoff, with GPT-4 dropping 18.54% per year on MC questions in the post-cutoff period versus 4.23% before it. The same downward pattern persists when BM25 retrieval supplies up to five recent articles (constrained open-book), and even when the gold article containing the answer is given, most models still show declining accuracy. The paper interprets this as evidence that part of the loss comes from outdated internal representations, not just missing facts, and argues that continuous pre-training or model updating is needed to keep forecasting performance stable.","pith_inferences":["A testable extension is to apply a small rolling continuous-pre-training update to each model and measure whether the slope of the accuracy decline flattens on Daily Oracle.","Given the near-zero human agreement on non-answerability, the benchmark may be more defensibly read as a measure of recency-weighted memory than of strict forecasting skill.","The same daily pipeline could be run with resolved forecasting-market events as a human-verified control set, separating question-generation artifacts from genuine temporal-generalization effects."],"forward_implications":["Static benchmarks that predate a model's training data cannot measure forecasting or temporal generalization; Daily Oracle provides a continuously refreshed alternative.","Outdated pretraining data alone causes measurable accuracy loss on forecasting questions, even when the model can retrieve up-to-date documents.","Performance declines both before and after the knowledge cutoff, but faster after the cutoff, so knowledge-cutoff dates are visible in accuracy curves.","Because even gold-article reading comprehension degrades over time, retrieval cannot fully repair the loss; models need updated internal representations, i.e., continuous pre-training.","The gold-article setting reaches roughly 90% accuracy, showing the benchmark questions are answerable and the decline is not simply unanswerable trivia."],"supporting_citations":[{"why":"Supplies the Common Crawl News data-collection approach and the forecasting-QA paradigm that Daily Oracle adapts.","marker":"Zou et al. (2022)"},{"why":"Provides the automatic QA-generation prompt framework and filtering steps that the pipeline extends.","marker":"Zhang et al. (2024)"},{"why":"Defines retrieval-augmented generation, the open-book mechanism tested against the degradation claim.","marker":"Lewis et al. (2020)"},{"why":"Motivates the word-frequency recurrence analysis that shows the benchmark's knowledge demands follow a stable temporal structure.","marker":"Anderson & Schooler (1991)"},{"why":"Source of the question-categorization prompt and the forecasting-market dataset compared in Appendix C.","marker":"Halawi et al. (2024)"},{"why":"Prior temporal-generalization forecasting benchmark whose volatile trends motivate Daily Oracle's uniform daily coverage.","marker":"Zhu et al. (2025)"},{"why":"Establishes the BM25 retrieval-evaluation protocol used in the constrained open-book setting.","marker":"Jin et al. (2021)"}],"fun_headline_variants":["LLM forecast accuracy crumbles as training data ages","Fresh news can't save LLMs from stale knowledge","LLMs' future prediction skills decay with model age","Daily Oracle reveals LLM forecasting decline over time","Retrieval doesn't fix LLM decline on future events"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's questions are genuine forecasts only if they cannot be answered from information published before the article's publication date, but human raters essentially disagree on that judgment (Fleiss' Kappa 0.02), so the central degradation result depends on the LLM filter's ability to enforce a standard humans cannot reliably apply.","fun_headline_variants_meta":{"raw":{"variants":["LLM forecast accuracy crumbles as training data ages","Fresh news can't save LLMs from stale knowledge","LLMs' future prediction skills decay with model age","Daily Oracle reveals LLM forecasting decline over time","Retrieval doesn't fix LLM decline on future events"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1492,"prompt_tokens":897,"completion_tokens":595,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":518}},"tokens_in":513,"tokens_out":595,"duration_ms":6365,"temperature":1.0,"reasoning_tokens":518,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:43:52.354524+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a random sample of Daily Oracle questions were reviewed by independent human forecasters with forecasting expertise, and the accuracy-over-time curve computed only on the subset they unanimously classify as unanswerable before publication showed no decline, the paper's claim of smooth degradation would be falsified.","supporting_citations":[],"review_version":1}