Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle

Hui Dai; Mengye Ren; Ryan Teehan

REVIEW 4 major objections 6 minor 1 cited by

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs lose forecasting accuracy as their training data ages.

desk verdict Useful new continuous forecasting benchmark, but the headline claim about outdated pretraining data is confounded by item difficulty and weak filtering reliability. read the letter →

arxiv 2411.08324 v2 pith:AOD3BX2J submitted 2024-11-13 cs.CL cs.AIcs.LG

Hui Dai , Ryan Teehan , Mengye Ren This is my paper · ORCID

classification cs.CLcs.AIcs.LG

keywords LLMevaluationtemporalgeneralizationeventforecastingdailynewsbenchmarkknowledgecutoffretrieval-augmentedgenerationcontinuouspre-training

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Daily Oracle, a benchmark that generates true/false and multiple-choice forecasting questions from daily news, so a model's ability to predict near-future events can be tested on new questions every day. Using this benchmark on eight LLMs with questions dated between January 2020 and December 2024, the authors find that accuracy on forecasting questions falls steadily as the gap between a model's training data and the question date grows, with a relative drop of 21.55% on True/False questions and 11.33% on Multiple Choice questions from the start to the end of the period. The decline appears even when the model is given retrieved news articles in an open-book setting, and even when it receives the exact article the question was generated from. The authors' conclusion is that outdated pre-training data degrades forecasting ability over time, that retrieval is not enough to offset it, and that continuous model updating is needed.

What carries the argument

The load-bearing object is the Daily Oracle pipeline: each day, a set of news articles is summarized, and an LLM generates two True/False and two Multiple Choice questions per article, with the article's publication date treated as the resolution date, so each question is a genuine forecast only if it is not answerable from pre-publication information. A seven-principle filter scores every candidate question on correctness, non-answerability before publication, absence of leakage, objectivity, clear temporal element, public interest, and non-obviousness, keeping items that score at least 13 out of 14. The evaluation machinery is the temporal axis: each model's knowledge cutoff is compared against each question's resolution date, and accuracy is plotted over time; the constrained open-book setting adds a RAG cutoff that limits retrieval to articles before a chosen date, isolating the effect of information recency. A word-frequency recurrence analysis inspired by a classic memory study shows that the knowledge demands of the benchmark follow a stable temporal structure, which supports interpreting the accuracy decline as a temporal-generalization effect.

What would settle it

If a random sample of Daily Oracle questions were reviewed by independent human forecasters with forecasting expertise, and the accuracy-over-time curve computed only on the subset they unanimously classify as unanswerable before publication showed no decline, the paper's claim of smooth degradation would be falsified.

Watch

Extended reading notes

Core claim

The central discovery is that an LLM's accuracy on forecast-style questions about recent events decreases smoothly and substantially as the interval between the model's knowledge cutoff and the question's resolution date grows. In the closed-book setting, accuracy on True/False questions falls on average from 64.68% to 50.74%, and on Multiple Choice questions from 58.30% to 51.69% between 2020 and 2024; year-over-year declines accelerate after the knowledge cutoff, with GPT-4 dropping 18.54% per year on MC questions in the post-cutoff period versus 4.23% before it. The same downward pattern persists when BM25 retrieval supplies up to five recent articles (constrained open-book), and even when the gold article containing the answer is given, most models still show declining accuracy. The paper interprets this as evidence that part of the loss comes from outdated internal representations, not just missing facts, and argues that continuous pre-training or model updating is needed to keep forecasting performance stable.

Load-bearing premise

The benchmark's questions are genuine forecasts only if they cannot be answered from information published before the article's publication date, but human raters essentially disagree on that judgment (Fleiss' Kappa 0.02), so the central degradation result depends on the LLM filter's ability to enforce a standard humans cannot reliably apply.

Editorial extensions

If this is right

Static benchmarks that predate a model's training data cannot measure forecasting or temporal generalization; Daily Oracle provides a continuously refreshed alternative.
Outdated pretraining data alone causes measurable accuracy loss on forecasting questions, even when the model can retrieve up-to-date documents.
Performance declines both before and after the knowledge cutoff, but faster after the cutoff, so knowledge-cutoff dates are visible in accuracy curves.
Because even gold-article reading comprehension degrades over time, retrieval cannot fully repair the loss; models need updated internal representations, i.e., continuous pre-training.
The gold-article setting reaches roughly 90% accuracy, showing the benchmark questions are answerable and the decline is not simply unanswerable trivia.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension is to apply a small rolling continuous-pre-training update to each model and measure whether the slope of the accuracy decline flattens on Daily Oracle.
Given the near-zero human agreement on non-answerability, the benchmark may be more defensibly read as a measure of recency-weighted memory than of strict forecasting skill.
The same daily pipeline could be run with resolved forecasting-market events as a human-verified control set, separating question-generation artifacts from genuine temporal-generalization effects.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the benchmark is a genuine contribution and worth having; the paper's central causal claim—that accuracy drops because pretraining data goes stale—is not actually pinned down by the experiments. The gold-article control, which is meant to remove missing-knowledge effects, also declines, and the key filtering criterion has near-zero human agreement. So the smooth degradation could be item difficulty, not knowledge staleness. The authors need a different experiment to make that claim.

What's new and good: Daily Oracle is the first continuously updated daily forecasting QA benchmark built automatically from news, with 31.5k questions over 2020–2024, code and data released. That is useful regardless of the interpretation. The degradation pattern is consistent across eight model families and persists in retrieval and gold-article settings, and the comparison to forecasting-market datasets in Appendix C is a thoughtful argument for why their dataset is better suited to trend analysis. The Anderson & Schooler-style analysis of word frequency is a nice touch. The human evaluation, even with its low kappa, is more than most benchmark papers do.

Soft spots: (1) The "non-answerability before publication" principle is the load-bearing filter, and Fleiss kappa 0.02 means human annotators effectively disagree on what counts as a forecast. If many questions are answerable from pre-publication articles, the benchmark measures memory of recency, not forecasting. (2) The gold-article setting shows declining accuracy for most models, which directly undercuts the "outdated knowledge" explanation: with the answer in context, a knowledge-staleness story predicts flat accuracy. It could be that models process recent articles less well, but that is a different claim. (3) No confidence intervals or significance tests anywhere; the YoY changes in Table 3 have no uncertainty. (4) The generator models change from GPT-3.5/GPT-4 to GPT-4o-mini/GPT-4o in October 2024, which could shift question difficulty in the later period—a simple control would help. (5) The knowledge-cutoff comparisons are confounded with model identity and question date; a controlled experiment holding the model and item set fixed while varying the cutoff would be the clean test.

Bottom line: this deserves peer review. The benchmark is real and useful; the empirical pattern is worth reporting with the causal language dialed back to what the data support. I would send it to a good referee with instructions to focus on the filtering reliability and the gold-article confound. If the authors add a controlled knowledge-cutoff analysis and confidence intervals, it could be a solid paper.

Referee Report

4 major / 6 minor

Summary. The paper proposes Daily Oracle, a continuously updated forecasting benchmark in which QA pairs (True/False and Multiple Choice) are automatically generated from daily news articles and filtered by an LLM judge against seven quality principles. The authors use a static subset of 31,510 questions (16,783 TF, 14,727 MC) spanning January 2020 to December 2024, with publication date treated as the resolution date, and evaluate eight LLMs in three settings: closed-book, constrained open-book with BM25 retrieval under varying RAG cutoffs, and gold-article reading comprehension. The central empirical claim is that forecasting accuracy degrades as the gap between a model's knowledge cutoff and the question date grows, with average relative drops of 21.55% on TF and 11.33% on MC questions from the start to the end of the evaluation period; the degradation persists under retrieval and, surprisingly, when the gold article containing the answer is provided. The paper interprets the residual decline as evidence of outdated internal representations and argues for continuous model updating.

Significance. If the causal claim were established, the paper would make a meaningful contribution: Daily Oracle is one of the largest continuously generated forecasting benchmarks, it anchors answers to real news outcomes rather than synthetic labels, it evaluates multiple model families, and it includes retrieval and gold-article controls that go beyond prior temporal-generalization work. The public release of code and data is a concrete asset. The paper's honest reporting of human evaluation, including the very low agreement on the key filtering criterion, is also a strength. However, the central claim that accuracy falls because pretraining data becomes outdated is not yet supported: the temporal axis is confounded with possible changes in intrinsic item difficulty, and the paper's own gold-article control exhibits the same declining trend that the knowledge-staleness story is meant to explain.

major comments (4)

[Section 3.3 / Table 4] The decisive filter for a 'genuine forecast' is 'Non-answerability Before the Publication Date', yet Table 4 reports Fleiss Kappa 0.02 for this principle among four human annotators, i.e., essentially no agreement. Because every question's status as a forecast depends on this criterion, the closed-book decline in Table 3 could be driven by a changing fraction of items that are actually answerable from pre-publication information, rather than by the staleness of model knowledge. Please provide an analysis restricted to items on which annotators (or a more reliable procedure) agree that the question is non-answerable before the resolution date, and report whether the temporal decline persists on that subset.
[Section 4.2 / Figure 5] The gold-article setting is introduced as a control that should demonstrate answerability and remove missing-knowledge effects, but Figure 5 shows the same downward trend for most models even when the answer is present in the prompt. The paper attributes this to 'outdated internal representations,' but an equally parsimonious explanation is that later-generated questions are intrinsically harder to read or answer, independent of knowledge cutoff. To support the causal interpretation, the authors should hold the item set fixed and compare models (or checkpoints with different knowledge cutoffs) on those same items, or alternatively regress out a per-item difficulty measure (e.g., a strong model's gold-article accuracy) and show that a significant temporal decline remains.
[Section 4.1 / Table 3 / Figure 3] The paper reports yearly averages and year-over-year accuracy changes as point estimates without confidence intervals or significance tests. Several models show small average yearly declines (e.g., Gemma-2-2B TF at -1.04%), and Figure 3 displays single moving-average trajectories. Please provide bootstrap confidence intervals clustered by month, or formal trend tests (e.g., Mann-Kendall), and state for which models the decline is statistically distinguishable from noise. Without this, the smoothness and universality of the degradation is asserted rather than demonstrated.
[Section 3.1 / Footnote 6] The dataset generation and filtering pipeline changed from GPT-3.5/GPT-4 to GPT-4o-mini/GPT-4o after October 2024, while the evaluation window extends through December 2024. Because the final months' questions are produced by a different generator, the observed decline could partly reflect a shift in question difficulty rather than model knowledge staleness. Please analyze the effect of this generator change, for example by measuring question difficulty before and after the switch under a fixed reference model, or by repeating the main analysis on only the pre-switch period.

minor comments (6)

[Abstract / Section 4.2] The 21.55% TF and 11.33% MC drops are relative declines (from 64.68% to 50.74%, and 58.30% to 51.69%, respectively); please state explicitly that these are relative percentages to avoid the reader interpreting them as absolute percentage-point changes.
[Section 4.2 / Figure 4] The main-text open-book figure shows only Mixtral-8x7B (TF) and Llama-3-8B (MC); please state in the caption that other models appear in Appendix B.4 and list the specific RAG cutoff dates shown in the figure.
[Section 3.3] The human evaluation sample is only 60 QA pairs, and the average Fleiss Kappa of 0.26 conceals a wide range (0.02 to 0.66). The near-zero agreement on the most important principle deserves discussion in the main text rather than only in the appendix.
[Appendix B.2 / Figure 9] Counting refusals as incorrect mechanically drives some models below the random baseline (e.g., Mistral-7B and Mixtral-8x7B on TF). Please present the refusal-excluded accuracy more prominently in the main text, since it is the more informative measure of forecasting ability.
[Appendix figures] Several appendix figure captions and axis labels appear corrupted with 'gid' strings (e.g., Figures 9-11); the final version should ensure proper rendering of these labels.
[Section 4.3 / Limitations] The Limitations paragraph states that the trend is 'weak within the time horizon available,' which is in tension with the conclusion's claim of a 'significant smooth decline.' Please reconcile these statements, ideally by adding the statistical analysis requested above.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Daily Oracle is anchored to external news ground truth, and the degradation claim is a direct measurement rather than a derived or fitted result.

full rationale

The paper's central claim is an empirical measurement, not a derivation: accuracy is scored against answers extracted directly from real news articles, so the evaluation is anchored to external ground truth rather than to the authors' own equations or fitted values. No parameter is fitted to the target finding; knowledge cutoffs are taken from public model cards, and the only regression in the paper (Figure 2, word frequency against future occurrence) is a descriptive replication of Anderson and Schooler, not an input to the accuracy results. The constrained open-book and gold-article settings provide independent controls, and the RAG cutoff is varied explicitly rather than tuned to produce the decline. The LLM-based generation and filtering pipeline could imprint LLM priors onto item difficulty, and the near-zero human agreement on the 'Non-answerability Before the Publication Date' criterion (Fleiss Kappa 0.02, Table 4) weakens construct validity as a forecasting benchmark, but these are measurement-validity concerns, not circular reductions: no equation or fitted parameter makes the performance degradation equal to the benchmark's construction, and no self-citation carries the argument. The paper is therefore self-contained as an evaluation study, with the noted confounding risks belonging to correctness, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The benchmark's central claim rests on the quality of the LLM-generated filtering and the accuracy of stated knowledge cutoffs. The key free parameters are hand-set thresholds and retrieval settings that affect the data composition and measured RAG benefit. No new physical or conceptual entities are introduced.

free parameters (4)

QA filtering score threshold = 13 out of 14
Hand-chosen cutoff in Section 3.1 (QA Filtering) for accepting generated questions; determines dataset composition and could shape the temporal difficulty trend.
Daily article selection count = 6 articles per day (3 random, 3 hot-topic)
Design choice in Section 3.1 that controls how many new questions enter the benchmark each day and which news events are represented.
BM25 top-k retrieved articles = 5
Retrieval setting in Section 4.1; the finding that RAG does not eliminate degradation could depend on the number of retrieved articles.
Retrieved article truncation length = 512 words
Truncation in Section 4.1 limits context; larger context might change RAG accuracy, so this is a hand-set experimental parameter.

assumptions (4)

domain assumption The five selected news sources and the Common Crawl News corpus provide a representative and correct oracle for event outcomes.
Section 3.1 uses articles from CBS, CNBC, CNN, Forbes, NPR as ground truth; if these sources miss or misreport events, the answers are wrong.
domain assumption The LLM-based QA filtering reliably enforces the seven quality criteria, especially non-answerability before publication.
Section 3.1 and Appendix A.4; this is the load-bearing assumption, and the low Kappa on the key criterion weakens it.
domain assumption The stated knowledge cutoff dates from model cards reflect the actual end of training data, so post-cutoff questions are unseen.
Section 4.1 defines pre/post cutoff splits; if a model's real cutoff is later than stated, the sharp post-cutoff decline is miscalibrated.
standard math Linear regression with exponential weighing is a valid way to summarize changing trends in noisy monthly accuracy.
Section 4.3, Figure 6 uses this to infer gradual vs rapid decline; it is descriptive and not justified by tests.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle." pith.science (2026). https://pith.science/paper/AOD3BX2J

@misc{pith2026241108324,
  author       = {Pith},
  title        = {Pith review of: Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AOD3BX2J}},
  note         = {Machine review of arXiv:2411.08324}
}

read the original abstract

Many existing evaluation benchmarks for Large Language Models (LLMs) quickly become outdated due to the emergence of new models and training data. These benchmarks also fall short in assessing how LLM performance changes over time, as they consist of a static set of questions without a temporal dimension. To address these limitations, we propose using future event prediction as a continuous evaluation method to assess LLMs' temporal generalization and forecasting abilities. Our benchmark, Daily Oracle, automatically generates question-answer (QA) pairs from daily news, challenging LLMs to predict "future" event outcomes. Our findings reveal that as pre-training data becomes outdated, LLM performance degrades over time. While Retrieval Augmented Generation (RAG) has the potential to enhance prediction accuracy, the performance degradation pattern persists, highlighting the need for continuous model updates. Code and data are available at https://agenticlearning.ai/daily-oracle.

Figures

Figures reproduced from arXiv: 2411.08324 by the authors.

**Figure 2.** Following Anderson & Schooler (1991), we plot the probability of a word occurring in an (left) article title, (middle) True/False question, or (right) Multiple Choice question given how frequently it had appeared in one over the past 100 days, computed over our entire dataset. We fit a linear regression and show a linear relationship in each case (R 2 = 0.843, 0.982, and 0.990 for left, middle, and right respectivel… view at source ↗

**Figure 3.** Results for the closed-book setting. We plot the 5-month moving average accuracy for TF and MC questions across various models, showing LLMs’ performance degradation in future event prediction [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (34 more)

**Figure 4.** Figure 4: Results for the constrained open-book setting, evaluating Mixtral-8x7B on TF questions and Llama-3-8B on MC questions with different RAG cutoff dates. Model Knowledge Cutoff [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Results for the gold article setting. Most of the models struggle with temporal generalization, even when provided with gold articles containing the answers. increasingly negative not long after the knowledge cutoff, giving evidence for a rapid decline in the near futu…

**Figure 6.** Figure 6: Coefficients for regressing accuracy on the MC questions against time, as the number of months grows. Using an initial window of 10 months, we progressively add data for additional months to our regression and plot the coefficient (slope) for the regression of accuracy…

**Figure 7.** Figure 7: shows how Daily Oracle is automatically generated as discussed in Section 3.1. Common Crawl News Article Selection Data Source QA Construction 1. Article Summary 2. QA Generation MC Q: What will …? a) … b) Answer c) … 3. Misleading Choices Generation TF Q: Will …? A: Y…

**Figure 8.** Figure 8: Question category distribution over time. B. Experiment Details B.1. Baseline Models Information [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

**Figure 9.** Figure 9: Accuracy excluding refusal rates and refusal rates under the closed-book setting. We plot the 5-month moving average refusal rates for TF and MC questions across different LLMs. We count refusal cases as incorrect both to maintain comparability across models and becaus…

**Figure 10.** Figure 10: Accuracy excluding refusal rates and refusal rates for Mixtral-8x7B under the constrained open-book setting. (b) shows that the open-book refusal rate (blue curves) is lower than in the closed-book setting (cyan curve), indicating that access to more up-to-date and re…

**Figure 11.** Figure 11: Accuracy excluding refusal rates and refusal rates under the gold article setting [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

**Figure 12.** Figure 12: Results for GPT-3.5 in the gold article setting. Compared to other models achieving around 0.9 accuracy, GPT-3.5 performs worse in both MC questions and, more notably, in TF questions. Thus, while we do not claim that LLM-generated questions are of inherently higher q…

**Figure 13.** Figure 13: Results for Claude-3.5-Sonnet in the constrained open-book setting [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]

**Figure 14.** Figure 14: Results for GPT-4 in the constrained open-book setting [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]

**Figure 15.** Figure 15: Results for Mixtral-8x7B in the constrained open-book setting. D. Prompts All the prompts we use are shown in this section. The QA generation prompts and evaluation prompts are adapted from Zhang et al. (2024), and the prompt to categorize our generated questions is t…

**Figure 16.** Figure 16: Results for Mistral-7B in the constrained open-book setting [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]

**Figure 17.** Figure 17: Results for Llama-3-8B in the constrained open-book setting [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]

**Figure 18.** Figure 18: Results for Qwen-2-7B in the constrained open-book setting. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]

**Figure 19.** Figure 19: Results for Gemma-2-2B in the constrained open-book setting. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]

**Figure 20.** Figure 20: An example of evaluating Mixtral-8x7B across different settings. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]

**Figure 21.** Figure 21: Dataset size and closed-book evaluation accuracy over time on the forecasting market dataset. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_21.png]

**Figure 22.** Figure 22: Distribution shift of the filtered forecasting market dataset. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_22.png]

**Figure 23.** Figure 23: Prompt in the Article Summary step, adapted from Zhang et al. (2024). 25 [PITH_FULL_IMAGE:figures/full_fig_p025_23.png]

**Figure 24.** Figure 24: Prompt in the QA Generation step (part 1), adapted from Zhang et al. (2024). 26 [PITH_FULL_IMAGE:figures/full_fig_p026_24.png]

**Figure 25.** Figure 25: Prompt in the QA Generation step (part 2), adapted from Zhang et al. (2024). 27 [PITH_FULL_IMAGE:figures/full_fig_p027_25.png]

**Figure 26.** Figure 26: Prompt in the Misleading Choices Generation step, adapted from Zhang et al. (2024). 28 [PITH_FULL_IMAGE:figures/full_fig_p028_26.png]

**Figure 27.** Figure 27: Prompt in the QA Filtering step (part 1). 29 [PITH_FULL_IMAGE:figures/full_fig_p029_27.png]

**Figure 28.** Figure 28: Prompt in the QA Filtering step (part 2). 30 [PITH_FULL_IMAGE:figures/full_fig_p030_28.png]

**Figure 29.** Figure 29: Prompt in the QA Filtering step (part 3). 31 [PITH_FULL_IMAGE:figures/full_fig_p031_29.png]

**Figure 30.** Figure 30: Prompt in the QA Filtering step (part 4). 32 [PITH_FULL_IMAGE:figures/full_fig_p032_30.png]

**Figure 31.** Figure 31: Closed-book evaluation prompt for TF questions, adapted from Zhang et al. (2024). System Prompt: You’re an expert in forecasting events. You will NEVER refuse to answer a forecasting question by saying “I cannot predict the future”, even if without 100% certainty. Use…

**Figure 32.** Figure 32: Closed-book evaluation prompt for MC questions, adapted from Zhang et al. (2024). System Prompt: You’re an expert in forecasting events. You will NEVER refuse to answer a forecasting question by saying “I cannot predict the future”, even if without 100% certainty. Use…

**Figure 33.** Figure 33: Constrained open-book evaluation prompt for TF questions, adapted from Zhang et al. (2024). 33 [PITH_FULL_IMAGE:figures/full_fig_p033_33.png]

**Figure 34.** Figure 34: Constrained open-book evaluation prompt for MC questions, adapted from Zhang et al. (2024). System Prompt: You’re an expert in forecasting events. You will NEVER refuse to answer a forecasting question by saying “I cannot predict the future”, even if without 100% cert…

**Figure 35.** Figure 35: Gold article evaluation prompt for TF questions, adapted from Zhang et al. (2024). System Prompt: You’re an expert in forecasting events. You will NEVER refuse to answer a forecasting question by saying “I cannot predict the future”, even if without 100% certainty. Us…

**Figure 36.** Figure 36: Gold article evaluation prompt for MC questions, adapted from Zhang et al. (2024). 34 [PITH_FULL_IMAGE:figures/full_fig_p034_36.png]

**Figure 37.** Figure 37: Prompt to categorize the generated questions, taken from Halawi et al. (2024). 35 [PITH_FULL_IMAGE:figures/full_fig_p035_37.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Pitfalls in Evaluating Language Model Forecasters
cs.LG 2025-05 accept novelty 6.0 of 10

A systematic critique showing temporal leakage and extrapolation flaws can undermine claims that LLM forecasters match or beat humans.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages · cited by 1 Pith paper

[1]

in the future

Article Summary2. QA Generation MCQ: What will …?a) …b) Answerc) …3. Misleading Choices Generation TFQ: Will …?A: Yes4. QA Filtering … … … MC TF MC TF MC TF Day 1Day 2 Day n Figure 7. Data Construction Process of Daily Oracle. A.2. Details for Article Selection We select daily articles that generate the QA pairs in two ways: (1) Random Selection: We rando...

work page 2024
[2]

Your summary should include key statistics, notable dates, and any significant statements to fully convey the context and content of the news story

Provide a comprehensive summary of the entire article in one paragraph, ensuring that all essential aspects are addressed. Your summary should include key statistics, notable dates, and any significant statements to fully convey the context and content of the news story

work page
[3]

he, she, they, it, them, etc

Please provide one keypoint that summarizes the new event from the article with the following rules: - Focus specifically on events that are newly occurring on the publication date of the article. If the article does not introduce a new event but instead discusses ongoing topics or is about non-news content like advertisements, state ‘No new event reporte...

work page 2024
[4]

Please provide the question that can be answered on {}, but only guessable not answerable before {}

The question should challenge the person to predict an outcome or development related to the article’s content as if they were being asked one week or one month ago. Please provide the question that can be answered on {}, but only guessable not answerable before {}

work page
[5]

Ensure that the answers do not predict outcomes that have not been explicitly stated in the article

The answer MUST be based on factual information from the article. Ensure that the answers do not predict outcomes that have not been explicitly stated in the article. [Rules:] Article: {}. Publishing date: {} Please generate four questions about the above article, along with answers. You should follow the instructions below:

work page
[6]

Please turn the key point “{}” into the question, with focusing more on whether the event will happen

work page
[7]

Please focus more on what happened rather than the implications after the event

The question should NOT be designed for reading comprehension. Please focus more on what happened rather than the implications after the event

work page
[8]

Will”, with the answer as “Yes

The question MUST be in future tense. - Start the first question with “Will”, with the answer as “Yes”. - Start the second question with “Will”, with the answer as “No”. - Start the third and fourth questions with a phrase like “What will”, “Who will”, “Where will”, “Which xxx will”, “How much will”, or “How many will”

work page

Show all 18 references

[9]

In {}

There must be a time element in the question. It can be phrases like “In {} ...”, “By {}, ...”, “... in {}?”
[10]

in the future

You MUST NOT use unclear implicit time element phrases like “in the future” or “in the upcoming weeks”
[11]

You should avoid: questions that require numerical reasoning; questions that require substantial world knowledge
[12]

The answer MUST be short and concise, avoiding using redundant words or repeating the information in the question
[13]

he, she, they, it, them, etc

The question must be grammatically correct and contain the information required to answer. NO “he, she, they, it, them, etc” allowed. Please clearly write out the entity you are referencing in the question
[14]

The question MUST be able to be answered by the article
[15]

How will

The question MUST NOT include the information that came out just now. It should be understandable to people from the past. Avoid using “How will” or “Why will” questions, as they imply that the event has already occurred. [Suggested questions and questions to avoid are detaile...

2024
[16]

Make sure you only return one of the options from the option list
[17]

Only output the category, and do not output any other words in your response
[18]

Answer: Figure 37

You have to pick a string from the above categories. Answer: Figure 37. Prompt to categorize the generated questions, taken from Halawi et al. (2024). 35

2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

in the future

Article Summary2. QA Generation MCQ: What will …?a) …b) Answerc) …3. Misleading Choices Generation TFQ: Will …?A: Yes4. QA Filtering … … … MC TF MC TF MC TF Day 1Day 2 Day n Figure 7. Data Construction Process of Daily Oracle. A.2. Details for Article Selection We select daily articles that generate the QA pairs in two ways: (1) Random Selection: We rando...

work page 2024

[2] [2]

Your summary should include key statistics, notable dates, and any significant statements to fully convey the context and content of the news story

Provide a comprehensive summary of the entire article in one paragraph, ensuring that all essential aspects are addressed. Your summary should include key statistics, notable dates, and any significant statements to fully convey the context and content of the news story

work page

[3] [3]

he, she, they, it, them, etc

Please provide one keypoint that summarizes the new event from the article with the following rules: - Focus specifically on events that are newly occurring on the publication date of the article. If the article does not introduce a new event but instead discusses ongoing topics or is about non-news content like advertisements, state ‘No new event reporte...

work page 2024

[4] [4]

Please provide the question that can be answered on {}, but only guessable not answerable before {}

The question should challenge the person to predict an outcome or development related to the article’s content as if they were being asked one week or one month ago. Please provide the question that can be answered on {}, but only guessable not answerable before {}

work page

[5] [5]

Ensure that the answers do not predict outcomes that have not been explicitly stated in the article

The answer MUST be based on factual information from the article. Ensure that the answers do not predict outcomes that have not been explicitly stated in the article. [Rules:] Article: {}. Publishing date: {} Please generate four questions about the above article, along with answers. You should follow the instructions below:

work page

[6] [6]

Please turn the key point “{}” into the question, with focusing more on whether the event will happen

work page

[7] [7]

Please focus more on what happened rather than the implications after the event

The question should NOT be designed for reading comprehension. Please focus more on what happened rather than the implications after the event

work page

[8] [8]

Will”, with the answer as “Yes

The question MUST be in future tense. - Start the first question with “Will”, with the answer as “Yes”. - Start the second question with “Will”, with the answer as “No”. - Start the third and fourth questions with a phrase like “What will”, “Who will”, “Where will”, “Which xxx will”, “How much will”, or “How many will”

work page