REVIEW 4 major objections 5 minor 1 cited by
Prompt Engineering Large Language Models' Forecasting Capabilities
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Prompt engineering barely moves large language model forecasting accuracy, two preregistered studies find.
desk verdict A solid, well-preregistered negative result on prompt engineering for LLM forecasting, but the 'negligible' language overstates what the data can support. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a preregistered linear mixed-effects model that predicts, for each forecast, the difference between the treatment prompt's Brier score and the control prompt's Brier score on the same question and model. The Brier score—the mean squared difference between the predicted probability and the binary outcome—is the accuracy measure, with random intercepts for questions and models and Benjamini-Hochberg correction for multiple comparisons. Study 2 reuses the same 100-question set and the same analytic model, adding compound and externally sourced prompts.
What would settle it
A preregistered replication on a fresh sample of several hundred forecasting questions that finds even one simple, single-paragraph prompt with a stable, multiple-comparison-adjusted Brier-score improvement over control of 0.01 or more would undercut the central claim.
Extended reading notes
Core claim
The paper claims that prompt engineering has minimal to nonexistent effects on LLM forecasting accuracy. In Study 1, 37 single-paragraph treatment prompts were compared with a control across Claude 3.5 Sonnet, Claude 3.5 Haiku, GPT-4o, and Llama 3.1 405B on 100 binary forecasting questions drawn from ForecastBench; only the Bayesian Reasoning and Propose-Evaluate-Select prompts showed statistically significant differences from control, and both reduced accuracy. In Study 2, compound prompts, prompts from automated generators, and prompts adapted from the forecasting literature were tested on the same questions with the original models plus o1 and o1-mini; only the Superforecaster-authored Conditional Odds-Ratio Prompt significantly reduced accuracy. The authors read these results as evidence that basic prompt refinements alone offer limited gains, and that more substantial performance improvements will require techniques such as fine-tuning, retrieval, or reinforcement learning.
Load-bearing premise
The results assume that 100 forecasting questions from ForecastBench, measured with Brier-score differences from control in the preregistered mixed-effects model, give enough statistical power to detect any prompt effect large enough to matter in practice.
Editorial extensions
If this is right
- Organizations should not expect simple prompt rewrites to meaningfully improve LLM forecasting accuracy.
- Base-rate and frequency-based prompts may offer small, safe improvements, but the effect is not robust enough to depend on.
- Prompts that explicitly instruct Bayesian updating or propose-then-evaluate protocols can actively degrade forecast accuracy.
- Meaningful accuracy gains in AI forecasting will likely require pipeline-level changes such as fine-tuning, retrieval, or reinforcement learning rather than text instructions.
- Reasoning models such as o1 and o1-mini do not appear to escape the general pattern of null prompt effects.
Reading between the lines
- A natural extension the authors do not pursue is whether prompt effects interact with question domain: prompts might matter more for questions with richer available context or for multi-turn forecasting systems.
- A testable follow-up would be to run the same prompt set on several independent samples of questions rather than one 100-item set; the current design is best equipped to detect large, uniform prompt effects.
- The paper's null results could reflect a ceiling on what a single static instruction can do when the model has no access to external information, rather than a universal limit on prompt engineering for forecasting.
- One could also measure prompt effects on calibration and resolution separately, since prompts that do not change Brier scores could still shift forecast distribution in ways that matter for decision-making.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript reports two preregistered studies on whether prompt engineering improves LLM forecasting accuracy. Study 1 compares 37 treatment prompts against a control prompt across four LLMs on 100 ForecastBench questions that resolve after the models' training cutoffs, using a linear mixed-effects model with random intercepts for questions and models as the primary analysis and one-sample t-tests as a secondary analysis, with Benjamini-Hochberg corrections. Study 2 tests compound prompts, automatically generated prompts, and prompts adapted from the literature across six models including reasoning models, using the same question set and analysis plan. The authors find that most prompts do not significantly change Brier scores relative to control, that Bayesian Reasoning and Propose-Evaluate-Select significantly worsen accuracy in Study 1, and that only a Superforecaster-authored Conditional Odds-Ratio prompt significantly worsens accuracy in Study 2. They conclude that basic prompt refinements offer limited gains and that prompt engineering is unlikely to play a major role in improving LLM forecasting performance.
Significance. The manuscript has several genuine strengths: it is preregistered, uses questions resolved after the models' training cutoffs, compares each prompt against a paired control rather than relying on between-model accuracy comparisons, applies multiple-comparison corrections, and covers a broad prompt space across frontier, small, open-weight, and reasoning models. If the conclusions are carefully scoped, this is a useful negative result for the LLM forecasting community. However, the strongest version of the claim, that prompt effects are 'negligible' or 'nonexistent,' is not fully supported by the reported analyses, which mostly show non-significance rather than equivalence and which contain at least one table whose reported p-values are inconsistent with the reported test statistics. The paper is worth publishing after the inferential claims are brought in line with what the design can establish.
major comments (4)
- [§4, General Discussion; §2.2, Table 5] The central claim that prompt engineering has a 'minimal to nonexistent' effect on forecasting accuracy is an equivalence claim, but the preregistered analysis never defines a smallest effect size of interest and never runs equivalence tests. Non-significance is not the same as absence of an effect. Indeed, the secondary t-tests in Table 5 show that Frequency-Based Reasoning, Base Rate First, and Step-Back significantly outperform control after Benjamini-Hochberg correction, with mean Brier improvements of 0.014 to 0.019; these effects attenuate but do not fully vanish in the mixed-effects model of Table 4. Given that differences of 0.01 to 0.02 Brier are considered meaningful in forecasting tournaments, the data robustly support only 'no large, consistent average improvements,' not 'negligible or nonexistent effects.' The authors should either add pre-specified equivalence bounds and report equivalence tests, or soften the abstract and discussion claims to match the precision of the design.
- [§2.2, Table 4] The p-values in Table 4 are internally inconsistent with the reported z-ratios. For example, Frequency-Based Reasoning has z = -2.563 but p = 0.128, Base Rate First has z = -2.069 but p = 0.253, and Anti-Biasing (Round Numbers) has z = 2.097 but p = 0.253; under a standard normal reference distribution, these z-ratios correspond to p-values of approximately 0.010, 0.038, and 0.036, respectively. Since the primary conclusion that only Bayesian Reasoning and Propose-Evaluate-Select remain significant after adjustment depends directly on these values, the table must be corrected and the primary results rechecked before the manuscript can be accepted.
- [§3.1.3, §3.2, and §4] Study 2 uses the same 100 questions both to select successful prompts from Study 1 and to evaluate the compound prompts constructed from those successes. This selection-then-test-on-the-same-data design means that the Study 2 estimates are conditional on Study 1 noise and do not constitute a clean out-of-sample replication. The manuscript should either use a holdout question set for Study 2 or explicitly adjust for selection effects; otherwise, the claim that compound prompts 'replicate' the absence of prompt effects is overstated.
- [§2.2, Table 4] The mixed-effects model includes random intercepts for questions and models but treats each prompt effect as a fixed constant across all questions. Consequently, a prompt that improves performance on some question types and harms it on others averages to zero, so a non-significant fixed prompt coefficient does not establish that prompt effects are negligible across heterogeneous forecasting tasks. Reporting prompt-by-question random slopes or variance components would directly address this concern and would strengthen the generality of the conclusions.
minor comments (5)
- [§3.2] The text contains a typo: 'Benjmanini-Hochberg' should be 'Benjamini-Hochberg.'
- [§3.2] The model name 'LLama 3.1 405B' should be capitalized consistently as 'Llama 3.1 405B.'
- [§4, General Discussion] The sentence 'In Study 2, we replicated the effect of Study 2' appears to be a typo and should read 'the effect of Study 1.'
- [§2.2, Table 5] The table and surrounding text note that eight prompts were significant before adjustment, but the rows labeled 'Deep Breath' and 'Explicit Uncertainty Sources' are not listed in the text's enumeration; the authors should make the list consistent with the table.
- [§3.2, Table 9] The notes state that after correction no prompt improved or reduced performance, but the Superforecaster-authored Conditional Odds-Ratio Prompt has an adjusted p-value of 0.124, which is consistent with the stated conclusion; still, the table would benefit from a confidence interval column to communicate precision.
Circularity Check
No significant circularity: forecasts are scored against externally resolved outcomes, and no fitted parameter or self-citation chain forces the null conclusion.
full rationale
The paper's central result is an empirical null finding: prompt modifications rarely improve Brier scores relative to a control. The outcome variable (Brier score) is computed from model forecasts and externally resolved binary outcomes, so the evaluation is not defined in terms of the prompts being tested. Study 1 fits a preregistered mixed-effects model with prompt effects as fixed effects and random intercepts for questions and models; this is a statistical test of differences, not a fitted parameter that is later relabeled as a prediction. The significant negative effects (Bayesian Reasoning, Propose-Evaluate-Select) and the lack of robust positive effects are read directly off the estimated differences, and the companion t-tests are secondary checks rather than circular confirmations. Study 2 constructs compound prompts from prompts that looked promising in Study 1's secondary analysis and then tests them on the same 100 questions. This is a selection-based overfitting concern and weakens claims of independent replication, but it is not circular: the Study 2 forecasts are still evaluated against the same externally resolved outcomes, and the analysis does not use the Study 2 results to redefine or re-fit the Study 1 claim. The paper's self-citations (e.g., Schoenegger et al. 2024 for the long superforecaster persona, Karger et al. 2024 for ForecastBench) supply prompts or benchmark data; they are not invoked as proof that prompt engineering has no effect. No uniqueness theorem, ansatz, or prior same-author result is used to forbid alternatives or justify the null conclusion. The strongest caveat is inferential, not circular: non-significance is not equivalence, and the claim of 'minimal to nonexistent' effects goes beyond the preregistered test. But that is an overstatement risk, not a derivation that reduces to its inputs. No circular step can be quoted or exhibited from the paper's own equations.
Assumptions & free parameters
assumptions (3)
- domain assumption Brier score is an appropriate measure of forecasting accuracy.
- domain assumption The sampled 100 ForecastBench questions are representative of forecasting tasks.
- domain assumption Forecasts requested as a single percentage 'Forecast: X%' capture the model's true capability.
Cite this review
Pith. "Pith review of Prompt Engineering Large Language Models' Forecasting Capabilities." pith.science (2026). https://pith.science/paper/OLSR3BJA
@misc{pith2026250601578,
author = {Pith},
title = {Pith review of: Prompt Engineering Large Language Models' Forecasting Capabilities},
year = {2026},
howpublished = {\url{https://pith.science/paper/OLSR3BJA}},
note = {Machine review of arXiv:2506.01578}
}
read the original abstract
Large language model performance can be improved in a large number of ways. Many such techniques, like fine-tuning or advanced tool usage, are time-intensive and expensive. Although prompt engineering is significantly cheaper and often works for simpler tasks, it remains unclear whether prompt engineering suffices for more complex domains like forecasting. Here we show that small prompt modifications rarely boost forecasting accuracy beyond a minimal baseline. In our first study, we tested 38 prompts across Claude 3.5 Sonnet, Claude 3.5 Haiku, GPT-4o, and Llama 3.1 405B. In our second, we introduced compound prompts and prompts from external sources, also including the reasoning models o1 and o1-mini. Our results show that most prompts lead to negligible gains, although references to base rates yield slight benefits. Surprisingly, some strategies showed strong negative effects on accuracy: especially encouraging the model to engage in Bayesian reasoning. These results suggest that, in the context of complex tasks like forecasting, basic prompt refinements alone offer limited gains, implying that more robust or specialized techniques may be required for substantial performance improvements in AI forecasting.
Forward citations
Cited by 1 Pith paper
-
Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs
BLF achieves state-of-the-art binary forecasting on ForecastBench by using linguistic belief states updated in tool-use loops, hierarchical multi-trial logit averaging, and hierarchical Platt scaling calibration.
Reference graph
Works this paper leans on
-
[1]
Analyze the question to determine any underlying factors or variables that might influence the estimate
-
[2]
Consider relevant data, trends, or knowledge that might influence the probability
-
[3]
After completing your analysis, provide your forecast as a percentage between 0% and 100%. 4. Format your final response exactly as follows: 'Forecast: X%'. Confidence: [Low/Medium/High] Where X is your numerical estimate. Example output (using a generic question): Forecast: 75% Confidence: Medium Remember to use your expertise and critical thinking skill...
-
[4]
Will it rain tomorrow in New York?
Based on your reasoning, assign a probabilistic estimate to the question, expressed as a percentage. # Output Format Your response should be formatted as: `Forecast: X%`, where X is the probabilistic estimate ranging from 0% to 100%. # Examples **Input:** "Will it rain tomorrow in New York?" **Output:** "Forecast: 40%" # Question {Question} Automatic prom...
arXiv 2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.