Pith. sign in

REVIEW 4 major objections 4 minor 20 references

Leveraging Log Probabilities in Language Models to Forecast Future Events

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that an LLM forecasting pipeline, which weights all candidate probability guesses by their token log-probabilities and then calibrates them with a regression model, achieves a Brier score of 0.186 on 72 held-out…

desk verdict Logprob weighting is a new idea, but the paper's evidence doesn't isolate it—needs ablations and intervals before the headline Brier claim is trustworthy. read the letter →

arxiv 2501.04880 v1 pith:KTNROTK5 submitted 2025-01-08 cs.CL cs.LG

Tommaso Soru , Jim Marshall This is my paper · ORCID
classification cs.CLcs.LG
keywords artificialintelligenceprobabilisticmodelslargelanguageforecastingforesightfuturesstudieslogprobabilitiesBrierscore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that an LLM forecasting pipeline can generate new future-event forecasts from current trends and assign them probabilities whose accuracy beats random chance and a vanilla GPT-4o baseline. The engine is a log-probability weighting: instead of reading only the model's top completion, the system averages every probability guess the model offers, weighting each by its token logprob, and uses the weighted standard deviation as an uncertainty estimate. After a support-vector regression calibration step, the pipeline reports a Brier score of 0.186 on 72 held-out forecasts, which the authors interpret as a 26% improvement over random chance and 19% over uncalibrated GPT-4o. If correct, this would be a practical demonstration that token-level confidence signals in LLMs carry usable information for autonomous foresight, not just for text generation.

What carries the argument

The mechanism is the log-probability-weighted average of all candidate probability guesses an LLM produces: $$\hat{P}=\frac{\sum_i $e^{{w_i}}$P_i}{\sum_i $e^{{w_i}}$}$$ with the weighted standard deviation $$\hat{U}=\sqrt{\frac{\sum_i $e^{{w_i}}$(P_i-\hat{P})^2}{\sum_i $e^{{w_i}}$}}$$ as the uncertainty estimate, where $w_i$ is the logprob of the token representing guess $P_i$. This converts the model's token-level confidence into a scalar forecast and an uncertainty value. A Support Vector Regression model then learns a transformation from raw $(\hat{P},\hat{U})$ outputs to calibrated probabilities, trained on half of 150 forecasts and evaluated on the remaining 72, with forecasts generated in February 2024 and fact-checked in October 2024.

What would settle it

A reader could falsify the attribution by rerunning the same 72 forecast-event pipeline with the log-probability weights in Eq. (1) replaced by uniform weights or by the top-token probability; if the Brier score stays at 0.186 or the gap to vanilla GPT-4o vanishes, the log-probability mechanism is not the cause. Bootstrapping the 72 test outcomes would also settle whether the 0.25 random-chance baseline lies outside the confidence interval for the reported 0.186.

Watch

Extended reading notes

Core claim

The paper's central claim is that a forecasting pipeline using log-probability weighting achieves a Brier score of 0.186 on a 72-item test set, a 26% improvement over random chance (0.250) and a 19% improvement over vanilla GPT-4o (0.236), and that this is the first LLM forecasting approach to use logprobs to compute the final probability. The system generates forecasts from current trends rather than requiring a human to specify events, then estimates probabilities via a multi-step probability estimator. The authors interpret the result as evidence that trend knowledge and uncertainty values provide strategic advantage, while noting the system is not yet at superforecaster level.

Load-bearing premise

The central claim rests on the assumption that the Brier-score gap on the 72-item test set comes from the log-probability weighting and the calibration step, rather than from noise, dataset construction, or overfitting of the calibration model.

Editorial extensions

If this is right

  • A calibrated logprob-weighted LLM forecaster can beat random chance and an uncalibrated widely-available AI on a 72-event backtest.
  • The reported 0.186 Brier score sits within the range of prediction markets (0.100–0.200) and within 0.036 points of the average superforecaster benchmark (0.150).
  • A Fact Checker with a perfect manual score of 150/150 allows the pipeline to evaluate forecasts automatically over time without human verification.
  • Because the Forecast Generator invents events from current trends, the same pipeline can produce multi-forecast scenarios and event probabilities for simulation agents, not just scores for human-supplied events.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test the paper does not run is an ablation that replaces the logprob weights in Eq. (1) with uniform weights or with the top-token probability; without it, the gain cannot be uniquely attributed to logprobs rather than to averaging many guesses.
  • If the result holds across larger and more diverse event sets, the same weighted-average trick could improve calibration in other LLM tasks that ask for verbal probabilities, such as medical triage or multiple-choice reasoning.
  • The comparison with Manifold's 480-day markets is not a matched test; a direct head-to-head would score identical forecasts on markets with the same duration before claiming parity with prediction markets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes an LLM-based forecasting pipeline that generates forecasts from topic names, estimates event probabilities via a weighted average of token probabilities (log-probabilities) with Eq. (1), optionally calibrates these estimates with a Support Vector Regression, and fact-checks outcomes automatically. The authors report a Brier score of 0.186 on a test set of 72 forecasts, claiming a 26% improvement over random chance and a 19% improvement over a vanilla GPT-4o baseline. The paper argues this is the first LLM forecasting approach that exploits log-probabilities to compute the final probability value.

Significance. If the reported result is robust, the paper would provide a practical demonstration that log-probability weighting and uncertainty estimation improve LLM forecasting, an area of active research. The authors should be credited for constructing a time-separated backtest dataset (forecasts generated February 2024, fact-checked October 2024), using a standard metric (Brier score), and reporting calibration plots. The main weakness is that the headline improvement is not attributable to the log-probability mechanism because the compared systems differ in multiple components and no statistical support is provided for the difference.

major comments (4)
  1. [§III-B, §IV, Table I] The central claim—that exploiting log-probabilities in Eq. (1) yields the 0.186 Brier score—is not supported by the reported comparison. The Probability Estimator pipeline includes six steps (query reformulation, trend search, semantic news retrieval, event extraction, exclusivity checking, and SVR calibration), while the baseline is a single vanilla GPT-4o prompt. The 0.186 vs. 0.236 comparison therefore measures the entire pipeline, not the log-probability weighting term. An ablation is required that holds all other components fixed and toggles only the log-prob weighting, or at least a baseline that includes the same retrieval and calibration steps but computes probabilities without Eq. (1). Without such an ablation, the novelty claim in §III-B is underdetermined.
  2. [§III-D] The SVR calibration step is a supervised transform fit on a training split and applied to a test split, but no hyperparameters, cross-validation procedure, or sensitivity analysis is reported. With roughly 72 training examples, the calibrated output could reflect overfitting of the SVR rather than the log-probability mechanism. The paper should report the calibration model details, demonstrate via cross-validation or a separate calibration analysis that the transform generalizes, and ideally show that the log-prob-derived uncertainty values are what the SVR uses to improve calibration.
  3. [§IV, Table I] The reported improvements lack any measure of uncertainty. With a test set of only 72 events, the difference between 0.186 and 0.236 (or 0.250) may not be statistically significant. The paper should provide bootstrap confidence intervals, a paired significance test (e.g., Wilcoxon or paired t-test on per-event Brier contributions), or at least a sensitivity analysis over the train/test split. Additionally, the stated '+19% improvement' is inconsistent with the numbers in Table I: (0.236 − 0.186) / 0.236 ≈ 0.212, i.e., about 21%, not 19%. The +26% figure versus random chance (0.250) is consistent, but the abstract and §IV should be corrected to avoid reporting an inaccurate percentage.
  4. [§III-C, §IV] The Fact Checker is used both to filter 'already happened' events and to determine outcomes for evaluation. The paper reports a manual reliability check of 150/150, but provides no details on how the manual labels were obtained, whether the manual labels are independent of the model’s output, or whether the Fact Checker’s automatic labels were used in the final Brier calculation. If the same model that estimates probabilities also labels outcomes, there is a risk of correlated errors inflating calibration, and the evaluation should clarify the independence of the outcome labels from the forecasting model.
minor comments (4)
  1. [§III-B, Eq. (1)] The notation in Eq. (1) is underspecified: the paper should define exactly what the guesses Pi are, how they are obtained from the model’s token distribution, and what range of values Pi can take, since this is central to reproducing the method.
  2. [§III-A, §IV] The paper states that forecasts were generated in February 2024 and fact-checked in October 2024, but does not report the exact date of probability estimation. If probability estimation occurred after February 2024, the model could have seen intervening information; the timeline should be specified precisely.
  3. [§II, Related Work] Reference [6] is described as 'very recently published' with a demo no longer available; the paper would benefit from a citation to the published version or a note on its availability, as readers cannot verify the claimed superhuman forecasting results.
  4. [Throughout] There are minor formatting issues, including the broken word 'F oresight' in §I and the use of 'T able' instead of 'Table' in the caption of Table I; these should be corrected in revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Brier score is an out-of-sample empirical measurement, and the log-prob weighting plus SVR calibration do not reduce to the test outcomes by construction.

full rationale

The paper's central quantity is an empirical Brier score computed on a held-out test set, not a quantity that is equal to its inputs by construction. The calibration step is explicitly out-of-sample: 'we split the dataset equally into training and test set, retaining a test set of N = 72 elements' and 'We employ a Support Vector Regression model to find the optimal transformation function and ultimately calibrate the model.' The reported 0.186 is therefore a fitted transform applied to held-out events, not a refit of the test labels. Equation (1) defines the Probability Estimator in terms of log-probability weights; it is a design choice, and the subsequent Brier score is an independent measurement against real, fact-checked outcomes. The Fact Checker is used both to filter invalid forecasts and to determine outcomes, but the paper claims a manual audit ('We manually evaluated the reliability of the Fact Checker; having achieved a perfect score of 150/150 on our dataset'); any residual dependence would be an evaluation-reliability concern, not a definitional identity. The only self-citation, [10], supplies the trend-extraction component ('Building on top of previous research on LLM-driven trend extraction and analysis [10]'); that component is an input to forecast generation, not to the probability-estimation claim, so the self-citation is not load-bearing. The paper's own limitations ('we could not find any system to compare against', 'these last results are not directly comparable') are benchmark-availability caveats, not circularity. The numerical mismatch between the claimed +19% improvement and the actual ~21% from Table I is a reporting error. No step in the derivation reduces the result to its inputs.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the usefulness of log-probability weighting, the reliability of the fact-checker, and the representativeness of the small training split. None of these assumptions are isolated or validated beyond the single reported Brier score.

free parameters (2)
  • SVR hyperparameters
    Kernel, C, epsilon, and gamma used in the Support Vector Regression calibration model (Section III-D) are not reported; they are presumably tuned on the training split, making them free parameters.
  • Trend relevance filtering threshold
    The semantic filter in the Forecast Generator (Section III-A) selects trends relevant to the topic; the threshold or similarity measure is not specified, so it is a hand-chosen parameter.
assumptions (5)
  • domain assumption Log probabilities of token guesses are meaningful weights for combining probability estimates.
    Section III-B uses exp(w_i) as weights in equations (1) and (2) without theoretical justification or empirical validation that this weighting scheme is appropriate.
  • domain assumption GPT-4o's token-level probability guesses are reasonable candidate probabilities for events.
    The Probability Estimator (Section III-B) assumes the model's outputs, before weighting, are valid probability estimates that can be averaged and calibrated.
  • domain assumption The Fact Checker is reliable enough to detect both invalid forecasts and event outcomes.
    Section III-C reports a manual evaluation of 150/150 on the dataset, but the fact-checker is also used to discard invalid forecasts and to determine outcomes for Brier scoring, so its reliability is load-bearing.
  • domain assumption The training set is representative of the test set for SVR calibration.
    Section III-D splits the remaining forecasts into a training and test set of 72 each; the assumption of stationarity across the small dataset is untested.
  • domain assumption Events are binary with determinable true outcomes.
    The Brier score (Section IV) treats outcomes as 0 or 1, but 'inconclusive' fact-check results are not defined in the evaluation; the paper implicitly assumes all events can be clearly judged.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging Log Probabilities in Language Models to Forecast Future Events." pith.science (2026). https://pith.science/paper/KTNROTK5

@misc{pith2026250104880,
  author       = {Pith},
  title        = {Pith review of: Leveraging Log Probabilities in Language Models to Forecast Future Events},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KTNROTK5}},
  note         = {Machine review of arXiv:2501.04880}
}
read the original abstract

In the constantly changing field of data-driven decision making, accurately predicting future events is crucial for strategic planning in various sectors. The emergence of Large Language Models (LLMs) marks a significant advancement in this area, offering advanced tools that utilise extensive text data for prediction. In this industry paper, we introduce a novel method for AI-driven foresight using LLMs. Building on top of previous research, we employ data on current trends and their trajectories for generating forecasts on 15 different topics. Subsequently, we estimate their probabilities via a multi-step approach based on log probabilities. We show we achieve a Brier score of 0.186, meaning a +26% improvement over random chance and a +19% improvement over widely-available AI systems.

Figures

Figures reproduced from arXiv: 2501.04880 by the authors.

Figure 1
Figure 1. An example of forecast generation. useful for forecasting and decision intelligence” [5]. We aim to challenge this view. The scope of this paper also extends to the application of LLMs in the context of Futures Studies – also known as Foresight – a discipline that is highly focused on evaluating the plausibility of different future scenarios. Typically, this task is performed manually by experts through detailed and… view at source ↗
Figure 2
Figure 2. An example of probability estimation. improvement over random chance and a +19% improvement over widely-available AI systems. This paper is organised as follows. Related work is intro￾duced in section II. We describe the approach in section III. We discuss results in section IV. Finally, we conclude. II. RELATED WORK The advancement of GPT models has recently trans￾formed natural language processing by introducing e… view at source ↗
Figure 3
Figure 3. Calibration plot for our Probability Estimator (above) and vanilla [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Probability estimation performance by topic. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 12 canonical work pages

  1. [1]

    Foundations of futures studies, volume 1: Human science for a new era (Vol

    Bell, W., 2009. Foundations of futures studies, volume 1: Human science for a new era (Vol. 1). Transaction publishers

    work page 2009

  2. [2]

    The art of conjecture

    De Jouvenel, B., 2017. The art of conjecture. Routledge

    work page 2017

  3. [3]

    et al., 2023

    Feng, Z., Ma, W., Yu, W. et al., 2023. Trends in integration of knowledge and large language models: A survey and taxonomy of methods, benchmarks, and applications. arXiv preprint arXiv:2311.05876

    arXiv 2023

  4. [4]

    and Steinhardt, J., 2024

    Halawi, D., Zhang, F., Yueh-Han, C. and Steinhardt, J., 2024. Approaching Human-Level Forecasting with Language Models. arXiv preprint arXiv:2402.18563

    arXiv 2024

  5. [5]

    and Chen, W., 2020

    He, P., Liu, X., Gao, J. and Chen, W., 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654

    arXiv 2020

  6. [6]

    and Chen, J., 2024

    Hsieh, E., Fu, P. and Chen, J., 2024. Reasoning and Tools for Human-Level Forecasting. arXiv preprint arXiv:2408.12036

    arXiv 2024

  7. [7]

    and Madry, A., 2024

    Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A.J., Welihinda, A., Hayes, A., Radford, A. and Madry, A., 2024. GPT-4o System Card. arXiv preprint arXiv:2410.21276

    arXiv 2024

  8. [8]

    and Beam, A., 2023

    Levine, D.M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S.G., Mehrotra, A. and Beam, A., 2023. The diagnostic and triage accuracy of the GPT-3 artificial intelligence model. MedRxiv

    work page 2023

Show all 20 references
  1. [9]

    et al., 2019

    Lewis, M., Liu, Y., Goyal, N. et al., 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461

    2019 arXiv

  2. [10]

    and Fenn, J., 2003

    Linden, A. and Fenn, J., 2003. Understanding Gartner’s hype cycles. Strategic Analysis Report Nº R-20-1971. Gartner, Inc, 88, p.1423

    2003

  3. [11]

    When Not to Use Generative AI

    McCartney A., 2024. When Not to Use Generative AI. Gartner, Inc

    2024

  4. [12]

    and Astels, S., 2017

    McInnes, L., Healy, J. and Astels, S., 2017. hdbscan: Hierarchical density based clustering. J. Open Source Softw., 2(11), p.205

    2017

  5. [13]

    LLMs Are Superhuman Forecasters

    Phan L., Khoja A., Mazeika M., Hendrycks D., 2024. LLMs Are Superhuman Forecasters. Pre-print

    2024

  6. [14]

    and Sutskever, I., 2018

    Radford, A., Narasimhan, K., Salimans, T. and Sutskever, I., 2018. Improving language understanding by generative pre-training

    2018

  7. [15]

    and Tetlock, P.E., 2024

    Schoenegger, P., Tuminauskaite, I., Park, P.S. and Tetlock, P.E., 2024. Wisdom of the silicon crowd: LLM ensemble prediction capabilities match human crowd accuracy. arXiv preprint arXiv:2402.19379

    2024 arXiv

  8. [16]

    doi: 10.5281/zenodo.7908658

    Soru, T., 2023, Spotting Signals in Text via Natural Language Understanding, Knowledge Graph Conference (KGC), New York City. doi: 10.5281/zenodo.7908658

    2023 doi

  9. [17]

    and Marshall, J., 2024, February

    Soru, T. and Marshall, J., 2024, February. Trend Extraction and Analysis via Large Language Models. In 2024 IEEE 18th International Conference on Semantic Computing (ICSC) (pp. 285-288). IEEE

    2024

  10. [18]

    Future shock, 1970

    Toffler, A., 1970. Future shock, 1970. Sydney. Pan

    1970

  11. [19]

    et al., 2023

    Touvron, H., Martin, L., Stone, K. et al., 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  12. [20]

    c/N o h ğ gL# @| ũN 1 : WrR1ܻj6|N @*ǙN u+ʳ&/ c/_-|P @N :7 ě& *+f, Z5 \ o2' t ˵WI M >V c)/s `i v, t,E 1 7Q u n n +hq[ I. ' R' 낧S4 v257ei jɓE s|4

    11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEco...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.