{"id":"06b2cfc1-754c-4e1f-9caa-6491821212b7","arxiv_id":"2501.04880","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A log-probability-weighted LLM forecast aggregator achieves Brier 0.186 on a small backtest, but the improvement over baseline is not statistically tested or isolated.","lead":"This paper describes an LLM-based forecasting pipeline that uses token log-probability weights to aggregate probability estimates, reporting a Brier score of 0.186 on a 150-item backtest. The result beats a plain GPT-4o baseline, but lacks statistical validation, ablations, and released code.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported Brier gain is not isolated to log-probability weighting; without an ablation that holds retrieval, trend, and calibration steps fixed, Eq. (1) is not shown to be the cause.","rationale":"The reader's conditional verdict is appropriate. The strongest claim is that logprob weighting yields a 0.186 Brier score, but the paper never isolates that mechanism. The pipeline has many active ingredients (trend retrieval, news search, event extraction, exclusivity checks, SVR calibration), and the baseline is vanilla GPT-4o without any of them. Thus the reported improvement could be due to retrieval or calibration rather than the novel logprob weighting. The SVR calibration is a particular concern because the train/test split is 50/50 on a small set, with no hyperparameters or variance estimates, so overfitting could produce the favorable score. The proposed ablation directly targets the only novel component: replacing the logprob-weighted average with uniform or top-token controls, while keeping all other steps fixed, would determine whether Eq. (1) is responsible. Since the reader already identified the same gap and recommended conditional acceptance pending additional evidence, my read does not change the verdict. This is not a rejection: the method may be sound, but the current evidence is insufficient to support the specific logprob-attribution claim, and the proposed check would settle whether the concern lands.","tokens_in":4466,"tokens_out":3182,"duration_ms":32224,"concrete_test":"Hold every pipeline component fixed and re-run the 72-event test set replacing the logprob-weighted average in Eq. (1) with (a) a uniform average over the same sampled probability tokens and (b) the top-token probability alone. Also compute 95% bootstrap confidence intervals over events for each variant and for the claimed 0.186. If either control matches 0.186 within about 0.01 Brier, or the confidence intervals overlap substantially, then log-probability weighting is not the driver of the improvement; if the weighted version is significantly better than both controls, the central claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that exploiting logprobs in Eq. (1) produces a 0.186 Brier score—is underdetermined by the evaluation. The Probability Estimator differs from the vanilla GPT-4o baseline in many components simultaneously: query reformulation, trend retrieval, news-source semantic search, event extraction, exclusivity checking, and SVR calibration (Sec. III-B, III-D). The reported 0.186 vs. 0.236 comparison therefore measures the whole pipeline, not the log-prob weighting. Since the paper's novelty claim is specifically about using logprobs to compute the final probability, the result would hold only if the weighting term is responsible for the improvement. No ablation, bootstrap interval, or significance test is given, and the test set is 72 events. A second confound: the SVR calibration is fit on a training half and applied to the test half, but no hyperparameters, cross-validation, or calibration sensitivity analysis is reported, so the 0.186 could reflect an overfit transform rather than the weighting mechanism. The numerical inconsistency (0.186 vs. 0.236 is about a 21% improvement, not 19%) further weakens precision, though it is not the core issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LLM-based forecasting pipeline that generates forecasts from topic names, estimates event probabilities via a weighted average of token probabilities (log-probabilities) with Eq. (1), optionally calibrates these estimates with a Support Vector Regression, and fact-checks outcomes automatically. The authors report a Brier score of 0.186 on a test set of 72 forecasts, claiming a 26% improvement over random chance and a 19% improvement over a vanilla GPT-4o baseline. The paper argues this is the first LLM forecasting approach that exploits log-probabilities to compute the final probability value.","tokens_in":4662,"tokens_out":2152,"duration_ms":22611,"significance":"If the reported result is robust, the paper would provide a practical demonstration that log-probability weighting and uncertainty estimation improve LLM forecasting, an area of active research. The authors should be credited for constructing a time-separated backtest dataset (forecasts generated February 2024, fact-checked October 2024), using a standard metric (Brier score), and reporting calibration plots. The main weakness is that the headline improvement is not attributable to the log-probability mechanism because the compared systems differ in multiple components and no statistical support is provided for the difference.","major_comments":[{"comment":"The central claim—that exploiting log-probabilities in Eq. (1) yields the 0.186 Brier score—is not supported by the reported comparison. The Probability Estimator pipeline includes six steps (query reformulation, trend search, semantic news retrieval, event extraction, exclusivity checking, and SVR calibration), while the baseline is a single vanilla GPT-4o prompt. The 0.186 vs. 0.236 comparison therefore measures the entire pipeline, not the log-probability weighting term. An ablation is required that holds all other components fixed and toggles only the log-prob weighting, or at least a baseline that includes the same retrieval and calibration steps but computes probabilities without Eq. (1). Without such an ablation, the novelty claim in §III-B is underdetermined.","section":"§III-B, §IV, Table I"},{"comment":"The SVR calibration step is a supervised transform fit on a training split and applied to a test split, but no hyperparameters, cross-validation procedure, or sensitivity analysis is reported. With roughly 72 training examples, the calibrated output could reflect overfitting of the SVR rather than the log-probability mechanism. The paper should report the calibration model details, demonstrate via cross-validation or a separate calibration analysis that the transform generalizes, and ideally show that the log-prob-derived uncertainty values are what the SVR uses to improve calibration.","section":"§III-D"},{"comment":"The reported improvements lack any measure of uncertainty. With a test set of only 72 events, the difference between 0.186 and 0.236 (or 0.250) may not be statistically significant. The paper should provide bootstrap confidence intervals, a paired significance test (e.g., Wilcoxon or paired t-test on per-event Brier contributions), or at least a sensitivity analysis over the train/test split. Additionally, the stated '+19% improvement' is inconsistent with the numbers in Table I: (0.236 − 0.186) / 0.236 ≈ 0.212, i.e., about 21%, not 19%. The +26% figure versus random chance (0.250) is consistent, but the abstract and §IV should be corrected to avoid reporting an inaccurate percentage.","section":"§IV, Table I"},{"comment":"The Fact Checker is used both to filter 'already happened' events and to determine outcomes for evaluation. The paper reports a manual reliability check of 150/150, but provides no details on how the manual labels were obtained, whether the manual labels are independent of the model’s output, or whether the Fact Checker’s automatic labels were used in the final Brier calculation. If the same model that estimates probabilities also labels outcomes, there is a risk of correlated errors inflating calibration, and the evaluation should clarify the independence of the outcome labels from the forecasting model.","section":"§III-C, §IV"}],"minor_comments":[{"comment":"The notation in Eq. (1) is underspecified: the paper should define exactly what the guesses Pi are, how they are obtained from the model’s token distribution, and what range of values Pi can take, since this is central to reproducing the method.","section":"§III-B, Eq. (1)"},{"comment":"The paper states that forecasts were generated in February 2024 and fact-checked in October 2024, but does not report the exact date of probability estimation. If probability estimation occurred after February 2024, the model could have seen intervening information; the timeline should be specified precisely.","section":"§III-A, §IV"},{"comment":"Reference [6] is described as 'very recently published' with a demo no longer available; the paper would benefit from a citation to the published version or a note on its availability, as readers cannot verify the claimed superhuman forecasting results.","section":"§II, Related Work"},{"comment":"There are minor formatting issues, including the broken word 'F oresight' in §I and the use of 'T able' instead of 'Table' in the caption of Table I; these should be corrected in revision.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short industry report with a plausible but unproven central claim. The main scientific issue is attribution: the headline Brier improvement is presented as evidence for log-probability weighting, yet the comparison conflates multiple design choices. This is fixable with an ablation, but the current manuscript would not convince a careful reader that Eq. (1) is responsible for the gain. The lack of confidence intervals and any calibration details further weaken the empirical case. I would encourage the authors to provide the ablation, the SVR details, and a corrected percentage, and to consider whether a larger test set or a paired baseline is feasible. The paper is within scope for the journal if the authors are willing to substantially strengthen the evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. The new thing is using logprobs as weights when averaging an LLM's probability guesses (Eq. 1). That looks new in the forecasting literature, and it's a natural idea. The pipeline around it (trend retrieval, news search, event extraction, SVR calibration) is built from existing pieces, but the combination is described clearly enough to reproduce. The 0.186 Brier score on 72 test events is plausible, but the paper doesn't show the logprob weighting is what causes it.\n\nWhat's good: the dataset is a real artifact—150 forecasts over 15 topics, generated in February 2024 and fact-checked in October 2024, with trends pulled from ~45k articles. The calibration plots give a useful visual. The authors also acknowledge the Manifold comparison isn't apples-to-apples. That's honest.\n\nThe soft spots are significant. The comparison is the full pipeline versus vanilla GPT-4o, and the logprob weighting is one of many differences: query reformulation, trend retrieval, news search, event extraction, exclusivity checks, and SVR calibration all change at once. No ablation holds those fixed and varies only the weighting, so the 0.186 vs 0.236 gap could come from any of them. The test set is 72, and there are no confidence intervals or significance tests. The SVR calibration hyperparameters are undisclosed, and the training split is only 78 forecasts, so an overfit transform is a real possibility. Minor arithmetic slip: 0.186 vs 0.236 is about a 21% improvement, not 19%. Worth fixing.\n\nOne more thing: the fact-checker is used both to discard forecasts that already happened and to label outcomes. That's a mild circularity—probably acceptable for a validity filter, but the same component determines the evaluation labels.\n\nThe stress-test note is on target: without an ablation, Eq. (1) isn't shown to be the cause. That said, I'd still send this to peer review. The idea is worth taking seriously, and the evaluation flaws are fixable. A good referee would ask for an ablation that isolates the weighting, bootstrap intervals on the Brier difference, and reported calibration details. If the weighting doesn't survive that, the paper still works as a pipeline description. It deserves a serious referee.","headline":"Logprob weighting is a new idea, but the paper's evidence doesn't isolate it—needs ablations and intervals before the headline Brier claim is trustworthy.","tokens_in":5211,"tokens_out":2982,"would_cite":true,"duration_ms":27005,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an LLM forecasting pipeline, which weights all candidate probability guesses by their token log-probabilities and then calibrates them with a regression model, achieves a Brier score of 0.186 on 72 held-out…","keywords":["artificial intelligence","probabilistic models","large language models","forecasting","foresight","futures studies","log probabilities","Brier score"],"falsifier":"A reader could falsify the attribution by rerunning the same 72 forecast-event pipeline with the log-probability weights in Eq. (1) replaced by uniform weights or by the top-token probability; if the Brier score stays at 0.186 or the gap to vanilla GPT-4o vanishes, the log-probability mechanism is not the cause. Bootstrapping the 72 test outcomes would also settle whether the 0.25 random-chance baseline lies outside the confidence interval for the reported 0.186.","tokens_in":4229,"feed_emoji":"📈","tokens_out":6317,"duration_ms":53588,"temperature":0.7,"pith_summary":"This paper argues that an LLM forecasting pipeline can generate new future-event forecasts from current trends and assign them probabilities whose accuracy beats random chance and a vanilla GPT-4o baseline. The engine is a log-probability weighting: instead of reading only the model's top completion, the system averages every probability guess the model offers, weighting each by its token logprob, and uses the weighted standard deviation as an uncertainty estimate. After a support-vector regression calibration step, the pipeline reports a Brier score of 0.186 on 72 held-out forecasts, which the authors interpret as a 26% improvement over random chance and 19% over uncalibrated GPT-4o. If correct, this would be a practical demonstration that token-level confidence signals in LLMs carry usable information for autonomous foresight, not just for text generation.","feed_headline":"Log-probability weighting lifts LLM forecast accuracy by 26 percent","feed_subtitle":"A 72-event backtest lands the calibrated pipeline at Brier 0.186, beating chance and vanilla GPT-4o","key_machinery":"The mechanism is the log-probability-weighted average of all candidate probability guesses an LLM produces: $$\\hat{P}=\\frac{\\sum_i $e^{{w_i}}$P_i}{\\sum_i $e^{{w_i}}$}$$ with the weighted standard deviation $$\\hat{U}=\\sqrt{\\frac{\\sum_i $e^{{w_i}}$(P_i-\\hat{P})^2}{\\sum_i $e^{{w_i}}$}}$$ as the uncertainty estimate, where $w_i$ is the logprob of the token representing guess $P_i$. This converts the model's token-level confidence into a scalar forecast and an uncertainty value. A Support Vector Regression model then learns a transformation from raw $(\\hat{P},\\hat{U})$ outputs to calibrated probabilities, trained on half of 150 forecasts and evaluated on the remaining 72, with forecasts generated in February 2024 and fact-checked in October 2024.","core_discovery":"The paper's central claim is that a forecasting pipeline using log-probability weighting achieves a Brier score of 0.186 on a 72-item test set, a 26% improvement over random chance (0.250) and a 19% improvement over vanilla GPT-4o (0.236), and that this is the first LLM forecasting approach to use logprobs to compute the final probability. The system generates forecasts from current trends rather than requiring a human to specify events, then estimates probabilities via a multi-step probability estimator. The authors interpret the result as evidence that trend knowledge and uncertainty values provide strategic advantage, while noting the system is not yet at superforecaster level.","pith_inferences":["A natural test the paper does not run is an ablation that replaces the logprob weights in Eq. (1) with uniform weights or with the top-token probability; without it, the gain cannot be uniquely attributed to logprobs rather than to averaging many guesses.","If the result holds across larger and more diverse event sets, the same weighted-average trick could improve calibration in other LLM tasks that ask for verbal probabilities, such as medical triage or multiple-choice reasoning.","The comparison with Manifold's 480-day markets is not a matched test; a direct head-to-head would score identical forecasts on markets with the same duration before claiming parity with prediction markets."],"forward_implications":["A calibrated logprob-weighted LLM forecaster can beat random chance and an uncalibrated widely-available AI on a 72-event backtest.","The reported 0.186 Brier score sits within the range of prediction markets (0.100–0.200) and within 0.036 points of the average superforecaster benchmark (0.150).","A Fact Checker with a perfect manual score of 150/150 allows the pipeline to evaluate forecasts automatically over time without human verification.","Because the Forecast Generator invents events from current trends, the same pipeline can produce multi-forecast scenarios and event probabilities for simulation agents, not just scores for human-supplied events."],"supporting_citations":[{"why":"Supplies the GPT pretraining architecture that underlies the language-model capability the system exploits.","marker":"[7]"},{"why":"Supplies the trend-extraction method the Forecast Generator builds on to obtain current trend data.","marker":"[10]"},{"why":"Establishes the retrieval-augmented LLM forecasting approach and Brier-score evaluation convention that this pipeline follows.","marker":"[1]"},{"why":"Provides the LLM-ensemble result that the paper uses as context for claiming forecasting accuracy near human crowds.","marker":"[8]"},{"why":"Is the GPT-4o system card, the model used both inside the Probability Estimator and as the vanilla baseline.","marker":"[3]"},{"why":"Another LLM forecasting framework with tools, cited as related work on which Brier comparisons are drawn.","marker":"[2]"}],"fun_headline_variants":["Log-probability boosts LLM forecast accuracy by 26%","LLM logprobs beat random chance by 26%","Logprobs give LLM forecasts a 26% edge over chance","Log-probability weighting improves LLM forecasts to Brier 0.186"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the Brier-score gap on the 72-item test set comes from the log-probability weighting and the calibration step, rather than from noise, dataset construction, or overfitting of the calibration model.","fun_headline_variants_meta":{"raw":{"variants":["Log-probability boosts LLM forecast accuracy by 26%","LLM logprobs beat random chance by 26%","Logprobs give LLM forecasts a 26% edge over chance","Log-probability weighting improves LLM forecasts to Brier 0.186"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001301,"raw_usage":{"total_tokens":5240,"prompt_tokens":808,"completion_tokens":4432,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":424,"completion_tokens_details":{"reasoning_tokens":4354}},"tokens_in":424,"tokens_out":4432,"duration_ms":28910,"temperature":1.0,"reasoning_tokens":4354,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:22:40.202149+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could falsify the attribution by rerunning the same 72 forecast-event pipeline with the log-probability weights in Eq. (1) replaced by uniform weights or by the top-token probability; if the Brier score stays at 0.186 or the gap to vanilla GPT-4o vanishes, the log-probability mechanism is not the cause. Bootstrapping the 72 test outcomes would also settle whether the 0.25 random-chance baseline lies outside the confidence interval for the reported 0.186.","supporting_citations":[{"cited_title":"and Fenn, J., 2003","cited_arxiv_id":null,"evidence_quote":"Supplies the trend-extraction method the Forecast Generator builds on to obtain current trend data."},{"cited_title":"Foundations of futures studies, volume 1: Human science for a new era (Vol","cited_arxiv_id":null,"evidence_quote":"Establishes the retrieval-augmented LLM forecasting approach and Brier-score evaluation convention that this pipeline follows."},{"cited_title":"and Beam, A., 2023","cited_arxiv_id":null,"evidence_quote":"Provides the LLM-ensemble result that the paper uses as context for claiming forecasting accuracy near human crowds."},{"cited_title":"The art of conjecture","cited_arxiv_id":null,"evidence_quote":"Another LLM forecasting framework with tools, cited as related work on which Brier comparisons are drawn."}],"review_version":1}