Pith. sign in

REVIEW 3 major objections 5 minor 16 references

Can LLM Improve for Expert Forecast Combination? Evidence from the European Central Bank Survey

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A zero-shot LLM can beat equal-weighted expert forecasts for one-year-ahead ECB predictions when its prompt includes recent realized values.

desk verdict The LLM is given historical actuals the baseline never sees, so the headline result is confounded—and the paper's own ablation proves it. read the letter →

arxiv 2506.23154 v1 pith:U3R7LCGE submitted 2025-06-29 stat.AP

Yinuo Ren , Jue Wang This is my paper
classification stat.AP MSC 62P2062J0568T50
keywords LLMensembleforecastcombinationsimpleaveragingSurveyofProfessionalForecasterszero-shotlearningmacroeconomicforecastingexpertdisagreementforecasterinattentiveness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a large language model can improve on the hard-to-beat baseline of equal-weighted averaging when combining expert macroeconomic forecasts. Using the European Central Bank's Survey of Professional Forecasters, the authors prompt a zero-shot LLM with three quarters of expert predictions and realized values, asking it to weight experts by historical accuracy, compensate for lag, and incorporate recent trends. In mixed-effects regressions on log absolute percentage error, the LLM ensemble is significantly more accurate for one-year-ahead forecasts, with the effect concentrated in GDP growth and unemployment and in periods of high expert disagreement. If the result is right, it offers a training-free way to get better consensus forecasts from existing expert surveys.

What carries the argument

The central object is the engineered prompt that turns a frozen LLM into a forecast combiner. The prompt gives the model three quarters of historical true values and expert prediction sequences and instructs it to apply historical accuracy weighting, lag compensation, and trend enhancement, with a strict output format that yields a single ensemble number. This is evaluated by a mixed-effects regression of log absolute percentage error on an LLM-ensemble indicator, with the realized value as a control and standard errors clustered by year; the indicator's coefficient is the paper's measure of the LLM's edge over simple averaging.

What would settle it

Give the simple-averaging baseline the same three quarters of realized outcomes and a trend adjustment, or rerun the LLM prompt without realized values, and check whether the significant negative coefficient on the LLM indicator survives; if it disappears, the improvement is driven by the extra information, not by LLM combination skill.

Watch

Extended reading notes

Core claim

The authors find that a zero-shot LLM ensemble outperforms the simple average of expert forecasts for short-horizon ECB SPF predictions. The one-year-ahead regression yields a negative coefficient of -0.121 (p = 0.016) on the LLM indicator, meaning the LLM ensemble's log absolute percentage error is lower than simple averaging; the two-year-ahead coefficient is also negative (-0.063) but not significant. Indicator-level results show the advantage is significant for real GDP growth and the unemployment rate at the one-year horizon, while HICP inflation shows no clear edge. The interaction between the LLM indicator and high expert disagreement is negative and significant at one year (-0.184, p = 0.043), suggesting the method is more resilient when experts diverge. Ablation results show that removing any one of the three prompt components—historical accuracy weighting, lag compensation, or trend enhancement—eliminates the significant advantage, and the main effect is directionally reproduced with a different LLM, Deepseek.

Load-bearing premise

The comparison assumes the LLM's advantage comes from how it combines experts, not from the realized outcomes and trend information in its prompt, which the simple-averaging baseline never sees.

Editorial extensions

If this is right

  • If the finding holds, central banks and other users of expert surveys could improve one-year-ahead consensus forecasts without retraining or parameter tuning.
  • The significant negative disagreement interaction implies the LLM ensemble is especially valuable when expert opinions diverge sharply, a condition where simple averages are most fragile.
  • The ablation results imply the full prompt—accuracy weighting plus lag compensation plus trend enhancement—is necessary; stripping any component loses the advantage.
  • The concentration of benefits in GDP growth and unemployment, with no clear edge for HICP inflation, suggests the method should be targeted at series whose expert forecasts lag or have weaker coverage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fair test of combination skill would feed the simple average the same realized-outcome and trend information, for example by applying a damped trend adjustment to the equal-weighted mean; if the LLM's advantage disappears, the paper has shown the value of that information, not of LLM weighting.
  • Shuffling expert identities across the prompt would test whether the LLM actually learns which experts are accurate or merely anchors on recent realized trends.
  • The emotional-prompt results suggest sentiment framing could be used to generate scenario ranges rather than point forecasts, an application the paper gestures at but does not develop.
  • Comparing against robust combination rules such as trimmed means or medians would clarify whether the LLM's disagreement resilience is a qualitatively new mechanism or an approximation of standard outlier-robust estimators.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies whether a zero-shot LLM (Qwen, with Deepseek as a robustness check) can improve on simple averaging when combining forecasts from the ECB Survey of Professional Forecasters. The authors build a mixed-effects regression of log absolute percentage error on a Qwen dummy, the realized value, and various fixed effects and interactions. They report a significant negative coefficient for Qwen at the one-year horizon (Table 1, -0.121, p=0.016) and interpret this as evidence that the LLM ensemble outperforms simple averaging, especially for short-horizon GDP and unemployment forecasts. Additional analyses examine disagreement, inattentiveness, alternative error metrics, and an ablation of the prompt components. The central claim is that the LLM's combination skill drives the improvement.

Significance. If the central claim were established, the paper would offer a useful, low-cost method for combining expert forecasts and would speak to the growing literature on LLMs in economic forecasting. The zero-shot design is attractive, and the authors are transparent about their prompts and report an ablation study. However, the design does not isolate combination skill from access to additional information. The LLM prompt in Section 3.2 includes historical true values and explicitly instructs the model to perform lag compensation and trend enhancement using those realized values, while the simple-average baseline sees only the expert forecasts. The paper's own ablation (Tables 22-24) shows that removing any of the three realized-value-dependent prompt components eliminates the significant advantage. Thus the paper has not shown that LLMs are better at combining expert forecasts; it has shown that a model given three quarters of realized outcomes plus expert history can beat an average that sees only contemporaneous expert forecasts. Since this is the paper's central contribution, the significance of the result as reported is substantially undercut.

major comments (3)
  1. [Section 5.2 and Tables 22-24] The comparison between the LLM ensemble and simple averaging is not a comparison of combination methods because the information sets differ. The prompt in Section 3.2 gives the LLM "historical true values and expert prediction sequences for the past three quarters" and instructs it to perform "Lag Compensation" and "Trend Enhancement" using recent realized values, while the simple-average baseline is computed from expert forecasts alone. The control for the contemporaneous true value X_j in Eq. (2) does not give the baseline the three-quarter history of realizations that the LLM uses to estimate and correct expert biases and trends. Consequently, the negative Qwen coefficient in Table 1 (-0.121, p=0.016) may reflect the LLM's access to ex-post information rather than a superior way of weighting expert opinions.
  2. [Section 5.2, Tables 22-24] The ablation study is decisive against the paper's central claim. Removing Historical Accuracy Weighting (Table 22), Lag Compensation (Table 23), or Trend Enhancement (Table 24) from the prompt reduces the Qwen coefficient from -0.121 (p=0.016) to -0.041 (p=0.258), -0.030 (p=0.335), and -0.028 (p=0.364), respectively, in the log(APE) specification. All three removed components depend on realized historical values. This pattern shows that the significant improvement in the full prompt is driven by the realized-outcome information, not by the LLM's ability to combine expert forecasts per se. The conclusion in Section 6 that the LLM-ensemble "demonstrates superior performance compared to simple averaging" is therefore not supported by the reported experiments.
  3. [Table 4] There is an internal inconsistency in Table 4 that calls the reliability of the reported statistics into question. For HICP inflation at h=2, the coefficient on True is -0.212 with standard error 0.0214, giving an implied t-statistic of approximately -9.9, yet the table reports t = -0.99 and p = 0.331. Similar sign/statistic inconsistencies appear elsewhere (e.g., Table 5, True h=1, coefficient -0.339 with t = 2.94; Table 13, same coefficient with t = 4.03). The authors should correct all tables, report the exact regression specification and estimation commands, and ideally provide replication code and data.
minor comments (5)
  1. [Keywords, Abstract] The keywords field contains "quadrupole exciton polariton WGM BEC," which appears to be a template artifact unrelated to the paper. The abstract also contains template boilerplate ("This template helps you to create a properly formatted LATEX manuscript"). These should be removed.
  2. [Section 2.3] Hypothesis 3 labels both the accuracy-decline prediction and the LLM-superiority prediction as "H3a." The second should presumably be H3b.
  3. [Section 3.1, Eq. (2)-(3)] The dependent variable Y_ij is defined as APE, and Eq. (3) defines APE as |F_ij - X_j| / X_j. The regression uses log(Y_ij), but the text refers to "log(APE)" and "log(MAE)" without stating whether the logarithm is taken before or after averaging. This should be clarified.
  4. [Section 4.2, Table 3, Section 5.3] There are several typos and grammatical errors: "the Grwth GDP" in Table 3's title, "nad" for "and," "the growth GDP nad the unemployment rate" in Section 4.2, and "combiantion" in Section 5.3. The manuscript should be carefully proofread.
  5. [References, Section 3.3] The Bates and Granger reference is misspelled as "Bates and and, 1969" in the text and in the reference list. Also, the reference to Shi et al. (2023) in Section 3.3 does not support the stated zero-shot bound; the cited paper is about contrastive learning, not transformer generalization bounds for LLMs.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the LLM ensemble outputs are not fitted to the target values, and no result reduces by construction to its inputs.

full rationale

The claimed result is that a zero-shot LLM ensemble achieves lower log absolute percentage error than simple averaging, with the headline Qwen coefficient of -0.121 (p=0.016) for one-year-ahead forecasts. The derivation chain is: an engineered prompt containing historical expert forecasts and historical true values produces an ensemble forecast; the forecast error is computed against the realized value via Eq. (3); and a mixed-effects regression in Eq. (2) estimates a treatment indicator for LLM ensemble versus simple averaging. Nothing in this chain defines the LLM ensemble's error in terms of the estimated coefficient, and the LLM's weights are not optimized on the realized outcomes used to compute APE. The ablation results in Tables 22-24 show that removing prompt components makes the coefficient insignificant, but sensitivity to an input is not equivalence of output to input and does not make the prediction tautological. The only self-citation, Radchenko, Vasnev and Wang (2023), supports a background claim about correlated expert forecast errors and is not load-bearing for the central result. A genuine concern is that the prompt supplies historical realized values and explicit lag-compensation and trend-enhancement instructions that the simple-average baseline does not receive, creating an asymmetric information set; that is a validity and identification problem for the comparison, not a circularity. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on hand-chosen thresholds and on domain assumptions about LLM determinism, information symmetry, and error structure. The zero-shot generalization bound in Eq. (5) is decorative and not established for this task; no new entities are introduced.

free parameters (3)
  • Inattentiveness threshold (lambda_t > 0.1) = 0.1
    Chosen by hand in Section 4.4 to split high versus low inattentiveness, with no sensitivity analysis reported.
  • Historical window length in prompt = 3 quarters
    The prompt in Section 3.2 fixes the lookback window at three quarters; no cross-validation of this choice is provided.
  • Disagreement binary split = median level
    Section 4.3 splits disagreement at the median; alternative continuous specifications give different significance, as shown in Tables 14 through 17.
assumptions (4)
  • domain assumption LLM output can be treated as a deterministic observation for statistical inference.
    Invoked implicitly in Section 4.1 regressions; no temperature, seed, or repeated sampling is reported for Qwen or Deepseek calls.
  • ad hoc to paper Giving the LLM historical true values while denying them to the simple-average baseline is a fair comparison of forecast combination methods.
    Section 3.2 prompt includes historical true values for trend enhancement and lag compensation; simple averaging uses only expert forecasts.
  • domain assumption The error term epsilon_j in Eq. (2) is normally distributed and clustering at year level captures the dependence structure.
    Section 3.1 model specification; no diagnostic tests are reported.
  • domain assumption The zero-shot generalization bound in Eq. (5) applies to LLM forecast combination.
    Section 3.3 cites Shi et al. for a bound that is not established for this task and is not used in the empirical analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can LLM Improve for Expert Forecast Combination? Evidence from the European Central Bank Survey." pith.science (2026). https://pith.science/paper/U3R7LCGE

@misc{pith2026250623154,
  author       = {Pith},
  title        = {Pith review of: Can LLM Improve for Expert Forecast Combination? Evidence from the European Central Bank Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U3R7LCGE}},
  note         = {Machine review of arXiv:2506.23154}
}
read the original abstract

This study explores the potential of large language models (LLMs) to enhance expert forecasting through ensemble learning. Leveraging the European Central Bank's Survey of Professional Forecasters (SPF) dataset, we propose a comprehensive framework to evaluate LLM-driven ensemble predictions under varying conditions, including the intensity of expert disagreement, dynamics of herd behavior, and limitations in attention allocation.

Figures

Figures reproduced from arXiv: 2506.23154 by the authors.

Figure 1
Figure 1. One-year and Two-year Forecasts of Three Macroeconomic Indicators. where 𝑛𝑡 is the number of experts in period 𝑡, 𝑓𝑖,𝑡(𝑦𝑡+ℎ ) is the forecast value of the 𝑖-th expert in period 𝑡 for variable 𝑦 at horizon ℎ, and 𝐼(⋅) is an indicator function equal to 1 if the condition is true and 0 otherwise [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Evolution of Forecaster Inattentiveness Over Time. systematic approach to weight allocation helps mitigate the impact of varying levels of expert engagement in the forecasting process: Hypothesis 4 (H4): The accuracy of expert forecasts will be positively correlated with the level of forecaster attention (H4a), while LLM ensemble performance will demonstrate greater resilience to fluctuations in expert attention lev… view at source ↗
Figure 3
Figure 3. GDP Growth Emsemble Results under Different Sentiment Orientations 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 1 2 3 4 5 Positive Neutral Negative [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: HICP Inflation Emsemble Results under Different Sentiment Orientations The key innovation of our approach lies in the development of a zero-shot learning framework that requires no pre-training or parameter tuning. Through carefully engineered prompts incorporating his…
Figure 5
Figure 5. Figure 5: Unemployment Rate Emsemble Results under Different Sentiment Orientations we offer an immediately deployable solution that can enhance forecast accuracy without requiring extensive model training or parameter optimization. Future research could extend this work in seve…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 4 canonical work pages

  1. [6]

    arXiv preprint arXiv:2310.01728

    Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728 . Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.,

    arXiv

  2. [7]

    Journal of Macroeconomics 67, 103266.URL: https://www.sciencedirect.com/science/article/pii/S0164070420301907,doi: https://doi.org/10.1016/j

    The role of macroeconomic and policy uncertainty in density forecast dispersion. Journal of Macroeconomics 67, 103266.URL: https://www.sciencedirect.com/science/article/pii/S0164070420301907,doi: https://doi.org/10.1016/j. jmacro.2020.103266. Liu, C., Xu, Q., Miao, H., Yang, S., Zhang, L., Long, C., Li, Z., Zhao, R.,

    arXiv

  3. [8]

    arXiv preprint arXiv:2406.01638

    Timecma: Towards llm-empowered time series forecasting via cross-modality alignment. arXiv preprint arXiv:2406.01638 . Maharjan,J.,Garikipati,A.,Singh,N.P.,Cyrus,L.,Sharma,M.,Ciobanu,M.,Barnes,G.,Thapa,R.,Mao,Q.,Das,R.,2024. Openmedlm:prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models. Scien...

    arXiv 2024

  4. [9]

    arXiv preprint arXiv:2305.13661

    On the risk of misinformation pollution with large language models. arXiv preprint arXiv:2305.13661 . Radchenko, P., Vasnev, A.L., Wang, W.,

    arXiv

  5. [10]

    The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learning

    The trade-off between universality and label efficiency of representations from contrastive learning URL:https://arxiv.org/abs/2303.00106, arXiv:2303.00106. Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.,

    work page Pith review arXiv

  6. [12]

    JMIRMedInform12,e55318

    An empirical evaluation of prompting strategies for large languagemodelsinzero-shotclinicalnaturallanguageprocessing:Algorithmdevelopmentandvalidationstudy. JMIRMedInform12,e55318. URL: https://medinform.jmir.org/2024/1/e55318, doi:10.2196/55318. Stock, J.H., Watson, M.W.,

    doi:10.2196/55318 2024

  7. [14]

    arXiv preprint arXiv:2302.13971

    Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 . Vapnik, V.N., Chervonenkis, A.Y.,

    arXiv

  8. [16]

    arXiv preprint arXiv:2403.09057

    A continued pretrained llm approach for automatic medical note generation. arXiv preprint arXiv:2403.09057 . CV Radhakrishnan et al.:Preprint submitted to Elsevier Page 21 of 21

    arXiv

Show all 16 references
  1. [1969]

    Journal of the Operational Research Society 20, 451–468

    The combination of forecasts. Journal of the Operational Research Society 20, 451–468. URL:https: //doi.org/10.1057/jors.1969.103, doi:10.1057/jors.1969.103, arXiv:https://doi.org/10.1057/jors.1969.103. Bi,X.,Chen,D.,Chen,G.,Chen,S.,Dai,D.,Deng,C.,Ding,H.,Dong,K.,Du,Q.,Fu,Z.,e...

    1969 arXiv

  2. [1971]

    Theory of Probability & Its Applications 16, 264–280

    On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability & Its Applications 16, 264–280. URL: https://doi.org/10.1137/1116025, doi: 10.1137/1116025, arXiv:https://doi.org/10.1137/1116025. Yang,A.,Yang,B.,Zhang,B.,Hui,B.,Zheng,B...

    arXiv

  3. [2004]

    Journal of Forecasting 23, 405–430

    Combination forecasts of output growth in a seven-country data set. Journal of Forecasting 23, 405–430. doi:https://doi.org/10.1002/for.928. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.,

    doi

  4. [2013]

    Journal of Monetary Economics 60, 967–982

    Inattentive professional forecasters. Journal of Monetary Economics 60, 967–982. URL:https://www. sciencedirect.com/science/article/pii/S0304393213001190, doi:https://doi.org/10.1016/j.jmoneco.2013.08.005. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W....

    2013 doi

  5. [2016]

    nature 529, 484–489

    Mastering the game of go with deep neural networks and tree search. nature 529, 484–489. Singhal,K.,Tu,T.,Gottweis,J.,Sayres,R.,Wulczyn,E.,Amin,M.,Hou,L.,Clark,K.,Pfohl,S.R.,Cole-Lewis,H.,etal.,2025. Towardexpert-level medical question answering with large language models. Nat...

    2025

  6. [2021]

    URL: https://www

    Are professional forecasters overconfident? International Journal of Forecasting 37, 716–732. URL: https://www. sciencedirect.com/science/article/pii/S0169207020301394, doi:https://doi.org/10.1016/j.ijforecast.2020.09.002. Chen,S.,Green,T.C.,Gulen,H.,Zhou,D.,2024. Whatdoeschat...

    2020 arXiv

  7. [2023]

    arXiv preprint arXiv:2309.16609

    Qwen technical report. arXiv preprint arXiv:2309.16609 . Bates, J.M., and, C.W.J.G.,

    arXiv

  8. [2024]

    arXiv preprint arXiv:2404.07613

    Medical mt5: an open-source multilingual text-to-text llm for the medical domain. arXiv preprint arXiv:2404.07613 . CV Radhakrishnan et al.:Preprint submitted to Elsevier Page 19 of 21 Leveraging social media news Table 29 Impact of Expert Inattentiveness on Forecast Methods w...

    2013 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.