{"id":"f3dc699e-99c2-4d97-a21b-645b4cd45289","arxiv_id":"2411.16569","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Using Beige Book text, a fine-tuned BERT predicts stock-bond correlations better than prompted GPT-3.5, which shows look-ahead bias.","lead":"The paper tests whether ChatGPT-style language models can predict how stocks and bonds move together by reading the Federal Reserve's Beige Book, and finds that GPT models leak future information while a simpler BERT model does a better job. Generalists should care because it suggests a practical caution about using LLMs with knowledge cutoffs for financial predictions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPT vs BERT comparison is confounded by training protocol: BERT is fine-tuned on labels, GPT is zero-shot; 'traditional models still triumph' is not established.","rationale":"The reader's weakest assumption focuses on the GPT-3.5 pretraining cutoff as a clean information boundary for the look-ahead bias test. That is a legitimate concern for one pillar of the paper. However, I see a more load-bearing issue in the central comparative claim: the BERT model is supervised fine-tuned on the exact prediction task, while GPT-3.5 is used zero-shot with no task-specific training. The paper's own conclusion admits 'we are treating the BERT model as a classification model, whereas we are still treating the GPT model as a generative model,' but it does not treat this as a confound. Any out-of-sample advantage for BERT (Tables VI, VIII; Sharpe ratios in IX, X) could simply reflect the well-known benefit of supervised fine-tuning, not an architectural superiority of transformer models or a fundamental limitation of LLMs. This directly undermines the abstract's 'traditional models still triumph' conclusion. The paper does have genuine strengths: a clean new dataset (Federal Reserve Beige Book), a non-standard target variable (correlations), and a clear temporal split. The look-ahead bias evidence, though not robust across prompt types, is an interesting cautionary result. I therefore keep the reader's conditional verdict: the claims are plausible but require an additional control (e.g., fine-tuning GPT) before the headline comparison can be accepted as stated.","tokens_in":8213,"tokens_out":5659,"duration_ms":50020,"concrete_test":"Fine-tune an open-source LLM (e.g., Llama-3-8B or GPT-3 via API) on the Beige Book articles with the same rounded correlation labels and train/test split used for BERT (1980-2021 train, 2021-10 to 2024-06 test). Compare its out-of-sample RMSE and portfolio Sharpe ratios against the zero-shot GPT-3.5 and the fine-tuned BERT. If the fine-tuned LLM meets or exceeds BERT, the 'traditional models still triumph' conclusion fails; if BERT still wins, the training-protocol confound is not the explanation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central comparison in Sections II and V pits a fine-tuned BERT model (Section II.C) against a zero-shot prompted GPT-3.5 (Section II.B). BERT is trained on Beige Book articles with labels derived from actual stock-bond correlations for 1980-2021, while GPT-3.5 is not fine-tuned on this task at all. The out-of-sample tables (VI, VIII) and Sharpe ratios (IX, X) showing BERT superiority therefore conflate model architecture with training protocol: a task-tuned model should generally beat a general-purpose model on a supervised prediction task. The abstract's claim that 'traditional models still triumph' is thus not a fair test of GPT's predictive power. For this claim to hold, GPT should be fine-tuned on the same labels and evaluated under the same protocol, or the conclusion should be limited to zero-shot GPT prompting. The look-ahead bias finding is also entangled with pretraining memorization, but the BERT comparison is the load-bearing pillar for the headline 'triumph' claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript asks whether GPT-3.5 Turbo can predict monthly stock-bond correlations from Federal Reserve Beige Book text, compares it with a fine-tuned BERT model and with a rolling-covariance baseline, and evaluates the resulting minimum-variance portfolios via Sharpe ratios. The authors report evidence of 'look-ahead bias' in GPT (lower in-sample RMSE than out-of-sample, especially in the binned version), no benefit from feeding historical correlations into the prompt, and superior out-of-sample RMSE and portfolio Sharpe ratios for BERT over GPT. They conclude that the Beige Book contains correlation-relevant information but that GPT has too much look-ahead bias and that 'traditional models still triumph.'","tokens_in":8456,"tokens_out":4551,"duration_ms":41633,"significance":"If established, the result would be a useful cautionary data point for using pretrained LLMs in financial forecasting, and the Beige Book correlation-prediction task is a reasonable, relatively clean test bed. The authors deserve credit for attempting an explicit hypothesis-testing framework, for using out-of-sample evaluation for the BERT model, and for recognizing the look-ahead-bias issue. However, the central comparisons are confounded by the training-protocol difference between GPT (zero-shot) and BERT (fine-tuned on labels), the look-ahead-bias test conflates memorization with time-period effects, and the Sharpe-ratio comparisons lack any inferential support. The paper is therefore a suggestive preliminary study rather than a definitive test of the claims in the abstract.","major_comments":[{"comment":"The comparison between GPT and BERT is confounded by training protocol. GPT-3.5 is used zero-shot with a prompt, while BERT is fine-tuned on 1980-2021 labels derived from the actual stock-bond correlations. The out-of-sample tables showing lower RMSE for BERT (Tables VII-VIII and the simulation results in Section V) therefore do not identify whether the BERT architecture or the supervised fine-tuning drives the improvement. The abstract's claim that 'traditional models still triumph' over GPT is not supported unless GPT is also fine-tuned on the same labels or the claim is explicitly restricted to zero-shot GPT prompting.","section":"Sections II.B, II.C, and Tables V-VIII"},{"comment":"The look-ahead-bias test compares RMSE on 1980-01 to 2021-09 ('training set') with RMSE on 2021-10 to 2024-06 ('testing set'). Because GPT-3.5's pretraining data include financial and web text up to September 2021, the historical period is in-distribution for the model, and its lower in-sample RMSE is better characterized as memorization of past outcomes than as look-ahead bias in the forecasting sense. The test also cannot separate a cutoff effect from a market-regime change between the two periods, and the t-test on monthly RMSE values ignores autocorrelation in correlation forecast errors; for n=12 the test set yields only two non-overlapping blocks, so the test is severely underpowered. The conclusion of 'unavoidable look-ahead bias' is therefore not established by the reported evidence.","section":"Section IV.A (Look-Ahead Bias)"},{"comment":"The Sharpe ratios are point estimates with no standard errors, confidence intervals, or hypothesis tests. The differences reported (e.g., 2.605 vs 2.768 vs 2.334 in Table IX; 0.423 vs 1.9208 vs 0.445 in Table X) could easily be within sampling variation given the small post-Covid sample and the absence of any significance testing. The statements that 'BERT tends to perform the best' and that GPT 'does even worse after Covid' are not statistically substantiated. The authors should report bootstrap/t-test results or at least the number of rebalancing periods used.","section":"Section V, Tables IX-X"},{"comment":"The simulation section selects 'BERT: Original Correlations V3' and 'GPT: Original Correlations V3' because 'V3 seems to produce a more accurate correlation than both V1 and V2.' If this selection was made using the same test period used in the simulations, the portfolio results inherit a selection bias. The paper must state on which sample (training set only) V3 was chosen, or present results for all three versions to demonstrate robustness.","section":"Section V.A (Choice of V3)"}],"minor_comments":[{"comment":"Table VI is captioned 'Training with Bins' but the text states that out-of-sample results are shown in Tables VI and VIII; the entries in Table VI (all 1.000) appear to refer to the test set, not the training set. This labeling error makes the results difficult to interpret.","section":"Tables V-VIII"},{"comment":"The prompt includes both 'You don't know anything that happens during this year or afterwards' and the year of the article. Since the Beige Book article is about conditions during that month, it would be clearer to state explicitly whether the requested correlation is for the same month (a nowcast) or for a subsequent month, because the look-ahead-bias interpretation depends on this timing.","section":"Section II.B"},{"comment":"The RMSE formula is missing parentheses and the index range; the equation should read RMSE = sqrt((1/n) * sum_{i=1}^n (y_i - yhat_i)^2).","section":"Section IV.A (Equation for RMSE)"},{"comment":"The algorithm refers to 'w_I' (presumably a risk-free or third asset weight) without defining how it is chosen or whether it is part of the two-asset minimization; the notation should be clarified.","section":"Section III.B (Simulation, Algorithm 2)"},{"comment":"The paper does not provide code, data availability, or the exact GPT/BERT model identifiers (e.g., gpt-3.5-turbo version date and BERT checkpoint), which limits reproducibility; at least the exact model versions and prompt templates should be included in an appendix.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This appears to be an early-stage research report. The core idea is interesting and the topic is timely, but the current analysis conflates architecture with training protocol, lacks inferential support for the portfolio claims, and relies on a post-hoc model-version choice. I would encourage the authors to either (a) fine-tune GPT on the same labels, or (b) reframe the conclusions as being specifically about zero-shot GPT prompting. The look-ahead-bias analysis also needs a more careful design, e.g., holding out a period that is truly post-cutoff for GPT and controlling for market regime. Given the current standard, I do not think this is acceptable without substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper's core comparison is confounded, and that matters because the abstract's punchline—'traditional models still triumph'—rests on it. BERT is fine-tuned on historical correlation labels; GPT-3.5 is prompted zero-shot. Out-of-sample superiority of BERT could just reflect supervised training, not a difference between models. The authors acknowledge this distinction in the conclusion, but the abstract and the simulation framing still overclaim.\n\nWhat's genuinely useful is the setup. Predicting stock-bond correlations from Beige Book text is a smart low-noise target compared to return forecasting, and the 1980–2024 span is solid. The look-ahead bias test is thought-provoking: the binned prompts show significantly larger out-of-sample errors, consistent with GPT-3.5 remembering pre-cutoff outcomes. That's a concrete, falsifiable observation.\n\nThe soft spots beyond the confound: the simulation uses version 3, chosen post hoc; the Sharpe ratios are point estimates without significance tests; and the look-ahead result weakens with the original three-point prompt. These are fixable, but they should be addressed before anyone takes the 'triumph' claim literally.\n\nIf you work on LLM evaluation or financial text, this is worth a skim, and I'd bring it to a reading group to discuss what separates a fair comparison from a convenient one. But I'd be cautious about citing it as evidence that traditional models beat GPT for this task.\n\nMy recommendation: send it to peer review, but with major revision—either fine-tune GPT on the same labels or narrow every conclusion to zero-shot prompting, pre-register the model selection, and add uncertainty measures for the Sharpe ratios. The empirical material is worth engaging; the current claims outrun the design.","headline":"Fine-tuned vs zero-shot confound undercuts the headline 'traditional models triumph' claim, but the Beige Book correlation task and look-ahead bias test are worth engaging.","tokens_in":8937,"tokens_out":3389,"would_cite":false,"duration_ms":31605,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompted GPT-3.5 can extract stock-bond correlation signals from Federal Reserve Beige Book text, but its apparent edge is look-ahead bias and a fine-tuned BERT model performs better out of sample.","keywords":["Large Language Models","GPT-3.5","BERT","Beige Book","stock-bond correlation","look-ahead bias","portfolio optimization","Sharpe ratio"],"falsifier":"Run the same prompts on a GPT model with a pretraining cutoff no later than 2019 and compare RMSE on months after that cutoff; if the test-period error no longer exceeds the training-period error, the look-ahead-bias story would be weakened. Alternatively, rewrite the Beige Book articles with dates shifted by one or two years and ask GPT-3.5 for correlations; if the outputs barely change, the model is responding to textual content rather than to memorized calendar outcomes.","tokens_in":7999,"feed_emoji":"📊","tokens_out":6303,"duration_ms":58013,"temperature":0.7,"pith_summary":"This paper asks whether a large language model can turn Federal Reserve Beige Book text into useful predictions of how stock and bond returns will co-move. It compares prompted GPT-3.5 with a BERT model fine-tuned to classify the same articles into correlation categories, then feeds the predicted correlations into minimum-variance portfolios. The paper concludes that the Beige Book does contain information about asset correlations, but GPT-3.5 carries too much look-ahead bias for its predictions to be trusted out of sample, and the traditional BERT model performs better. The practical upshot for a reader is that generative prompting alone is not a reliable way to harvest economic text for portfolio decisions, and a simpler supervised classifier can beat it.","feed_headline":"GPT-3.5's market edge is data leakage; BERT wins","feed_subtitle":"Fed Beige Book text predicts stock-bond correlations, but fine-tuned BERT beats prompted GPT-3.5 with less look-ahead bias.","key_machinery":"The load-bearing objects are two prediction pipelines over the same Federal Reserve Beige Book articles: a prompted GPT-3.5 with temperature zero, asked to return a coarse correlation category, and a BERT model fine-tuned as a classifier on the actual historical correlation labels. The argument uses RMSE computed over non-overlapping n-month blocks, with the pretraining cutoff defining the train/test boundary, and then feeds the predicted correlations into minimum-variance portfolio optimization, using a closed-form two-asset weight and a Lagrangian multi-asset version. The train/test split is what isolates look-ahead bias, and the portfolio simulation is what turns correlation forecasts into an economic comparison.","core_discovery":"The paper's central claim is that GPT-3.5's apparent skill at predicting stock-bond correlations from Beige Book text is substantially an artifact of look-ahead bias, not genuine forecasting ability. In the paper's design, the model is prompted as a financial agent with no knowledge after the article date, but because GPT-3.5 was pretrained through September 2021, the entire 1980-2021 training period is visible to it. When the same prompt is evaluated on 2021-10 through 2024-06, errors rise significantly, especially in the binned-response version, while a BERT model trained as a classifier on actual correlation labels holds up better out of sample. In portfolio simulations, BERT-based weights yield higher Sharpe ratios than GPT-based weights in both two-asset and multi-asset settings. The authors summarize it as: the Beige Book contains correlation-relevant information, yet the GPT model has too much look-ahead bias and traditional models still triumph.","pith_inferences":["A natural extension is to retest with a model whose pretraining cutoff lies well before the evaluation window; if the out-of-sample gap vanishes, the paper's main evidence for look-ahead bias would be sharpened.","The comparison is tilted by design: BERT is fine-tuned on labels from the same period, while GPT-3.5 is used zero-shot, so the result says more about prompting versus supervised fine-tuning than about the architectures themselves.","The same template could be applied to other scheduled economic releases, such as FOMC statements or jobs reports, where text timing is clean, to see whether classification beats generative prompting for other macro-financial quantities.","The paper implicitly shows that the sign and coarse bin of the correlation matter more than exact magnitude for minimum-variance allocation; a test using continuous correlation targets could reveal whether GPT's extra granularity is wasted."],"forward_implications":["If the paper is right, a GPT-style model that has seen the test period during pretraining cannot serve as a clean out-of-sample forecaster for financial text, even when the prompt instructs it to ignore future knowledge.","Prompting GPT with recent historical correlations does not help: in the original three-choice version the errors got worse, which suggests noisy numerical context can degrade the text signal.","A supervised BERT classifier trained on actual outcomes is the better way to extract correlation information from the Beige Book, and the resulting portfolios have higher Sharpe ratios than GPT-driven or rolling-average portfolios.","The Beige Book text itself carries information about stock-bond comovement; the failure is in the extraction method, not the data source."],"supporting_citations":[{"why":"Bybee (2023) provides the baseline approach of applying LLMs to economic text to measure beliefs and sentiment, which this paper extends by predicting correlations rather than returns.","marker":"[3]"},{"why":"The Federal Reserve Beige Book is the dataset whose articles are fed to both GPT and BERT.","marker":"[4]"},{"why":"Kalyani et al. introduces news-sentiment stock trend prediction, the research line the paper positions itself against.","marker":"[1]"},{"why":"Ren et al. uses a deep bidirectional LSTM on news to predict stock price changes, motivating the transformer-based comparison.","marker":"[2]"}],"fun_headline_variants":["Look-ahead bias explains GPT-3.5's market edge, not skill","BERT beats GPT-3.5 in clean Fed Beige Book test","GPT-3.5's stock reads are data leakage, BERT wins","Honest test: BERT beats GPT-3.5 on market correlations","Traditional BERT trumps LLM GPT-3.5 without look-ahead bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on GPT-3.5's September 2021 pretraining cutoff being a clean information boundary, so that a lower in-sample error is evidence of look-ahead bias rather than of a genuinely better model.","fun_headline_variants_meta":{"raw":{"variants":["Look-ahead bias explains GPT-3.5's market edge, not skill","BERT beats GPT-3.5 in clean Fed Beige Book test","GPT-3.5's stock reads are data leakage, BERT wins","Honest test: BERT beats GPT-3.5 on market correlations","Traditional BERT trumps LLM GPT-3.5 without look-ahead bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1448,"prompt_tokens":953,"completion_tokens":495,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":392}},"tokens_in":569,"tokens_out":495,"duration_ms":5899,"temperature":1.0,"reasoning_tokens":392,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:58:24.159720+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same prompts on a GPT model with a pretraining cutoff no later than 2019 and compare RMSE on months after that cutoff; if the test-period error no longer exceeds the training-period error, the look-ahead-bias story would be weakened. Alternatively, rewrite the Beige Book articles with dates shifted by one or two years and ask GPT-3.5 for correlations; if the outputs barely change, the model is responding to textual content rather than to memorized calendar outcomes.","supporting_citations":[{"cited_title":"(2023, November)","cited_arxiv_id":null,"evidence_quote":"Bybee (2023) provides the baseline approach of applying LLMs to economic text to measure beliefs and sentiment, which this paper extends by predicting correlations rather than returns."},{"cited_title":"Federal Reserve Board - Beige Book","cited_arxiv_id":null,"evidence_quote":"The Federal Reserve Beige Book is the dataset whose articles are fed to both GPT and BERT."},{"cited_title":"(2020, July 27)","cited_arxiv_id":null,"evidence_quote":"Ren et al. uses a deep bidirectional LSTM on news to predict stock price changes, motivating the transformer-based comparison."}],"review_version":1}