Forecasting in the Fog: Real-Time versus Revised-Data Evidence on Machine Learning's Edge over the Phillips Curve

Louis Agyekum; Obed Obese

REVIEW 3 major objections 6 minor 2 references

Forecasting in the Fog: Real-Time versus Revised-Data Evidence on Machine Learning's Edge over the Phillips Curve

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Random Forest trained on revised U.S. inflation data leans on PCE inflation, while the same model trained on real-time data barely uses it.

desk verdict Genuinely new out-of-sample SHAP across ALFRED vintages, but the headline PCE swing may rest on an unstated reference-month mismatch that the authors must pin down. read the letter →

arxiv 2608.09033 v1 pith:OOXOZQHU submitted 2026-08-10 stat.AP

Louis Agyekum , Obed Obese This is my paper · ORCID

classification stat.AP MSC 62P2062M20

keywords real-timedatarevisionsmachinelearningPhillipscurveinflationforecastingfeatureattributionrecursiveevaluationhindsightbias

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Machine-learning inflation forecasts are usually trained and tested on fully revised data, though a real-time forecaster never has that data. This paper rebuilds both information sets from archived data vintages and re-estimates eight models recursively, and it reports that the real-time/revised gap in point accuracy is small: across thirty-six model-horizon comparisons, only one ML-versus-Phillips-curve comparison reaches even 10 percent significance (Gradient Boosting at six months), and the random walk stays hard to beat at short horizons. The accuracy picture, however, hides a large behavioral divergence: a Random Forest trained on revised data makes PCE inflation (a broad price index) its top predictor (mean absolute attribution 0.778), while the same model trained on real-time data gives PCE negligible importance (0.039) and leans on current CPI inflation (0.834). That roughly twenty-fold swing, larger under out-of-sample attribution than under in-sample attribution, is invisible to root-mean-squared forecast error and shows the model's PCE reliance is substantially a hindsight artifact.

What carries the argument

The load-bearing device is a walk-forward, out-of-sample feature-attribution procedure paired with the same recursive expanding-window evaluation used for accuracy. At each monthly forecast origin, the model is re-estimated only on data available up to that date, and per-prediction attribution values are computed for that month's out-of-sample point and then averaged across the roughly 150 recursive steps, so reported importances reflect the model that genuinely existed at each date rather than a single in-sample fit. The companion identity is the Revision Sensitivity Index, $RSI = RMSFE_{RT} - RMSFE_{REV}$: positive values mean apparent accuracy is partly hindsight. Both panels are built with an as-of operator that recovers, for any historical date, the value of each series as published by that date, producing vintage-consistent real-time and revised feature vectors that share an identical final target.

What would settle it

Rebuild the two Random Forest panels using an explicitly stated quarterly-GDP alignment (for example, carrying the latest quarter forward versus interpolating it across months) and recompute the out-of-sample PCE attribution; if the roughly twenty-fold gap (0.039 versus 0.778) narrows to near zero under any plausible alignment, the hindsight-artifact conclusion would fail, and if it persists across alignments, the claim is supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that machine learning's forecasting edge over the Phillips curve survives the transition to genuine real-time data at the level of point accuracy, but the story the models tell about what drives inflation does not. Using real-time and revised panels that share the same final target, the authors find that accuracy gaps are generally small and statistically indistinguishable from zero; Gradient Boosting is the only model with consistent positive out-of-sample skill at the 6- and 12-month horizons (out-of-sample $R^2$ of 0.51 and 0.63 in real time), while every model fails to beat a random walk at three months. The consequential finding is interpretational: a walk-forward, out-of-sample feature-attribution analysis of a Random Forest shows PCE inflation's importance rising from 0.039 (rank 6) in real time to 0.778 (rank 1) in revised data, while current CPI inflation falls from 0.834 (rank 1) to 0.223 (rank 2), an approximately twenty-fold swing that RMSFE-based comparisons cannot detect. The authors conclude that the model's PCE reliance is substantially a hindsight artifact.

Load-bearing premise

The central comparison assumes the real-time vintage panels accurately reconstruct what a forecaster knew each month; in particular, the paper never states how quarterly real GDP is aligned to monthly reference dates, and a different plausible alignment could change the accuracy gaps and the feature ranking.

Editorial extensions

If this is right

Point-accuracy audits of ML inflation forecasts can largely keep the revised-data convention: 35 of 36 tested model-horizon comparisons show no statistically reliable real-time/revised gap once a flagged support-vector-regression anomaly is set aside.
The unregularized autoregressive distributed-lag model (ADL-OLS) pays the largest real-time penalty, with its Revision Sensitivity Index rising from +0.071 at three months to +0.395 at twelve months; Elastic Net shows a parallel but smaller pattern.
Any claimed ML skill is horizon-dependent: at three months no model beats the random walk out of sample, while Gradient Boosting is the only model with large positive out-of-sample R-squared at six and twelve months.
Interpreting a revised-data model's feature importances as a description of real-time behavior is unsafe: the Random Forest's top revised-data predictor is PCE inflation, which a real-time version of the same model barely uses.
Out-of-sample attribution should be part of the audit toolkit; the conventional in-sample attribution understates the PCE swing by about a third (roughly fourteen-fold versus twenty-fold).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: the same real-time versus revised attribution test should be run on Gradient Boosting at the 6- and 12-month horizons; its real-time RSI is negative, so its revised-data feature ranking may differ from its real-time ranking even more than the Random Forest's.
Inference: because CPI and PCE inflation are highly correlated at low frequencies, the model can swap one for the other with little cost in squared error; this suggests feature-importance audits can detect hindsight effects that accuracy metrics are structurally blind to in any setting where predictors are collinear.
Inference: the paper never states how quarterly real GDP is aligned to monthly reference dates; a robustness exercise that varies carry-forward versus interpolation could shift GDP's attributed importance and the RSI values, though the PCE swing would need to persist to keep the central claim.
Inference: the Revision Sensitivity Index framework extends naturally to probabilistic forecasts: predictive distributions trained on real-time versus revised data could diverge in width or calibration even when point errors do not.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper does something real: it builds a fully recursive, vintage-consistent evaluation of eight inflation models and pairs it with walk-forward out-of-sample SHAP, which is a clear step beyond the usual in-sample interpretability exercises. The core finding—PCE inflation's importance jumping from rank 6 (0.039) to rank 1 (0.778) between real-time and revised Random Forests—is striking and, as far as I know, new. The point-forecast results are less novel but sensible: random walk stays hard to beat, Gradient Boosting has genuine long-horizon skill, and the real-time/revised accuracy gap is mostly small and insignificant. The RSI diagnostic is simple and could be useful. The DM/HLN testing is appropriate, and the authors honestly flag the SVR variance-degeneracy anomaly. Good empirical habits overall.

Now the soft spots, in proportion.

The serious one: Section 4.1 defines the real-time feature vector by evaluating as-of at the date CPI inflation for month r was first published. But PCEPI for month r is not released on that date under the BEA calendar—it comes out later. The paper never states how p_t is constructed in the real-time panel. If it is carried forward from an earlier month or taken from the first available PCE vintage, then the real-time model's PCE feature is not PCE_r but PCE_{r-k}. In that case the 0.039-vs-0.778 swing conflates a reference-month mismatch with a genuine revision effect. This is load-bearing because PCE is the analyzed variable. The fix is straightforward: state explicitly, for each predictor, which vintage and which reference month enters the model at each forecast origin, and ideally show that PCEPI(r) was actually available at τ. If that cannot be shown, the headline claim needs substantial caveating. This is not a kill-shot; it is a demand for a precise definition.

Smaller issues: the GDPC1 quarterly-to-monthly mapping is unstated, which affects the ADL-OLS and ML models, though GDP is a minor SHAP feature. The headline SHAP result is one model, one horizon, no uncertainty intervals—bootstrap or repeated-split intervals would help. The code is promised as supplementary but no commit hash or archive link appears, so reproducibility is asserted rather than checkable.

Who gets value: anyone doing real-time macro forecasting evaluation, and anyone using SHAP for model audits. The paper is worth a serious referee, but it needs revision before the headline result can be treated as established. I'd send it out, and in the letter require the authors to (1) define the real-time construction of every predictor explicitly, especially PCE, (2) re-run the SHAP with a correctly dated PCE feature, and (3) add uncertainty characterization. If the PCE alignment checks out, this is a solid contribution to an under-studied gap.

Referee Report

3 major / 6 minor

Summary. This paper compares eight U.S. inflation-forecasting models—random walk, AR(1), Phillips curve, ADL-OLS, Random Forest, Gradient Boosting, Elastic Net, and SVR—under real-time (ALFRED vintage) and revised data, using a fully recursive expanding-window design at 3-, 6-, and 12-month horizons over 2000–2026. It introduces the Revision Sensitivity Index (RSI), defined as the difference between real-time and revised RMSFE, and applies a walk-forward, out-of-sample SHAP procedure to the Random Forest at the 3-month horizon to compare feature importances across data vintages. The paper reports that real-time/revised accuracy gaps are generally small and statistically insignificant, that Gradient Boosting shows genuine skill at longer horizons while the random walk remains a strong short-horizon benchmark, and that PCE inflation has negligible SHAP importance in real time (mean |SHAP| = 0.039, rank 6) but dominant importance in the revised-data model (mean |SHAP| = 0.778, rank 1), which it interprets as a hindsight artifact.

Significance. The paper is a useful contribution to the real-time evaluation literature. Its strengths are the recursive re-estimation design, the use of the Harvey–Leybourne–Newbold correction in Diebold–Mariano tests, and the walk-forward SHAP procedure, which is a clear improvement over in-sample interpretability exercises. The RSI is a simple and reusable diagnostic for quantifying vintage sensitivity. If the data-construction issues identified below are resolved, the paper would provide credible evidence that point-forecast accuracy comparisons are largely robust to data vintage in this application, while also offering a cautionary example that SHAP-based narratives can be vintage-dependent. The promised reproducible ALFRED pipeline would be valuable to the community.

major comments (3)

[§4.1, Eq. (3)] The real-time feature vector is evaluated at the date CPI inflation for month r was first published, but PCEPI for month r is not published on that date under the standard BEA release calendar. The manuscript does not state how the PCE predictor p_t is obtained in the real-time panel—whether from the first available PCE vintage (which would be for an earlier month), by carry-forward from month r−1, or by evaluating the as-of operator at a later date. If p_t is not the first-release value of PCE_r, the headline SHAP comparison in Table 8 (0.039 versus 0.778) conflates a reference-month mismatch with a genuine revision effect. Please define p_t explicitly and, ideally, re-run the revised-data model with the same availability-driven lag structure (e.g., final PCE_{r−1}) to isolate the revision effect.
[§4.1] Real GDP (GDPC1) is quarterly, but all predictors are indexed by monthly reference month t. The manuscript never states how quarterly observations are mapped to monthly values (interpolation, carry-forward, or assignment to a specific month). This mapping affects the ADL-OLS and all four ML models, the RSI values, and the Diebold–Mariano tests in Tables 3–7, and it could also shift the SHAP ranking of GDP in Table 8. Please state the alignment rule and test sensitivity to a reasonable alternative, such as carrying forward the most recent quarter's growth rate.
[§5.5, Table 8] The headline SHAP result is reported as a single scalar per feature—the mean absolute SHAP value over N=208 recursive forecasts—with no uncertainty quantification. The paper should report the dispersion of per-period SHAP values (e.g., standard errors or bootstrap confidence intervals) and, most importantly, a formal test or permutation-based comparison of the real-time versus revised PCE importance. Without this, the twenty-fold swing could be within sampling noise, and the conclusion that 'the model's PCE reliance is substantially a hindsight artifact' is not statistically supported.

minor comments (6)

[§4.7] The paragraph after Eq. (19) contains a garbled, duplicated sentence ending '...forecasts:ng data unavailable at start equation t.' Please rewrite it.
[§5.5.1] The opening sentence is ungrammatical and the reference to 'Table 9' for out-of-sample values is incorrect; the out-of-sample values are in Table 8.
[§3.2, Table 2] The maximum absolute revision for GDPC1 year-over-year growth is reported as 15.130 percentage points (and '15.1 percentage points' in the text), which is implausibly large for annual growth rates; please verify the units and correct if this is a decimal or data-processing error.
[§5.4] Please state the significance threshold explicitly and note that the one 10%-significant result does not survive a 5% threshold; moreover, with 12 comparisons at the 10% level, one significant result is expected by chance, so the interpretation in Section 6 should be tempered.
[§4.7] Please specify the SHAP implementation (e.g., TreeSHAP, background dataset, feature perturbation method), since absolute SHAP values depend on these choices and the comparison across data conditions could be sensitive to them.
[§4.1, Eq. (3)] The notation τ_t^first is used before being defined; please define it as the first CPI release date for reference month t in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline quantities are empirical outputs of the recursive evaluation, not fitted inputs renamed as predictions.

full rationale

The paper's central quantities are all measured outputs of a stated evaluation design. The Revision Sensitivity Index (RSI, Eq. 13) is defined as the difference between real-time and revised RMSFEs computed from the recursive forecast exercise; nothing is fitted to the RSI itself, and no parameter is tuned to reproduce the gap. The SHAP feature importances (Eq. 19) are computed by re-fitting the Random Forest at each recursive origin and averaging the Shapley contributions on the single out-of-sample test point; the reported PCE inflation swing (0.039 vs. 0.778) is an empirical outcome of that walk-forward procedure, not an input imposed by construction. The Diebold-Mariano tests compare forecast-error loss differentials using standard formulas and a stated small-sample correction; there is no step in which a prediction is defined in terms of the target it claims to predict. Self-citations to Agyekum (2026) and Agyekum et al. (2026) are used to motivate the research question and to note consistency of the random-walk finding with an exchange-rate study; they do not supply any parameter, theorem, or estimate used in the estimation here, so they are not load-bearing in the derivation. The possible release-date mismatch for PCEPI in the real-time panel is a data-construction/validity concern, not a circularity concern, because no equation makes the SHAP ranking true by definition; it is an empirical claim that could fail under a different alignment rule. Overall, the derivation chain is self-contained: models, forecasts, accuracy metrics, significance tests, and SHAP values are all generated from the data and algorithm definitions rather than from the conclusions they are used to support.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The estimation relies on hand-chosen ML hyperparameters rather than tuned values. The core assumptions are about the fidelity of ALFRED vintages, the revised-target convention, the validity of DM/HLN and TreeSHAP, and the unstated GDP-to-monthly mapping. The only new construct is the descriptive RSI, which carries no independent evidential weight.

free parameters (4)

Random Forest hyperparameters = B=200; max_depth=4; min_samples_leaf=5
Hand-chosen in Section 4.3.1 to limit overfitting; not tuned. The headline SHAP result is specific to this configuration.
Gradient Boosting hyperparameters = 150 stages; max_depth=2; learning_rate=0.05
Hand-chosen in Section 4.3.2; affects the claim that GB is the only model with consistent skill at longer horizons.
Elastic Net hyperparameters = alpha=0.5; lambda=0.1
Hand-chosen in Section 4.3.3, no cross-validation; affects RSI and DM comparisons.
SVR hyperparameters = C=1.0; epsilon=0.1; RBF default bandwidth
Hand-chosen in Section 4.3.4; SVR is the weakest model and the authors flag hyperparameter sensitivity in Section 7.

assumptions (5)

domain assumption ALFRED vintages and first-release dates accurately reconstruct the real-time information set.
Section 4.1 applies the as-of operator to ALFRED; if vintage records are incomplete, the real-time panels are wrong.
domain assumption Using final revised inflation as the target in both real-time and revised experiments is the appropriate convention.
Equation (3) sets y_{t+h} to final revised inflation in both arms; Section 7 acknowledges an alternative first-release target could serve as a robustness check.
standard math The Diebold-Mariano test with Harvey-Leybourne-Newbold correction is valid for overlapping h-step-ahead errors.
Section 4.6, equations (15)-(17).
standard math Mean absolute SHAP values across recursive test points estimate predictive relevance.
Section 4.7, equation (19); assumes TreeSHAP consistency and a representative test set.
domain assumption Quarterly GDPC1 can be mapped to monthly reference months; the mapping rule is unstated.
Section 4.1 uses y_t as a monthly predictor; no interpolation or carry-forward rule is described.

invented entities (1)

Revision Sensitivity Index (RSI)
purpose: Summarize the gap between a model's real-time and revised-data forecast accuracy (Eq. 13).
A deterministic difference of two RMSFEs; it has no falsifiable handle outside the paper and is not validated against an external benchmark.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Forecasting in the Fog: Real-Time versus Revised-Data Evidence on Machine Learning's Edge over the Phillips Curve." pith.science (2026). https://pith.science/paper/OOXOZQHU

@misc{pith2026260809033,
  author       = {Pith},
  title        = {Pith review of: Forecasting in the Fog: Real-Time versus Revised-Data Evidence on Machine Learning's Edge over the Phillips Curve},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OOXOZQHU}},
  note         = {Machine review of arXiv:2608.09033}
}

read the original abstract

ML inflation forecasts are almost universally trained on fully revised data, even though real-time forecasters never have such data, and reported feature importances are typically computed in-sample, conflating predictive relevance with retrospective fit. This paper asks whether the ML advantage over the Phillips curve documented in Agyekum (2026) survives when models are trained and evaluated on real-time (ALFRED) vintages rather than revised series, and whether SHAP feature-importance rankings are an artifact of in-sample estimation. Using 2000-2026 U.S. data on unemployment, CPI and PCE inflation, payrolls, real GDP, and the 10-year-2-year Treasury spread, vintage-consistent panels are built for four traditional models (random walk, AR(1), Phillips curve, ADL-OLS) and four ML models (Random Forest, Gradient Boosting, Elastic Net, SVR), re-estimated recursively at 3-, 6-, and 12-month horizons (208, 206, 204 forecasts). Real-time/revised accuracy differences are small and, apart from one exception at 6 months (Gradient Boosting vs. Phillips curve, DM = -1.671, p = 0.097), indistinguishable under Diebold-Mariano tests; Gradient Boosting alone shows consistent positive skill at longer horizons. The random walk remains a strong short-horizon benchmark, consistent with the puzzle in Agyekum et al. (2026) for exchange rates. Using walk-forward, out-of-sample SHAP, a Random Forest on revised data assigns dominant importance to PCE inflation (mean |SHAP| = 0.778, rank 1 of 9), while on real-time data it assigns PCE negligible importance (0.039, rank 6), relying instead on current CPI (0.834 vs. 0.223). This twenty-fold swing, larger than the in-sample estimate, is invisible to point-forecast metrics and shows the model's PCE reliance is substantially a hindsight artifact. An RSI summarizes the accuracy gap by model and horizon, with implications for auditing ML inflation forecasts.

Figures

Figures reproduced from arXiv: 2608.09033 by the authors.

**Figure 2.** Distribution of revision magnitudes for CPI, payrolls, and GDP (year-over-year growth terms). 4. Methodology This section formalizes the construction of the real-time and revised information sets, states each of the eight forecasting models in explicit mathematical notation, and details the recursive evaluation, significance testing, and interpretability procedures [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** The eight-stage analytical pipeline: from ALFRED vintage acquisition through recursive backtesting to the interpretability case study [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Rolling 24-month out-of-sample RMSE by model, h = 3, real-time condition [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

**Figure 5.** Figure 5: Three-month-ahead forecasts against realized inflation, real-time vs. revised data conditions. 6. Discussion Three findings merit emphasis. First, at the point-forecast level, the hypothesis that ML’s advantage over the Phillips curve is a hindsight artifact is not str…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages

[1]

Agyekum, L. (2026). Forecasting and explaining the Phillips curve: A SHAP-based comparison of machine learning and traditional time -series models for Canadian unemployment and inflation. arXiv. https://arxiv.org/abs/2607.22453 Agyekum, L., Agyemang, E. F., Ampomah, O. -A., Acheampong, K., Boadi, E., Amakye, P. Y., Tchorly, F. S., Bonsu, E. A., & Nyarko, ...

work page Pith review arXiv 2026
[2022]

Y., & Thompson, S

Campbell, J. Y., & Thompson, S. B. (2008). Predicting excess stock returns out of sample: Can anything beat the historical average? Review of Financial Studies, 21(4), 1509–1531. Clark, T. E., & West, K. D. (2007). Approximately normal tests for equal predictive accuracy in nested models. Journal of Econometrics, 138(1), 291–311. Clements, M. P., & Hendry...

2008

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Agyekum, L. (2026). Forecasting and explaining the Phillips curve: A SHAP-based comparison of machine learning and traditional time -series models for Canadian unemployment and inflation. arXiv. https://arxiv.org/abs/2607.22453 Agyekum, L., Agyemang, E. F., Ampomah, O. -A., Acheampong, K., Boadi, E., Amakye, P. Y., Tchorly, F. S., Bonsu, E. A., & Nyarko, ...

work page Pith review arXiv 2026

[2] [2022]

Y., & Thompson, S

Campbell, J. Y., & Thompson, S. B. (2008). Predicting excess stock returns out of sample: Can anything beat the historical average? Review of Financial Studies, 21(4), 1509–1531. Clark, T. E., & West, K. D. (2007). Approximately normal tests for equal predictive accuracy in nested models. Journal of Econometrics, 138(1), 291–311. Clements, M. P., & Hendry...

2008