REVIEW 3 major objections 4 minor 42 references
Forecasting observables with particle filters: Any filter will do!
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper claims that, in Bayesian state-space forecasting, the choice of particle filter used inside a particle marginal Metropolis-Hastings sampler has virtually no effect on one-step-ahead forecast accuracy, despite large differences…
desk verdict A useful methods paper with two new unbiased filters and a plausible but under-evidenced invariance claim; worth peer review with major revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is PMMH (particle marginal Metropolis-Hastings), a scheme that embeds an unbiased particle-filter estimate of the likelihood inside a Metropolis-Hastings sampler so that the chain's stationary distribution is the posterior. Forecasting then adds one more filter pass to produce the one-step-ahead predictive density for each posterior parameter draw, and averages those densities pointwise into the forecast distribution. The two new filters extend the class of valid building blocks: the data-driven particle filter (DPF) proposes particles by inverting the measurement equation from draws of the measurement error, and the unscented data-driven particle filter (UDPF) blends that idea with a Gaussian proposal built from unscented transformations of the state transition.
What would settle it
Re-run the paper's SV simulation with N=300 particles but only about 500 post-burn-in MCMC draws per posterior refresh, and score the one-step-ahead predictive densities with both the log score and a 5 percent tail quantile score. If the invariance is only an artifact of heavy averaging, the DPF, with its inefficiency factor near 370, should fall well behind the BPF and UDPF; if the paper's claim is general, the scores should stay close.
Extended reading notes
Core claim
The central discovery is an empirical invariance: for the one-step-ahead predictive distribution of the observed variable, the estimating filter does not matter. With 300 particles and 5,000 MCMC draws held fixed for every method, the bootstrap particle filter, the unscented particle filter, and the two new data-driven filters produced average log scores that differ by less than about 0.035 across the board, even though the filters' inefficiency factors range from roughly 20 to nearly 390. The paper explains the invariance through the forecast construction itself: the predictive density is formed by averaging one-step-ahead conditional densities over the posterior draws of the parameters, and that averaging smooths away filter-specific variation in the conditional forecasts. The paper also proves that the likelihood estimators produced by both new filters are unbiased, so the PMMH sampler preserves its exact posterior target when they are used.
Load-bearing premise
The near-equality of forecast scores is shown for smooth, averaged criteria with fixed particle and MCMC budgets; if the forecast target were a rare tail event or the MCMC chain were shortened so an inefficient filter's effective sample size collapsed, the invariance could disappear.
Editorial extensions
If this is right
- A forecaster whose objective is prediction can choose the cheapest or simplest particle filter for the model at hand; in the settings studied, the average log score moved by less than about 0.035.
- The invariance survives model misspecification in the tested designs, so filter choice need not be driven by fear that the state-space model is wrong.
- For parameter inference, the filter choice still matters: the new UDPF often needs fewer particles and mixes faster, especially at low signal-to-noise ratios.
- Both new filters are drop-in tools for PMMH because their likelihood estimators are unbiased, so the sampler retains its exact posterior target.
- After averaging over posterior draws, the competing one-step-ahead forecast densities are visually indistinguishable, not merely equal in a single summary score.
Reading between the lines
- The averaging explanation suggests the invariance should extend to other smooth proper scoring rules and to multi-step-ahead forecasts, but the paper does not test those cases.
- If the MCMC budget were sharply reduced, or the forecast target were a rare tail event rather than a smooth log score, the poorly mixing DPF chain might separate from the others; this is a direct, testable consequence of the proposed mechanism.
- The message is tied to PMMH-style forecasting, where the posterior average is the forecast; online or sequential updating schemes that do not average over a posterior chain could still be filter-sensitive.
- The near-identical scores suggest that, in these models, the binding constraint on forecast accuracy is the model specification rather than the filtering approximation; comparing the same filters under a different model class would clarify how far the invariance extends.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies whether the choice of particle filter inside a particle marginal Metropolis-Hastings (PMMH) algorithm affects one-step-ahead forecast accuracy in state space models. It introduces two new filters, the data-driven particle filter (DPF) and the unscented data-driven particle filter (UDPF), proves that their likelihood estimators are unbiased (Theorems 1 and 2, Appendix A.3), and compares them with the bootstrap particle filter, the unscented particle filter, and, in the linear Gaussian case, a fully adapted auxiliary particle filter. The first simulation exercise documents filter-specific computational cost and Markov chain inefficiency factors across linear Gaussian, stochastic conditional duration, and stochastic volatility models under low and high signal-to-noise ratios. The second exercise and the empirical S&P 500 study fix the number of particles and MCMC draws across filters and report average log scores (ALS) and average absolute differences in log scores (ADLS) for one-step-ahead forecasts under correct specification, misspecification, and on real data. The central claim is that, despite large differences in filter efficiency, the filters produce virtually identical forecast accuracy, summarized by the title 'Any filter will do!'.
Significance. If the invariance claim holds, the paper has clear practical value: forecasters using PMMH could choose the simplest particle filter without sacrificing out-of-sample predictive accuracy. The paper also makes a solid methodological contribution by introducing two new filters. The unbiasedness proofs for the DPF and UDPF are carefully laid out in Appendix A.3, and the pseudo-code in Appendix A.1 matches the weight equations in Section 3.2, so the new filtering methodology is credible and reproducible in principle. The computational comparison in Section 4.4, including the Nopt and ALCT measures, is a useful addition to the PMMH literature. However, the central empirical claim rests on point estimates from a single simulated trajectory per DGP and a single empirical data set, without any quantification of Monte Carlo or particle-induced uncertainty, which is the main weakness identified in this report.
major comments (3)
- [Section 4.5, Tables 6 and 7] The central invariance claim is supported only by point estimates of ALS and ADLS from a single simulated trajectory per DGP and a single empirical data set. No standard errors, confidence intervals, paired tests, or repeated-seed results are reported. With ADLS values between 0.003 and 0.034, and with per-period log scores that evidently vary by several times this amount (the ADLS is an average of absolute deviations), the reported differences are within the range that Monte Carlo and particle noise can produce. To make the claim 'virtually identical forecasting accuracy' load-bearing, please provide repeated-seed experiments, paired standard errors for the ALS differences, or an equivalent uncertainty quantification in Tables 6 and 7.
- [Section 4.5, Table 5 and footnote 9] The DPF chain has inefficiency factors of roughly 300-390, so with MH = 5000 the effective number of posterior draws is only about 13-17, and the posterior is refreshed only once every 50 forecast periods. Because the forecast density is an average over these MCMC draws, a poorly mixing chain can still produce a smooth average that hides large conditional differences, as the individual draws in Figures 1 and 2 already illustrate. Please report effective sample sizes for the forecast scores, or results with a substantially larger MH budget, to demonstrate that the near-equal ALS values are not an artifact of averaging over a very small effective posterior sample.
- [Section 4.3 and Section 4.5] The auxiliary particle filter is excluded from the forecast comparisons because its likelihood estimates were unstable in the non-linear models (Section 4.3), yet the title and abstract claim 'Any filter will do!'. The forecast evidence covers only BPF, DPF, UDPF, and UPF; the universal wording is not supported by the tested filter class. Please either include a stable APF variant in the forecast comparison or qualify the claim to the class of filters actually evaluated.
minor comments (4)
- [Section 4.5, text near Figures 1 and 2] The text reads 'the estimated forecast forecast distributions'; the duplicated word should be removed.
- [Appendix A.3, proof of Lemma 1] The sentence 'Since the N permutations of the previous particles are mutually exclusive, each of the terms within the double summation appears exactly L times' is terse; a brief explanation of why cyclic permutations yield this equality would improve readability.
- [Section 3.2.2, Eq. (18)] The notation in Eq. (18) writes a Gaussian approximation to the measurement density with a proportionality constant, but the normalizing constant is not tracked in the subsequent weight formula. Since Eq. (21) uses the exact p(yt+1|xt+1,θ) in the numerator, the final weight is correct, but the presentation would be clearer if the approximation in (18) were explicitly labeled as used only for proposal construction.
- [Section 4.5, footnote 9] The text in Section 4.5 says the posterior is refreshed only after 50 forecast periods, but it is not immediately clear whether every forecast uses the most recent posterior or whether the same set of MCMC draws from the refresh point is used for all 50 periods. Please state this explicitly.
Circularity Check
No circularity: forecast invariance is an empirical finding, and the new filters' unbiasedness is proved in-paper via standard importance sampling.
full rationale
The paper's central claim is that different particle filters, embedded in PMMH, yield virtually identical one-step-ahead forecast accuracy. This is an empirical result reported from simulations and an empirical study (Tables 6-7), not a quantity fitted to the forecast outcomes. The DPF and UDPF likelihood estimators are not calibrated to forecasting performance; Appendix A.3 proves their unbiasedness under explicit conditions C1-C3 using a standard importance-sampling argument adapted from Pitt et al. (2012). That unbiasedness guarantees the PMMH chain targets the same posterior p(theta|y1:T) for any filter, but it does not by construction force the finite-sample average log scores to be nearly equal; the near-equality is documented rather than imposed. The self-citation to Ng et al. (2013) supplies the delta-function representation in equation (11), which is a parameter-free change-of-variables identity, and the paper itself derives the DPF proposal weights and proves the required unbiasedness of the resulting likelihood estimators. This is not a load-bearing self-citation chain nor an imported uniqueness theorem. No fitted parameter is relabeled as a prediction, and no ansatz is smuggled in solely through a citation. The skeptical concerns about Monte Carlo uncertainty, single simulation trajectories, and the high inefficiency factor of the DPF chain are statistical-evidence concerns, not circularity: they do not show that a result is equivalent to its inputs. Therefore, no specific circular reduction can be exhibited, and the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (5)
- Number of particles N in forecast experiments =
300
- Number of PMMH draws MH =
5000
- Posterior refresh period =
50
- DPF multiple match count L in Section 4.4 =
30
- Likelihood variance target for Nopt =
0.85
assumptions (4)
- domain assumption Conditions C1-C3: the measurement equation h(x,eta) is strictly monotone in eta with continuous non-zero partial derivatives, and the implied inverse x(y,eta) is monotone with finite first two moments of the normalized measurement density.
- standard math The unbiased likelihood estimator property of pseudo-marginal MCMC (Andrieu et al., 2010) holds for all filters used.
- domain assumption The Gaussian proposal in the UDPF, with moments from the unscented transformation, is a sufficiently accurate approximation for the models considered.
- domain assumption The PMMH chains have converged and the final particle filter with N=300 gives a consistent estimate of the filtered state distribution for forecasting.
Cite this review
Pith. "Pith review of Forecasting observables with particle filters: Any filter will do!." pith.science (2026). https://pith.science/paper/JRQFS4OH
@misc{pith2026190807204,
author = {Pith},
title = {Pith review of: Forecasting observables with particle filters: Any filter will do!},
year = {2026},
howpublished = {\url{https://pith.science/paper/JRQFS4OH}},
note = {Machine review of arXiv:1908.07204}
}
read the original abstract
We investigate the impact of filter choice on forecast accuracy in state space models. The filters are used both to estimate the posterior distribution of the parameters, via a particle marginal Metropolis-Hastings (PMMH) algorithm, and to produce draws from the filtered distribution of the final state. Multiple filters are entertained, including two new data-driven methods. Simulation exercises are used to document the performance of each PMMH algorithm, in terms of computation time and the efficiency of the chain. We then produce the forecast distributions for the one-step-ahead value of the observed variable, using a fixed number of particles and Markov chain draws. Despite distinct differences in efficiency, the filters yield virtually identical forecasting accuracy, with this result holding under both correct and incorrect specification of the model. This invariance of forecast performance to the specification of the filter also characterizes an empirical analysis of S&P500 daily returns.
Figures
Reference graph
Works this paper leans on
-
[1]
Andrieu, C., Doucet, A. and Holenstein, R. 2010. Particle Markov chain Monte Carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 72(3), pp. 269-342
work page 2010
-
[2]
Andrieu, C. and Roberts, G. 2009. The pseudo-marginal approach for efficient Monte Carlo computations. The Annals of Statistics , 37(2), pp. 697-725
work page 2009
-
[3]
Au, C. and Tam, J. 1999. Transforming variables using the Dirac generalized function. The American Statistician 53, 270-272
work page 1999
-
[4]
Beaumont, M. A. 2003. Estimation of population growth or decline in genetically monitored populations. Genetics 164, 1139–1160
work page 2003
-
[5]
Bauwens, L. and Veredas, D. 2004. The stochastic conditional duration model: a latent variable model for the analysis of financial durations. Journal of Econometrics , 119(2), pp. 381-412
work page 2004
-
[6]
Chopin, N. and Singh S.S. 2015. On particle Gibbs sampling. Bernoulli, 21(3), pp. 1855–1883
work page 2015
-
[7]
A survey of sequential Monte Carlo methods for economics and finance
Creal, D., 2012. A survey of sequential Monte Carlo methods for economics and finance. Econo- metric Reviews, 31(3), pp. 245-296
work page 2012
-
[8]
Deligiannidis, G., Doucet, A. and Pitt, M.K. 2018. The correlated pseudomarginal method, Journal of the Royal Statistical Society, Series B, 80, 839-870
work page 2018
Show all 42 references
-
[9]
Del Moral, P. 2004. Feynman-Kac formulae: Genealogical and Interacting Particle Systems with Applications. New York, Springer Verlag
2004
-
[10]
and Zhang, X
Del Moral, P., Jasra, A., Lee, A., Yau, C. and Zhang, X. 2015. The alive particle filter and its use in particle Markov chain Monte Carlo. Stochastic Analysis and Applications , 33(6), pp. 943-974
2015
-
[11]
and Murray, L.M
Del Moral, P. and Murray, L.M. 2015. Sequential Monte Carlo with highly informative observa- tions. SIAM/ASA Journal on Uncertainty Quantification , 3(1), pp. 969-997
2015
-
[12]
and Johansen, A.M
Doucet, A. and Johansen, A.M. 2011. A tutorial on particle filtering and smoothing: fifteen years later. In: Crisan, D., and Rozovskii, B. (eds.) The Oxford Handbook of Nonlinear Filtering , New York, Oxford University Press. 31
2011
-
[13]
and Lee, A
Doucet, A. and Lee, A. 2018. Sequential Monte Carlo Methods. Handbook of Graphical Models, Eds M. Maathuis. M. Drton, S. Lauritzen and M. Wainwright, Chapman and Hall/CRC
2018
-
[14]
and Kohn, R
Doucet, A., Pitt, M.K., Deligiannidis, G. and Kohn, R. 2015. Efficient implementation of Markov chain Monte Carlo, Biometrika, 102, 295-313
2015
-
[15]
Duffie D, Pan J, Singleton K. 2000. Transform analysis and asset pricing for affine jump-diffusions. Econometrica, 68: 1343–1376
2000
-
[16]
and Shephard, N
Flury, T. and Shephard, N. 2011. Bayesian inference based only on simulated likelihood: particle filter analysis of dynamic economic models. Econometric Theory, 27(5), pp. 933-956
2011
-
[17]
Fox, D., Thrun, S., Burgard, W., and Dellaert, F. 2001. Particle filters for mobile robot localiza- tion. In: Doucet, A., de Freitas, N., and Gordon, N. (eds.) Sequential Monte Carlo Methods in Practice. New York, Springer Verlag
2001
-
[18]
and Amisano, G
Geweke, J. and Amisano, G. 2010. Comparing and evaluating Bayesian predictive distributions of asset returns. International Journal of Forecasting, 26, pp. 216-230
2010
-
[19]
Giordani, P., Pitt, M.K., and Kohn, R. 2011. Bayesian inference for time series state space Models. In: Geweke, J., Koop, G., and van Dijk, H. (eds.) The Oxford Handbook of Bayesian Econometrics, New York, Oxford University Press
2011
-
[20]
and Smith, A.F
Gordon, N.J., Salmond, D.J. and Smith, A.F. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. Radar and Signal Processing, IEE Proceedings F, 140(2), pp. 107-113
1993
-
[21]
and Lee, A
Guarniero, P., Johansen, A.M. and Lee, A. 2017. The iterated auxiliary particle filter. Journal of the American Statistical Association , 112, 1636-1647
2017
-
[22]
Iyer, H., Lai, R.C.S
Hannig, J. Iyer, H., Lai, R.C.S. and Lee, T.C.M. (2016). Generalized fiducial inference: a review and new results. Journal of the American Statistical Association , 111(515), pp. 1346-1361
2016
-
[23]
and Uhlmann, J.K
Julier, S.J. and Uhlmann, J.K. 1997. New extension of the Kalman filter to nonlinear systems. Proceedings Signal Processing, Sensor Fusion, and Target Recognition VI, 3068, pp. 182-193
1997
-
[24]
J., Uhlmann, J.K
Julier, S. J., Uhlmann, J.K. and Durrant-Whyte, H.F. 1995. A new approach for filtering non- linear systems. Proceedings of the 1995 American Control Conference , 3, pp. 1628-1632
1995
-
[25]
J., Uhlmann, J.K
Julier, S. J., Uhlmann, J.K. and Durrant-Whyte, H.F. 2000. A new method for the nonlinear transformation of means and covariances in filters and estimators. IEEE Transactions on Auto- matic Control, 45(3), pp. 477-482
2000
-
[26]
and Doucet, A
Klaas, M., de Freitas, N. and Doucet, A. 2012. Toward practical N2 Monte Carlo: the marginal particle filter. arXiv preprint arXiv:1207.1396. 32
2012 arXiv
-
[27]
and Chen, R
Lin, M.T., Zhang, J.L., Cheng, Q. and Chen, R. 2005. Independent particle filters. Journal of the American Statistical Association , 100(472), pp. 1412-1421
2005
-
[28]
and Liu, J.S
Lin, M., Chen, R. and Liu, J.S. 2013. Lookahead strategies for sequential Monte Carlo. Statistical Science, 28(1), pp. 69-94
2013
-
[29]
and Sch¨ on, T.B
Lindsten, F., Jordan, M.I. and Sch¨ on, T.B. 2014. Particle Gibbs with ancestor sampling.Journal of Machine Learning Research 15, pp. 2145-2184
2014
-
[30]
and Martin, G.M
Maneesoonthorn, W., Forbes, C.S. and Martin, G.M. 2017. Inference on self-exciting jumps in prices and volatility using high-frequency measures. Journal of Applied Econometrics , 32, pp. 504-532
2017
-
[31]
M¨ uller, C.L. 2010. Exploring the common concepts of adaptive MCMC and Covariance Ma- trix Adaptation schemes. Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum f¨ ur Informatik
2010
-
[32]
and McCabe, B.P
Ng, J., Forbes, C.S., Martin, G.M. and McCabe, B.P. 2013. Non-parametric estimation of forecast distributions in non-Gaussian, non-linear state space models. International Journal of Forecast- ing, 29(3), pp. 411-430
2013
-
[33]
and Shephard, N
Pitt, M.K. and Shephard, N. 1999. Filtering via simulation: Auxiliary particle filters. Journal of the American Statistical Association , 94(446), pp. 590-599
1999
-
[34]
and Kohn, R
Pitt, M.K., dos Santos Silva, R., Giordani, P. and Kohn, R. 2012. On some properties of Markov chain Monte Carlo simulation methods based on the particle filter. Journal of Econometrics , 171(2), pp. 134-151
2012
-
[35]
Villani V
Quiroz, M., Tran, M-N. Villani V. and Kohn, R. 2018. Speeding up MCMC by delayed acceptance and data subsampling, Journal of Computational and Graphical Statistics , 27, 12-22
2018
-
[36]
Villani V
Quiroz, M., Tran, M-N. Villani V. and Kohn, R. 2019. Speeding up MCMC by efficient data subsampling, Journal of the American Statistical Association , 114(526), pp. 831-843
2019
-
[37]
Shephard, N. 2005. Stochastic Volatility: Selected Readings. New York, Oxford University Press
2005
-
[38]
and Santos, A.A.F
Smith, J.Q. and Santos, A.A.F. 2006. Second-order filter distribution approximations for financial time series with extreme outliers. Journal of Business & Economic Statistics , 24(3), pp. 329-337
2006
-
[39]
Forbes, C.S
Strickland, C.M. Forbes, C.S. and Martin, G.M. 2006. Bayesian analysis of the stochastic condi- tional duration model, Computational Statistics and Data Analysis, Special Issue on Statistical Signal Extraction and Filtering, 50, pp. 2247-2267. 33
2006
-
[40]
Taylor, S.J. 1982. Financial returns modelled by the product of two stochastic processes, a study of daily sugar prices 1961-79. In: Anderson, O. D. (ed.), Time Series Analysis: Theory and Practice 1, North-Holland, Amsterdam
1982
-
[41]
van de Merwe, R., Doucet, A., de Freitas, N., and Wan, E. 2000. The un- scented particle filter, advances in neural information processing systems. Available at http://books.nips.cc/papers/files/nips13/MerweDoucetFreitasWan.pdf
2000
-
[42]
and Lee, A
Whiteley, N. and Lee, A. 2014. Twisted particle filters. The Annals of Statistics , 42(1), pp. 115-141. 34
2014
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.