{"id":"235f4f43-a959-4a56-97d6-f1fff6ae94a7","arxiv_id":"2507.09121","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The lookahead PAL filter has substantial positive Monte Carlo bias on the spatiotemporal measles model, so the log-likelihood advantages reported for PAL over a block particle filter are not reliable.","lead":"This paper tests a recently published fast likelihood approximation, the Poisson approximate likelihood (PAL) filter, on a 40-city measles model and finds that its lookahead version overstates log-likelihoods by hundreds of log units. The result challenges the published comparison table that promoted PAL and argues that a simpler block particle filter remains competitive.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that 'PAL does not outperform a block particle filter' is based on comparing different stochastic models: BPF uses Euler/Gaussian (E11), PAL uses discrete/binomial (E14/E15), so the abstract's comparative claim is not directly established.","rationale":"The paper convincingly establishes that the lookahead PAL implementation has substantial positive Monte Carlo bias: E14 vs E15 (same algorithm, only particle count changed) shows an 833.83 log-unit decrease, and E4 vs E5 shows the same on a single unit. This supports the conclusion that WWR's Table 3 does not accurately represent PAL's capabilities. The gap in the argument is the abstract's comparative claim. E11, the BPF benchmark on real data, evaluates a different model (Euler process, Gaussian measurement) from E14/E15 (discrete process, binomial measurement). The paper's own PALV estimate (E13) on the same model as PALL is below E11, which would support the claim, but the paper does not present this as the formal comparison; instead the abstract's wording suggests a direct algorithmic comparison that the experiments do not provide. The reader flagged 'different model parametrizations' as a weakness, which is related but broader; the most specific confound is fproc/fmeas differing between the PAL and BPF evaluations. The reader's other highlighted weakness, the toy-theory transfer in Section 3, is less load-bearing because E9 vs E10 and E14 vs E15 isolate the lookahead empirically, independent of the theoretical mechanism. A single like-for-like BPF run on the discrete/binomial model would settle whether the comparative clause survives.","tokens_in":8808,"tokens_out":16800,"duration_ms":204543,"concrete_test":"Run bpfilter from spatPomp on the same model used by WWR's PAL on the real 40-town data: fproc='discrete', fmeas='binomial', with WWR's case A parameters (or case C if computational cost permits), increasing J until the jackknife standard error of the log-likelihood is small (e.g., <5). Compare this BPF-on-PAL-model estimate with E13's PALV estimate (-74137.62, same discrete/binomial case A model). If BPF >= PALV, the 'PAL does not outperform BPF' claim is supported; if BPF < PALV, the comparative claim fails. As a robustness check, repeat the comparison on simulated data from the discrete/binomial model, where the ground-truth likelihood is available, to verify that BPF on this model achieves the known value with acceptable Monte Carlo error.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's strongest evidence is the bias demonstration: E14 vs E15 shows PALL's log-likelihood drops by 833.83 log units when the particle count increases from J1=5000 to J2=1e5, with a jackknife standard error of about 60 for E14, convincingly indicating positive Monte Carlo bias at J1 and supporting the conclusion that WWR's Table 3 is unreliable. The load-bearing problem is the separate comparative clause in the abstract: 'PAL does not outperform a block particle filter.' The BPF benchmark E11 uses fproc=Euler and fmeas=Gaussian with case C parameters re-optimized via iterated BPF, while the PAL estimates E14/E15 use fproc=discrete and fmeas=binomial with WWR's case A parameters. These are different stochastic models, so the log-likelihood values are not directly comparable; parameter re-tuning cannot make two different likelihood functions equivalent. The simulated comparisons E8–E10 have the same issue: BPF uses the correct coupling C2 while PAL uses C1, so those comparisons mix model misspecification with algorithmic performance. An unbiased, same-model reference for PAL is available in the paper: PALV (E13) on the discrete/binomial case A model gives -74137.62 with small standard error, far below E11 (-67761.82). But the abstract's claim is built on E11 rather than on this like-for-like comparison. Additionally, Section 3's bias theory is demonstrated only for the simpler Rimella et al. lookahead filter, not for PALL, so the size of the bias correction needed to compare a debiased PALL value with E11 is not derived. Thus the headline comparative claim rests on a confounded comparison, even though the bias finding is solid.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper challenges the Poisson approximate likelihood (PAL) filter proposed by Whitehouse et al. (2023) for a spatiotemporal measles metapopulation model. The authors distinguish between the vanilla PAL (PALV) and the lookahead version (PALL) used by WWR, and present a series of numerical experiments on simulated and real data. On a single city with known ground truth, PALL overestimates the log-likelihood while PALV underperforms slightly. On a 40-city system, PALL again overstates the log-likelihood, and increasing the number of particles from 5e3 to 1e5 decreases the PALL estimate by about 834 log units (E14 vs. E15), which signals positive Monte Carlo bias. The paper also argues that PAL does not outperform a block particle filter and offers a toy example to illustrate how self-normalizing resampling in a lookahead filter can produce positive bias. Code and data are provided.","tokens_in":9129,"tokens_out":8898,"duration_ms":88687,"significance":"The numerical demonstration of positive bias in PALL is a significant and credible contribution. The use of multiple independent lines of evidence – known ground truth, comparison with the vanilla filter, and the particle-count dependence – makes the bias claim robust. The paper also contributes a reproducible workflow and careful attention to Monte Carlo standard errors. However, the comparative claim against the block particle filter is not yet supported, because the BPF is evaluated on a different model specification (Euler/Gaussian, case C) than the PALL (discrete/binomial, case A). Furthermore, the theoretical explanation in Section 3 is not directly derived for PALL. These issues limit the paper's current impact but are addressable in revision.","major_comments":[{"comment":"The claim in the abstract that 'PAL does not outperform a block particle filter' is not established by the experiments as reported. The BPF estimate (E11) is computed for a model with fproc=Euler, fmeas=Gaussian, and case C parameters re-optimized for that model, whereas the PALL estimates (E14/E15) use fproc=discrete, fmeas=binomial, and WWR's case A parameters. These are different statistical models, so the log-likelihood values cannot be compared directly; the gap between E14 and E11 mixes model difference with any algorithmic bias. To support the comparative claim, the paper should provide a same-model comparison, such as running BPF on the discrete/binomial model with the WWR parameters, or rephrase the claim to focus on the demonstrated bias.","section":"Abstract and §2 (Table 2, E11 vs E14/E15)"},{"comment":"The theoretical explanation of the positive bias is presented for a two-point example of the Rimella et al. lookahead filter, not for PALL. The text asserts that the bias is a property of the lookahead mechanism and therefore transfers, but it does not show that PALL's resampling scheme has the structure required for the positive bias (e.g., proposal probabilities that can be arbitrarily small). The numerical evidence stands on its own, but the theory section should be framed as an illustrative mechanism rather than an explanation of the PALL bias, unless the connection is made precise.","section":"§3"},{"comment":"In the 40-unit simulated-data experiments, the filter is run with coupling C1 on data simulated from C2, so the comparison of λ10 with λ8 (PALL versus BPF ground truth) conflates model misspecification with algorithmic bias. The comparison of λ10 with λ9 (PALL versus PALV, both using C1) is a valid same-model bias estimate, and it could be presented as the primary evidence from that experiment; the comparison to λ8 should be interpreted more cautiously.","section":"§2, E8–E10"}],"minor_comments":[{"comment":"The sentence 'the decrease of the log-likelihood by λ14−λ15 = -833.83 log units' has a sign error; since λ14 > λ15, the decrease is 833.83, or equivalently λ15 − λ14 = -833.83.","section":"§2 (after Table 3)"},{"comment":"The statement that E2 and E3 'failed to outperform the ARMA benchmark' compares log-likelihoods from simulated data with benchmark values that appear to be computed on the real data; this comparison is not meaningful unless the ARMA model is fitted to the same simulated data.","section":"§2, E2–E3"},{"comment":"The abbreviations PALL and PALV are used from the abstract onward but are not defined until Section 2; please define them at first use.","section":"§1"},{"comment":"The toy example uses notation (e.g., ε, the resampled weights, the role of the lookahead Y) that is compressed; expanding the derivation would improve clarity.","section":"§3"},{"comment":"The notation for parameter values (θ̂*BPF, θ̂PAL) in Table 1 is not explained in the table caption; please specify that these are optimized parameter vectors for each filter.","section":"Table 1"},{"comment":"Each simulated-data treatment is based on a single dataset; the reported standard errors are Monte Carlo errors, but dataset-to-dataset variability is not quantified. Given the large effect sizes, this does not affect the qualitative conclusion, but it limits the precision of quantitative bias estimates such as 80.55 per unit.","section":"§2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a critique of a recent JRSS-B publication, and the authors are developers of the competing block particle filter methodology. The numerical evidence for bias is strong and appropriately cautious in its use of standard errors. The main weakness is the comparative claim against BPF, which is not based on a like-for-like comparison; I would ask the authors to address this before publication. The paper seems suitable for a statistics methodology journal, though the fit to the specific journal's scope should be checked."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core of this paper is the bias finding, and it is solid. E14 versus E15 is the clincher: on the WWR code and parameters, raising the particle count from 5,000 to 100,000 drops the log-likelihood estimate by 834 log units, with a jackknife SE that makes the decrease unambiguous. That is exactly the signature of positive Monte Carlo bias, and it directly undermines the credibility of Table 3 in Whitehouse et al. (2023). The simulated-data experiments with known ground truth (E1, E8) reinforce the point. The paper also deserves credit for shipping code and data on Zenodo and for using the WWR code to generate the PALL numbers, which takes away the usual excuse that the critique is based on a strawman implementation.\n\nThe soft spots are real but they do not touch the central bias result. The abstract's claim that 'PAL does not outperform a block particle filter' is not actually established by the experiments shown. E11 uses the BPF with an Euler/Gaussian model and re-optimized case C parameters; E14/E15 use the discrete/binomial case A model with WWR's published parameters. Those are different likelihood functions. Re-optimizing parameters does not make them commensurable. The same issue contaminates E8–E10, where BPF filters the correct C2 coupling while PAL filters the misspecified C1, mixing model error with algorithmic performance. A like-for-like comparison is available in the paper—E13 (PALV at high particle count on case A) gives -74137.62, well below E11's -67761.82—so the comparative headline should have been built on that, or dropped.\n\nSection 3's theoretical explanation is honest about being a toy demonstration for the Rimella et al. lookahead filter rather than a derivation for PALL specifically. That is a limitation, not a fatal one, because the numerical evidence stands independently. The single simulated dataset per treatment is a minor design weakness, not a reason to doubt the bias.\n\nThe paper is written by the BPF developers, so there is an obvious conflict of interest in any comparison between PAL and BPF. That does not make the bias evidence circular—no quantity in the bias estimate is fit to the conclusion—but it should be acknowledged in a referee report.\n\nWho is this for? Anyone who relies on the WWR measles results or plans to use lookahead PAL. The bias warning generalizes, even if the exact magnitude does not. I would send this to a serious referee with a request to fix the comparative claims and to add the like-for-like E13 versus E11 discussion. The central finding is reproducible and important, and the paper deserves engagement, not desk rejection.","headline":"Solid, reproducible bias demonstration against the lookahead PAL filter on the 40-town measles model, but the abstract's comparative claim against the block particle filter overreaches because it compares different stochastic models.","tokens_in":9698,"tokens_out":1118,"would_cite":true,"duration_ms":15254,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62M05","65C05"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that the Poisson approximate likelihood filter used on a 40-town measles model reports log-likelihood estimates with positive Monte Carlo bias, so its reported advantage over a block particle filter and ARMA benchmarks…","keywords":["Poisson approximate likelihood","lookahead filter","block particle filter","Monte Carlo bias","log-likelihood estimation","spatiotemporal measles model","partially observed Markov process","proper scoring rule"],"falsifier":"Use a modified lookahead filter that is unbiased for the likelihood (for instance, by replacing self-normalization with a correct weighting scheme) on the same 40-town measles data; if the log-likelihood estimate no longer decreases when particles are increased from 5,000 to 100,000 and still beats the block particle filter, the paper's central claim is wrong.","tokens_in":8574,"feed_emoji":"🦠","tokens_out":8461,"duration_ms":89193,"temperature":0.7,"pith_summary":"This paper revisits the Poisson approximate likelihood (PAL) filter applied to a 40-town measles metapopulation model and argues that the method's reported success is an artifact of positive Monte Carlo bias. The lookahead variant of PAL, called PALL, estimates log-likelihood values that fall as the number of particles grows: in the real-data comparison, increasing particles from 5,000 to 100,000 lowers the estimate by about 834 log units. On simulated data with known likelihood, PALL overshoots the true value while the vanilla version falls slightly below it, as an approximate filter should. The paper concludes that PAL does not outperform a standard block particle filter on this problem, and that the published comparison table does not represent PAL's true capabilities.","feed_headline":"More particles lower a measles filter's likelihood by 834 log units","feed_subtitle":"A lookahead twist, not better modeling, explains the filter's claimed edge over standard particle filters.","key_machinery":"The central machinery is the lookahead filter, a sequential Monte Carlo method whose one-step predictions may use data at or after the observation time. Two load-bearing ideas carry the argument. The first is the proper-scoring-rule property of log-likelihood: a genuine forecasting filter cannot beat the exact prediction distribution on average, so simulated data with a known likelihood provide a ground truth against which biased filters can be detected. The second is a two-point toy example of self-normalizing resampling in the lookahead mechanism, showing that with a fixed number of particles most resampled swarms overstate the likelihood while rare swarms carry enormous weights, so the likelihood estimate is positively biased. The paper then uses controlled experiments that vary the filter (PALL, vanilla PAL, block particle filter), particle count, number of towns, and spatial coupling, with the 5,000-versus-100,000 particle comparison isolating the bias in the published 40-town setting.","core_discovery":"The paper's central claim is that the lookahead implementation of PAL inflates its own log-likelihood estimates, so the strong showing reported in the original measles analysis is not trustworthy. The authors establish this by two routes: on datasets simulated from the model, PALL reports log-likelihoods above the known true value, and on both simulated and real data, increasing Monte Carlo effort moves the PALL estimate downward, a behavior inconsistent with unbiased estimation. The most direct evidence is the change from -64014.93 to -64848.76 (about 834 log units) when particles increase from 5,000 to 100,000 for the 40-town data. They further show the bias grows roughly linearly with the number of towns, and that a block particle filter reproduces or exceeds PALL's performance without the bias.","pith_inferences":["Beyond the paper's specific case, the same self-normalizing-lookahead mechanism could inflate likelihood estimates in other lookahead sequential Monte Carlo methods, making the warning a general one for filter developers.","A practical diagnostic follows directly: report log-likelihood as a function of particle count for any lookahead filter, since a decreasing curve is a red flag for positive bias even when the true likelihood is unknown.","Extending the two-point toy example to a broader class of weight distributions might yield a quantitative bias formula in terms of particle count and weight variance, which could let practitioners correct the estimate.","On the modeling side, these results shift the favorable evidence for the measles models back onto the block particle filter, so the overdispersion and coupling parameters from the PAL analysis should be re-estimated with an unbiased filter before being used."],"forward_implications":["The log-likelihood values in the original Table 3 for the measles application should not be treated as reliable evidence for PAL's usefulness on high-dimensional problems.","A plain block particle filter, which applies to a broader class of models, is a sufficient and less biased tool for likelihood evaluation on this metapopulation measles model.","Because the bias scales roughly linearly with the number of towns, conclusions drawn from PAL on larger spatial systems could be even more overstated.","Researchers using lookahead filters for model comparison should check whether log-likelihood estimates move with Monte Carlo effort before interpreting high values as model support.","Vanilla PAL without lookahead fails to beat the ARMA benchmark in these experiments, so the PAL approximation itself does not rescue the high-dimensional measles analysis."],"supporting_citations":[{"why":"Introduces the PAL and PALL filters and supplies the Table 3 measles results that the paper re-examines.","marker":"Whitehouse et al. (2023)"},{"why":"Provides the lookahead mechanism whose self-normalizing resampling the paper identifies as the source of bias.","marker":"Rimella et al. (2023)"},{"why":"Introduces the block particle filter used as the unbiased comparison method.","marker":"Rebeschini and Van Handel (2015)"},{"why":"Establishes that basic particle filters are unbiased for the likelihood, used to justify ground-truth estimates.","marker":"Del Moral (2004)"},{"why":"Supplies the proper-scoring-rule property that makes simulated-data log-likelihood comparisons meaningful.","marker":"Gneiting and Raftery (2007)"},{"why":"Provides the measles data and model specification that both filters are applied to.","marker":"Park and Ionides (2020)"},{"why":"Prior reanalysis of the rotavirus PAL results motivates the current re-examination of the measles application.","marker":"Hao et al. (2024)"},{"why":"The iterated block particle filter is used to optimize parameters for the BPF comparison.","marker":"Ning and Ionides (2023)"}],"fun_headline_variants":["PAL filter's lookahead bias inflates measles likelihood by 834 log units","Block particle filter beats PAL's biased measles likelihood estimates","PAL's lookahead adds 834 log units to measles likelihood","Increasing particles drops PAL's measles log-likelihood by 834","Measles model filter: 834-log-unit bias from PAL's lookahead"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the positive bias measured in the 40-town measles experiments is caused by the self-normalizing lookahead resampling illustrated in the two-point toy example, and not by the Poisson approximation itself or by implementation details specific to this model.","fun_headline_variants_meta":{"raw":{"variants":["PAL filter's lookahead bias inflates measles likelihood by 834 log units","Block particle filter beats PAL's biased measles likelihood estimates","PAL's lookahead adds 834 log units to measles likelihood","Increasing particles drops PAL's measles log-likelihood by 834","Measles model filter: 834-log-unit bias from PAL's lookahead"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000725,"raw_usage":{"total_tokens":3228,"prompt_tokens":904,"completion_tokens":2324,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":2228}},"tokens_in":520,"tokens_out":2324,"duration_ms":19236,"temperature":1.0,"reasoning_tokens":2228,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:03:47.202492+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a modified lookahead filter that is unbiased for the likelihood (for instance, by replacing self-normalization with a correct weighting scheme) on the same 40-town measles data; if the log-likelihood estimate no longer decreases when particles are increased from 5,000 to 100,000 and still beats the block particle filter, the paper's central claim is wrong.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the PAL and PALL filters and supplies the Table 3 measles results that the paper re-examines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the lookahead mechanism whose self-normalizing resampling the paper identifies as the source of bias."},{"cited_title":"and Van Handel, R","cited_arxiv_id":null,"evidence_quote":"Introduces the block particle filter used as the unbiased comparison method."},{"cited_title":"Feynman-Kac Formulae: Genealogical and Interacting Particle Systems with Applications","cited_arxiv_id":null,"evidence_quote":"Establishes that basic particle filters are unbiased for the likelihood, used to justify ground-truth estimates."},{"cited_title":"and Raftery, A","cited_arxiv_id":null,"evidence_quote":"Supplies the proper-scoring-rule property that makes simulated-data log-likelihood comparisons meaningful."},{"cited_title":"and Ionides, E","cited_arxiv_id":null,"evidence_quote":"Provides the measles data and model specification that both filters are applied to."},{"cited_title":"Poisson approximate likelihood compared to the particle filter","cited_arxiv_id":"2409.12173","evidence_quote":"Prior reanalysis of the rotavirus PAL results motivates the current re-examination of the measles application."},{"cited_title":"and Ionides, E","cited_arxiv_id":null,"evidence_quote":"The iterated block particle filter is used to optimize parameters for the BPF comparison."}],"review_version":1}