{"id":"bd61bc06-6f92-46c0-a093-208326b043ae","arxiv_id":"2412.14339","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A two-component Bayesian model, combining an asymmetric-Gaussian ILI forecast with a discrepancy term and a hospitalization regression, placed second among 21 non-ensemble models in the 2023-24 CDC FluSight challenge.","lead":"Statisticians built a two-component Bayesian model that forecasts flu-like illness and then maps those forecasts to hospitalization counts. In the 2023-24 CDC FluSight challenge, their model placed second among 21 non-ensemble forecasts, suggesting a practical way to use years of ILI data for the newer hospitalization target.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The out-of-sample validity of the 2023 FluSight comparison hinges on whether the empirical Bayes prior for the ASG baseline parameter λ is estimated using full-season 2023 ILI data; the text is ambiguous, so the 'outperformed all but one' claim is not yet verifiable.","rationale":"The most load-bearing assumption is the unstated timing of the empirical Bayes prior for λ_s. If the prior is centered using an MLE computed from the full target season, the 2023-24 forecasts are not out-of-sample and the comparison with FluSight teams is unfair; this would directly undermine the claim that ASGD NORM2 ranked second. This is more fundamental than the normal-versus-log error distribution mismatch, because the latter concerns the adequacy of the forecast distribution, not the integrity of the evaluation. The paper's simulation study (Section 4) is a leave-one-season-out design and appears to respect temporal separation, but it simulates log-hospitalizations while the real-data model uses normal errors on raw counts; even if this discrepancy were resolved, it would not settle the timing of the empirical Bayes prior. I therefore agree with the reader's CONDITIONAL verdict: the paper is competent, the simulation supports the framework, and the conclusion includes a sensible caveat that performance may not carry to every season, but the real-data headline claim cannot be fully assessed until the prior-estimation data window is clarified and, if needed, the evaluation is re-run with a strictly causal prior. The reader's formal weakest_assumption (normal versus log) is not the same as mine, but the reader's rationale already names the empirical-Bayes timing as the most serious concern, so there is partial agreement.","tokens_in":19429,"tokens_out":8935,"duration_ms":71183,"concrete_test":"Inspect the model-fitting code or supplementary material to determine the data window used to compute the empirical Bayes prior for λ. For a representative forecast date (e.g., the week containing the 2023 season peak), check whether the MLE of λ_2023 is computed from (a) the full 2023 season ILI series (weeks 1–52 or 1–53), (b) only ILI data available on or before that forecast date, or (c) no 2023 data at all. If (a), re-run the 2023 forecast evaluation with the prior centered using only seasons 2010–2022 plus, if needed, the 2023 data up to each forecast date; then recompute the LWIS ranking against the 20 FluSight models. If ASGD NORM2 no longer ranks second, the central claim is not out-of-sample. If the code is unavailable, re-implement the model and compare forecast scores under the two prior-centering schemes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that ASGD NORM2 'outperformed all but one' of 20 FluSight models in 2023-24—requires that the forecasts be genuinely out-of-sample. Section 3.6 introduces an empirical Bayes prior for the ASG baseline parameter: 'we use empirical Bayes and center the first component of the hierarchical prior θ to be the mean of MLEs for λ_s across all seasons and set the variance parameter so the prior was tight around the mean.' The text never states whether 'all seasons' includes the 2023 season, nor, if it does, whether the 2023 λ MLE is computed from the complete ILI time series or only from data available at each forecast week. If the full 2023 trajectory is used, every forecast made during that season is informed by future observations, violating the out-of-sample condition that the FluSight comparison depends on. This could inflate the apparent skill of ASGD NORM2 and change its rank. The reader flagged this as the most serious concern in the rationale, though the formal weakest_assumption focused on the normal-versus-log error specification; the latter is a model-adequacy issue, whereas the empirical-Bayes timing directly threatens the validity of the headline comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a two-component Bayesian framework for probabilistic forecasting of weekly influenza hospitalizations in the US. The first component is a hierarchical nonlinear model for influenza-like illness (ILI), using either an SIR compartmental model or an asymmetric Gaussian (ASG) function, with an optional reverse-random-walk discrepancy term. The second component is an ARX(1) model in which hospitalizations are regressed on ILI forecasts, with the two modules combined in the style of a Bayesian cut. The authors evaluate the framework in a leave-one-season-out simulation study of log-hospitalizations and in a real-data analysis of 2023-24 state and national hospitalization forecasts, comparing their ASGD NORM2 model against 20 non-ensemble FluSight submissions. The central claim is that the proposed approach outperformed all but one of those 20 models over the 2023 season.","tokens_in":19702,"tokens_out":5596,"duration_ms":50199,"significance":"If the central claim survives scrutiny, the paper provides a practically useful and computationally feasible forecasting framework that leverages the longer ILI record to forecast the newer hospitalization target. The evaluation has notable strengths: it uses proper scoring rules, compares against external FluSight benchmarks rather than only internal baselines, and reports shared-target relative scores (RLWIS) to mitigate missing-forecast artifacts. The simulation study is genuinely out-of-sample through the leave-one-season-out design. However, two load-bearing issues currently prevent the headline claim from being accepted at face value: the timing of the empirical Bayes prior for the ASG baseline parameter is not specified relative to the forecast season, and the simulation study validates a log-hospitalization model while the real-data evaluation uses a normal model on raw counts with truncation. These concerns are fixable but require additional analysis or explicit clarification.","major_comments":[{"comment":"The out-of-sample validity of the 2023 FluSight comparison is not established because the empirical Bayes prior for the ASG parameter lambda_s is described as being centered at the mean of MLEs 'across all seasons' without stating whether the 2023 season is included or whether the 2023 MLE, if used, is computed only from data available at each forecast week. Section 3.6 also states that the prior variance was set so that the prior was 'tight around the mean' and that this choice 'led to improved mixing of posterior draws and better forecasts,' but the text does not say whether this tuning was carried out on the 2023 forecast targets. If the full 2023 ILI trajectory enters the prior, then every forecast in Section 5 is informed by future observations, which would invalidate the comparison against the 20 FluSight models. The authors must state precisely which seasons and which data windows are used to construct this prior, and if the full 2023 season enters, rerun the analysis with a prior constructed only from seasons and weeks preceding each forecast date.","section":"Section 3.6 and Section 5"},{"comment":"The simulation study validates a different hospitalization model from the one used in the real-data headline analysis. Section 4 states that 'we simulated log-hospitalizations' and that the data-generating parameters were estimated by fitting model (6) after log-transforming hospitalizations, whereas Section 3.8 describes a normal-error model on raw hospitalization counts whose negative predictive draws are truncated to zero for the Section 5 analysis. The LOSO simulation therefore provides evidence for the log-scale version of the model, but it does not directly support the adequacy of the normal-on-raw-counts model with truncation that produces the reported 2023 forecasts. Without a simulation or calibration check for that exact error specification, the favorable scores of ASGD NORM2 in Table 2 could be driven by an error model that the simulation never tested. The authors should either rerun the simulation under the normal/truncated specification used in Section 5 or add an explicit demonstration that the two error specifications lead to equivalent forecast performance for the 2023 targets.","section":"Section 4 versus Sections 3.8 and 5"},{"comment":"The abstract and conclusion state that the proposed method 'outperformed all but one of 20 competing models' without specifying that this ranking is under the log-weighted interval score (LWIS). Table 2 shows that ASGD NORM2 has the best mean WIS but is second to UGA flucast-INFLAenza on mean LWIS and median LWIS, so the claim is sensitive to the scoring rule. The authors should state the criterion in the abstract and conclusion, or limit the claim to the LWIS metric, to avoid the impression that the proposed model dominates under all reported scores.","section":"Abstract, Section 5, and Table 2"}],"minor_comments":[{"comment":"The description of the posterior predictive sampling says that negative values were 'set to 0,' but it does not describe how the corresponding quantiles were recomputed after truncation; a brief explanation of the quantile construction from the truncated samples would improve reproducibility.","section":"Section 3.8"},{"comment":"The weighted interval score formula uses the index k in the summation and the weight w_k, while the surrounding text defines the weights as w_b; this notational inconsistency should be fixed.","section":"Equation (9)"},{"comment":"The caption says 'the 21 models being compared' while the text says 'all 22 models being compared'; the count should be reconciled, especially because Table 2 includes both ASGD NORM2 and SIRD NORM2 in addition to the 20 external FluSight models.","section":"Figure 9 caption and Section 5 text"},{"comment":"There are numerous typos and misspellings, including 'assymetric,' 'intveral,' 'hierarhical,' 'remeiniscent,' 'indicateds,' 'referrred,' and 'NOMR2' in Section 5; a careful copyedit is needed.","section":"Throughout"},{"comment":"The simulation study fixes the hospitalization parameters at posterior means from the 2022 season and chooses weeks 14, 20, 26, 32, and 38 for forecasting, but it does not state a random seed or provide code for replicating the 500 Monte Carlo replicates; including these details would strengthen reproducibility.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue for the editor is the empirical Bayes timing question in Section 3.6. If the full 2023 season is used in the prior, the 'outperformed all but one' claim is not an out-of-sample result as currently presented, and the manuscript would need a substantive rerun of the 2023 evaluation. If the authors can confirm that the prior is constructed from pre-forecast data only, and the simulation/real-data error-model mismatch is addressed with an additional experiment or explicit justification, the paper could be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid applied forecasting paper, but the headline 'outperformed all but one of 20 models' rests on an ambiguity the authors need to clear up before that claim is verifiable.\n\nWhat's actually new: they couple an asymmetric-Gaussian ILI model with a shared discrepancy term (from Osthus et al.) and pipe ILI forecasts into a hospitalization ARX model via a Bayesian cut. The specific combination is new, and the LOSO simulation is properly out-of-sample: ASGD beats the other ILI models, especially at the holiday week, and the coverage plots show real calibration gains. The real-data comparison against 20 FluSight submissions is done fairly on shared forecasts, and the paper reports both WIS and LWIS. The conclusion is appropriately hedged, saying performance may vary by season.\n\nWhere it's soft: the stress-test concern is the one that matters. Section 3.6 says the empirical Bayes prior for the ASG baseline λ is centered on the mean of MLEs 'across all seasons,' but it never states whether the 2023 season's MLE is computed from data available at each forecast week or from the full season. If it's the full season, every forecast in the 2023 comparison is informed by the future, and the 'outperformed all but one' claim is inflated. The paper also doesn't describe the weekly refitting protocol for the real-data forecasts, so I can't tell when the EB prior is updated. This is not a manufactured flaw; it's an unstated detail on which the central empirical claim depends.\n\nThe second soft spot is the normal-versus-log mismatch: the simulation validates a log-hospitalization model, but the real data uses a normal model on raw counts with negative draws truncated. That's a lesser issue because they tried alternatives and the normal worked, but it does weaken the simulation's direct support.\n\nMinor: one season of real-data evaluation, small score margins in places, and the model selection across variants isn't pre-registered. None of that by itself kills the paper.\n\nWho's it for: applied statisticians and infectious-disease forecasters who want an off-the-shelf Bayesian cut framework for two-component forecasting. It's worth serious refereeing; a referee should push for a clear statement of the EB prior timing, and ideally a sensitivity analysis showing the comparison holds when the prior is fit only on pre-2023 data.","headline":"A useful Bayesian two-component forecasting framework with a real simulation study, but the headline 2023-24 comparison depends on an unstated detail about when the empirical Bayes prior is fit.","tokens_in":20208,"tokens_out":2968,"would_cite":false,"duration_ms":31086,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","62P10","92D30"],"pacs":[],"model":"deepseek-v4-flash","headline":"A Bayesian two-module scheme with a discrepancy term produces accurate probabilistic forecasts of weekly influenza hospitalizations, beating 19 of 20 non-ensemble FluSight submissions in the 2023-24 season.","keywords":["influenza forecasting","Bayesian hierarchical model","model discrepancy","probabilistic forecasting","hospitalization forecasts","FluSight","asymmetric Gaussian function","weighted interval score"],"falsifier":"Refit the ASGD ILI module with log-normal, truncated-normal, and negative-binomial hospitalization error models and rerun the 2023-24 FluSight comparison on exactly the same targets; if any of these alternatives beats ASGD NORM2 or changes its rank by more than one position, the normal-error assumption is load-bearing rather than incidental. A complementary check is to apply the log-scale model that was validated in the simulation to the raw hospitalization data and see whether its interval scores match the paper's real-data results.","tokens_in":19148,"feed_emoji":"🦠","tokens_out":8364,"duration_ms":67900,"temperature":0.7,"pith_summary":"The paper claims that weekly influenza hospitalizations can be forecast reliably by chaining two Bayesian models: a hierarchical model that fits seasonal flu-like-illness (ILI) data with a nonlinear curve plus a systematic discrepancy term, and a regression model that converts forecast ILI into hospitalization counts. Because ILI records go back to 2010 while reliable hospitalization data exist only since 2021, the framework lets the long ILI history drive forecasts of the newer target. In a simulation study the discrepancy-augmented asymmetric Gaussian (ASGD) model gave the lowest weighted interval scores and the closest-to-nominal interval coverage among six models. In the real 2023-24 season, the ASGD NORM2 configuration outperformed 19 of the 20 non-ensemble FluSight submissions by mean log-weighted interval score and showed better calibration than the competitors. If the claim holds, public health agencies get an inexpensive, fast, and competitive probabilistic hospitalization forecast from data that are already collected.","feed_headline":"Bayesian two-stage flu model outperforms 19 of 20 FluSight teams","feed_subtitle":"Coupling 13 years of ILI data with a discrepancy-adjusted curve yields sharp, well calibrated hospitalization forecasts.","key_machinery":"The load-bearing object is the two-module forecast pipeline joined by a Bayesian cut. Module one is a hierarchical Beta regression for ILI with logit mean $f_{\\theta_s}(w)+\\gamma_w+\\upsilon_{s,w}$, where $f_{\\theta_s}$ is either the infectious compartment of an SIR model or the asymmetric Gaussian (ASG) function, a two-piece bell-shaped curve whose rise and fall rates can differ; $\\gamma_w$ is a reverse-random-walk discrepancy shared by all seasons and $\\upsilon_{s,w}$ is a current-season discrepancy. Module two is the hospitalization model $H_{s,w}=\\alpha_{0s}+\\alpha_{1s}(\\mathrm{ILI}_{s,w}P)+\\alpha_{2s}(\\mathrm{ILI}_{s,w}P)^2+\\phi H_{s,w-1}+\\epsilon_{s,w}$, an ARX(1) regression. The modules are fit separately with MCMC, and draws from the ILI posterior predictive distribution are plugged into the hospitalization model to produce forecast quantiles. This machinery carries the argument because it transfers information from thirteen seasons of ILI data to a target with only two seasons of hospitalization data, and the discrepancy terms correct systematic shape errors such as the underprediction of the holiday-week peak that a smooth curve alone would miss.","core_discovery":"The paper's central claim is that a two-component Bayesian framework—an ILI module whose logit mean is a nonlinear trajectory function plus a season-shared and a season-specific reverse-random-walk discrepancy term, coupled through a Bayesian cut to an ARX(1) hospitalization module with ILI as a linear or quadratic predictor—produces accurate and well calibrated 1- to 4-week-ahead probabilistic forecasts of state and national flu hospitalizations. The preferred configuration, ASGD NORM2, uses the asymmetric Gaussian function with discrepancy for ILI and a normal-error hospitalization model with a quadratic ILI term. In the 2023 FluSight comparison the authors report that this scheme outperformed all but one of the 20 non-ensemble competing models on the log-weighted interval score, and that the discrepancy component was particularly valuable around the holiday-week ILI peak. The paper intends these results as evidence that long ILI records can compensate for the short hospitalization record and that careful discrepancy modeling, not a more complex mechanistic model, is what most improves forecast skill.","pith_inferences":["A natural stress test the paper does not run is a season with an early or late ILI peak: the discrepancy random walk is trained on historical holiday-week patterns, so a shifted peak could erode the forecast gains.","If the normal-error mismatch between the simulation and real-data analyses is real, switching to a count-valued or log-scale hospitalization model that preserves calibration could make the FluSight ranking more robust.","The same two-module template should transfer to other respiratory diseases that have long syndromic surveillance and short hospitalization records, since nothing in the structure is influenza-specific.","Spatially informed pooling of the ILI or hospitalization modules is the most direct route to closing the small gap to the winning FluSight model and to improving forecasts for states like Montana, where ASGD NORM2 underperformed."],"forward_implications":["State and national influenza hospitalization forecasts can exploit ILI data back to 2010, sidestepping the short hospitalization record.","Adding a discrepancy term to the ILI curve improves forecast skill and calibration, especially for forecasts that straddle the holiday-week ILI peak.","Fitting the ILI and hospitalization modules separately and coupling them through a Bayesian cut is faster, more stable, and more accurate in this application than fitting the joint model.","A relatively simple two-stage regression scheme can compete with specialized FluSight forecast submissions, including mechanistic and machine-learning models.","The residual gap to the best FluSight model points to spatial structure as the next improvement, since the one model that outperformed ASGD NORM2 was a spatial model."],"supporting_citations":[{"why":"Supplies the hierarchical discrepancy modeling framework and the SIR-based ILI model that motivates the discrepancy terms in the ILI module.","marker":"[Osthus et al., 2019]"},{"why":"Supplies the asymmetric Gaussian function and Bayesian hierarchical ILI model that the ASGD configuration adapts.","marker":"[Ulloa, 2019]"},{"why":"Introduces the model discrepancy concept used to capture systematic differences between the nonlinear function and observed ILI.","marker":"[Kennedy and O'Hagan, 2001]"},{"why":"Provides the Bayesian cut formalism that justifies fitting the ILI and hospitalization modules separately and plugging ILI forecasts into the hospitalization model.","marker":"[Plummer, 2015]"},{"why":"Defines the FluSight hospitalization forecast target and scoring setup used in the 2023-24 real-data comparison.","marker":"[Mathis et al., 2024]"},{"why":"Supplies the weighted interval score (WIS) and log-weighted interval score (LWIS) that are the paper's evaluation metrics.","marker":"[Bracher et al., 2021]"},{"why":"Source of the weekly NHSN hospitalization counts used to fit and evaluate the hospitalization model.","marker":"[HealthData.gov, 2024]"}],"fun_headline_variants":["Bayesian two-stage model beats 19 of 20 FluSight teams","Discrepancy term boosts flu hospitalization forecast accuracy","ILI data sharpens hospital flu forecasts despite short record","Two-stage Bayesian model tops 19 of 20 flu forecasters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The real-data evaluation assumes that the normal-error ARX model on raw hospitalization counts, with negative forecast draws truncated to zero, is adequate for the 2023-24 season even though the simulation study validated a log-transformed version of the model; if that error specification is misspecified, the reported forecast scores and calibration may not reflect the method's true performance.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian two-stage model beats 19 of 20 FluSight teams","Discrepancy term boosts flu hospitalization forecast accuracy","ILI data sharpens hospital flu forecasts despite short record","Two-stage Bayesian model tops 19 of 20 flu forecasters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000364,"raw_usage":{"total_tokens":2028,"prompt_tokens":1083,"completion_tokens":945,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":699,"completion_tokens_details":{"reasoning_tokens":875}},"tokens_in":699,"tokens_out":945,"duration_ms":8215,"temperature":1.0,"reasoning_tokens":875,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:19:53.750992+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Refit the ASGD ILI module with log-normal, truncated-normal, and negative-binomial hospitalization error models and rerun the 2023-24 FluSight comparison on exactly the same targets; if any of these alternatives beats ASGD NORM2 or changes its rank by more than one position, the normal-error assumption is load-bearing rather than incidental. A complementary check is to apply the log-scale model that was validated in the simulation to the raw hospitalization data and see whether its interval scores match the paper's real-data results.","supporting_citations":[],"review_version":1}