{"id":"aef3f973-fe2b-4467-8a78-0d0159cb96ee","arxiv_id":"2506.11028","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On EU and US COVID-19 data, mobility data and GCN modules fail to significantly improve transformer-based incidence forecasts, while mortality and hospitalization data markedly improve accuracy.","lead":"This paper tests whether mobility data and graph neural networks improve COVID-19 forecasting in Europe and the US. It finds they do not help much, while adding mortality and hospitalization data clearly improves accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central negative claim about mobility and GCN modules is supported only by null P-values from five-seed t-tests with no power analysis or confidence intervals; the evidence is consistent with an underpowered experiment, so the abstract's 'do not significantly enhance' overstates what is shown.","rationale":"The reader's weakest assumption matches my concern: the experimental design has five seeds, no power analysis, and no confidence intervals, and all reported P-values near 0.5 are compatible with no effect or with an effect too small or noisy to detect. The central claim in the strongest_claim includes the negative conclusion about GCN modules and mobility data, and that is the load-bearing part for the paper's policy message. I do not see an internal contradiction in the reported numbers, and the availability of code and data is a genuine strength, but reproducibility does not cure the power problem. The pilot-study claim that mortality and hospitalization data improve forecasting is on firmer ground because those improvements are significant, although the interpretive step from 'improved prediction' to 'mobility information mined' is not directly tested; that is a secondary concern. The lockdown-correlation analysis is partly circular because the dynamic maps were trained during the same pandemic period, but it is ancillary to the main forecasting claim. Therefore, the conditional verdict is appropriate: the paper should add a power analysis or confidence intervals and soften the abstract unless the concern is settled by the proposed check.","tokens_in":16650,"tokens_out":7329,"duration_ms":76171,"concrete_test":"Reproduce the central comparison with more seeds: run Trans and Trans+Adp on the EU fold-2 split at output length 12 for 20 random initializations instead of 5, keeping all other settings fixed, and compute the paired 95% confidence interval for the mean percentage MAE difference. If the confidence interval is wide enough to include a 3% improvement (for example, roughly -5% to +1%), then the five-seed null is underpowered and the negative claim should be reworded as inconclusive in the abstract. If the 20-seed confidence interval excludes a 3% improvement and is centered near zero, the concern is resolved and the original conclusion stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main negative conclusion (Main experiments, 'Contribution of the GCN module') is an absence-of-evidence argument, not a demonstrated equivalence. All comparisons are one-sided paired t-tests over five random initializations (Methods: 'we train all models five times with different random initializations'), and the paper reports no confidence intervals, effect-size bounds, or power analysis. With N=5, a true improvement of a few percent in MAE will frequently yield P-values in the 0.4-0.5 range when seed-level noise is comparable to the observed fold-level variation; for example, EU Trans+Adp at output length 12 shows -1.83% MAE with P=0.441, and the pooled fold-1 Trans+Adp result is -2.85% with P=0.369. The paper's literal statement that the data 'lack evidence' is defensible, but the abstract and discussion assert that mobility data and GCN modules 'do not significantly enhance' forecasting, and the policy implication that simple temporal and clinical models should be preferred requires ruling out practically meaningful gains, not merely failing to detect them. This is the load-bearing weak point: if the experiment is underpowered, the central negative claim could be wrong even though every reported P-value is insignificant.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper evaluates whether mobility data and graph convolutional modules improve COVID-19 incidence forecasting across EU and US regions. In a pilot study, Google mobility data did not significantly improve a transformer-based model, whereas adding mortality and hospitalization data did. In main experiments, adding a GCN with physical-distance and/or attention-derived adjacency matrices did not significantly improve MAE/RMSE over a vanilla transformer under 5-fold progressive cross-validation and five random seeds. The authors additionally analyze attention-derived dynamic maps, introducing a mobility indicator Π, and report correlations with lockdown orders. Sensitivity analyses compare against ARIMA, tree-based models, and DLinear.","tokens_in":16886,"tokens_out":5180,"duration_ms":49660,"significance":"If the negative results are robust, they provide a useful cautionary finding for epidemic forecasting practice, suggesting that mobility feeds and GCN modules may add little over strong temporal baselines and that clinical surveillance data are more valuable. The study is notable for external validation on two distinct regions, held-out test folds, and public code and data. The main limitation is that the central negative claim is supported only by null-hypothesis tests without a power or equivalence analysis; additionally, the mobility-indicator analysis contains a formula/reporting inconsistency and a potential circularity due to training-set overlap.","major_comments":[{"comment":"The conclusion that GCN modules \"do not significantly enhance\" forecasting is an absence-of-evidence claim. All comparisons are one-sided paired t-tests over N=5 random seeds, and the paper reports no confidence intervals, effect-size bounds, or power analysis. For example, the EU Trans+Adp MAE improvement of -1.83% at output length 12 carries P=0.441, and the pooled fold-1 improvement of -2.85% carries P=0.369; these P-values are exactly what an underpowered study would produce for a modest true effect. Please either add a non-inferiority/equivalence analysis with pre-specified equivalence margins, report effect-size confidence intervals, or explicitly state in the abstract and discussion that the evidence only warrants \"no significant improvement detected,\" not \"no improvement.\"","section":"Main experiments: Contribution of the GCN module; abstract"},{"comment":"The mobility indicator Π is defined with the indicator function I(M_n,(i,j)) applied to the attention score matrix M_n. Since M_n is a softmax-normalized matrix (Eqs. (9)-(10)), every element is strictly positive and Π would equal 1 for all maps. The reported values (e.g., Π=44.40% for EU and 9.48% for US) must have been computed on the thresholded/sparsified adjacency matrix A_s from Eq. (11). Please correct Eq. (3) and ensure all quantitative claims about Π refer to the intended matrix; as written, the formula is inconsistent with the reported numbers.","section":"Correlation between dynamic maps and human movement restrictions, Eq. (3)"},{"comment":"The paper states that \"we analyze all available maps, even if lockdown periods overlap with the training and validation sets.\" Because the dynamic maps are outputs of models trained on data spanning the same lockdown periods, the observed association between Π and lockdown orders could reflect memorization of training labels rather than a learned, generalizable sensitivity to mobility restrictions. Please retrain the map-generating models on pre-lockdown data only (or otherwise exclude lockdown periods from training) to establish that the maps track mobility restrictions out-of-sample, or temper the corresponding claims.","section":"Correlation between dynamic maps and human movement restrictions"},{"comment":"The claim that \"mobility information can be effectively mined from mortality and hospitalization data\" is not directly established by the experiments. The IMH input adds mortality and hospitalization time series, which are clinically informative for future incidence through direct temporal pathways; the paper does not include any control or mediation analysis to attribute the improvement to recovered mobility information. Please either soften this causal claim or add an analysis (e.g., an ablation using shuffled clinical data or an intermediate mobility prediction task) that isolates the mobility component.","section":"Pilot study; Discussion"}],"minor_comments":[{"comment":"The section heading \"Perfomrance analysis for different lengths\" contains a typo and should read \"Performance.\"","section":"Perfomrance analysis for different lengths (heading)"},{"comment":"In Algorithm 1, line 3 sets ρ equal to the density of M_n when Ag exists, which would always be 1 for a softmax-normalized matrix; this contradicts the surrounding text (\"if geographical adj. matrix exists, its sparsity is used\"). Please correct the pseudocode.","section":"Methods, Algorithm 1"},{"comment":"The authors write that the AutoArima function comes from the \"scikit-learn\" package; AutoARIMA is not part of scikit-learn but is available in pmdarima or sktime (the latter is correctly cited in the Data Availability section).","section":"Methods, ARIMA and Linear Layer Deep Learning"},{"comment":"The text refers to \"Supplementary Table 2 and Table 3\" without distinguishing which numbers belong to which table; please make the citations specific and consistent with the supplement numbering.","section":"Correlation between dynamic maps and human movement restrictions"},{"comment":"The caption states the heatmaps are from \"Trans+Adp\" without specifying which output length or data split produced them; please clarify because the values of Π are not defined in the figure.","section":"Figure 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The negative-result framing is valuable and likely of interest to the journal's readership, but the statistical support for the central negative claim needs strengthening before publication. The mobility-indicator analysis is the most novel contribution and should be presented with a corrected formula and a non-circular evaluation; as it stands, the reported Π values cannot be reconciled with Eq. (3). The paper would also benefit from a more careful separation of \"no significant evidence of improvement\" from \"no improvement\" throughout the abstract and discussion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent head-to-head evaluation of mobility data and GCN modules on a transformer backbone for COVID-19 incidence forecasting across the EU and US. The strongest result is the pilot study: adding mortality and hospitalization data (IMH) consistently beats incidence-only and incidence+mobility, with significant gains in both regions. That finding looks real and is worth taking seriously. The second result—that GCN modules with static or dynamic adjacency matrices do not help—is plausible but not demonstrated as a true null. All the t-tests are one-sided over five random seeds, with no power analysis or confidence intervals. The paper itself says 'lacks evidence,' which is defensible, but the abstract says 'do not significantly enhance,' which overstates what an underpowered experiment can support. A true effect of a few percent MAE would frequently produce P-values in the 0.4–0.5 range with N=5. So the negative conclusion should be framed as 'we could not detect a benefit' rather than 'there is no benefit.' The lockdown-correlation analysis is the weakest part: it is qualitative, depends on post hoc threshold choices, and is partly circular because the dynamic maps come from models trained on data spanning the same lockdown periods. I would treat that as illustrative, not evidence. The missing citations for the transformer and GCN are minor but sloppy. On the plus side, the design is mostly clean: held-out test folds, progressive splits, external validation across two regions, and code and data are released. The sensitivity analysis with ARIMA, tree models, and DLinear is useful context. This paper deserves a serious referee: the empirical comparison is exactly the kind of negative result the field needs, and the IMH finding is actionable. The main revision request should be to add a power analysis or equivalence bounds, and to fix the abstract and discussion so they do not claim more than the statistics support. Yes, send it to review.","headline":"A useful empirical null result about mobility data and GCNs in epidemic forecasting, but the abstract overstates the negative claim: the paper shows 'no evidence' rather than 'no effect'.","tokens_in":17433,"tokens_out":1523,"would_cite":true,"duration_ms":14620,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mobility data and GCN layers fail to lift COVID-19 forecast accuracy","keywords":["COVID-19 forecasting","mobility data","graph convolutional networks","multi-task learning","transformer","spatiotemporal forecasting","Google mobility","epidemic modeling"],"falsifier":"A replication using the same transformer backbone and data but with, say, fifty random seeds and per-architecture hyperparameter search that yields a significant MAE or RMSE decrease for the GCN-augmented models over the vanilla transformer would refute the claim that GCN modules lack evidence of contribution.","tokens_in":16420,"feed_emoji":"🦠","tokens_out":4728,"duration_ms":70294,"temperature":0.7,"pith_summary":"This paper sets out to determine whether two popular ingredients in epidemic-forecasting models—mobility data and graph convolutional layers—actually improve predictions of daily COVID-19 incidence. Across 28 EU countries and 49 US states, with a transformer as the temporal backbone, it finds no statistically significant gains from adding mobility data or from adding a GCN module with static, dynamic, or both adjacency matrices. The positive result is that incorporating mortality and hospitalization rates through multitask learning consistently improves forecast accuracy, suggesting that clinical surveillance data can stand in for explicit mobility information. A secondary observation ties the attention-derived dynamic spatial maps to lockdown periods, hinting that these maps may be usable as mobility indicators even when they do not help forecasts.","feed_headline":"Mobility data and GCN layers fail to lift COVID-19 forecast accuracy","feed_subtitle":"Clinical signals like hospitalizations and deaths carry more predictive power, a 28-country/49-state study finds.","key_machinery":"The load-bearing mechanism is a transformer encoder-decoder backbone augmented with a graph convolutional module. Spatial structure enters through two adjacency matrices: a static physical-distance matrix built from a thresholded Gaussian kernel on site coordinates, and a dynamic matrix produced by a spatial attention head with a hard truncating constraint that sparsifies weak weights. A multitask-learning setup feeds incidence, mortality, hospitalization, and, in the pilot study, mobility rates as supplementary variables. The comparison logic is a one-sided Student t-test over five random seeds and progressive folds, with MAE and RMSE as the outcome metrics.","core_discovery":"The paper's central claim is that, on average, the GCN module with either a physical-distance adjacency matrix, an adaptively generated dynamic matrix, or both, lacks evidence of improving transformer-based COVID-19 incidence forecasting; reported one-sided t-tests yield P values near 0.5 for the MAE and RMSE comparisons. In the pilot study, adding mobility data to incidence data also fails to show consistent improvement, whereas replacing mobility with mortality and hospitalization data yields significantly lower errors in both regions. The paper interprets this as evidence that mobility information can be effectively mined from clinical outcomes, and that temporal information dominates spatial graph structure in this forecasting setting.","pith_inferences":["If the null result holds beyond this setup, forecasters should weight clinical surveillance data and parsimonious temporal models more heavily than mobility feeds or graph layers.","The paper's 'no evidence' wording leaves open the possibility of small real effects; a replication with more seeds or per-architecture hyperparameter tuning could detect them.","Because the dynamic maps track lockdown timing, they may be useful as policy surveillance tools even where they do not improve prediction.","A natural next test is whether other mobility sources, such as cell-phone location records, change the pilot-study verdict; the paper only assessed one mobility dataset."],"forward_implications":["Adding mobility data to incidence data does not reliably improve short- or long-term COVID-19 case forecasts.","Mortality and hospitalization rates, used as auxiliary tasks, improve accuracy more than mobility data does and can act as a substitute source of mobility information.","GCN modules offer no significant benefit over a vanilla transformer for this task, and combining static and dynamic adjacency matrices tends to worsen MAE.","Attention-derived dynamic spatial maps correlate with lockdown periods in the EU and may serve as indicators of human movement restrictions.","Spatial modeling may still help in harder settings—longer output horizons and early, data-scarce folds—though the gains are not statistically significant."],"supporting_citations":[{"why":"Supplies the incidence, mortality, and hospitalization data for both the EU and US regions.","marker":"[47]"},{"why":"Provides the multitask-learning idea used to share representations across clinical variables.","marker":"[31]"},{"why":"Supports the use of mobility and syndromic surveillance data as proxies in neural forecasting models.","marker":"[18]"},{"why":"Documents associations between mobility changes and COVID-19 transmission, the basis for treating mobility as informative.","marker":"[21]"},{"why":"Supplies the thresholded Gaussian kernel adjacency matrix construction used for the physical-distance graph.","marker":"[42]"},{"why":"Provides the linear-layer baseline (DLinear) used in the sensitivity analysis for multivariate time series.","marker":"[46]"},{"why":"Exemplifies the cross-location GCN forecasting approach whose contribution this paper tests.","marker":"[27]"}],"fun_headline_variants":["Mobility data and GCNs don't lift COVID-19 forecast accuracy","Clinical signals beat mobility in epidemic forecasting","Hospitalizations and deaths, not mobility, improve COVID forecasts","Graph networks and mobility add nothing to COVID forecasting","Epidemic forecasting: clinical data beats mobility and GCNs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The negative conclusions rest on the assumption that five random seeds, five progressive folds, and shared hyperparameters give the experiments enough statistical power to detect any practically meaningful improvement; if the design is underpowered, the insignificant P values do not rule out real gains.","fun_headline_variants_meta":{"raw":{"variants":["Mobility data and GCNs don't lift COVID-19 forecast accuracy","Clinical signals beat mobility in epidemic forecasting","Hospitalizations and deaths, not mobility, improve COVID forecasts","Graph networks and mobility add nothing to COVID forecasting","Epidemic forecasting: clinical data beats mobility and GCNs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000633,"raw_usage":{"total_tokens":2864,"prompt_tokens":828,"completion_tokens":2036,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":1955}},"tokens_in":444,"tokens_out":2036,"duration_ms":13749,"temperature":1.0,"reasoning_tokens":1955,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:36:58.148849+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication using the same transformer backbone and data but with, say, fifty random seeds and per-architecture hyperparameter search that yields a significant MAE or RMSE decrease for the GCN-augmented models over the vanilla transformer would refute the claim that GCN modules lack evidence of contribution.","supporting_citations":[{"cited_title":"& Gardner, L","cited_arxiv_id":null,"evidence_quote":"Supplies the incidence, mortality, and hospitalization data for both the EU and US regions."},{"cited_title":"Multitask learning","cited_arxiv_id":null,"evidence_quote":"Provides the multitask-learning idea used to share representations across clinical variables."},{"cited_title":"& Chollet, F","cited_arxiv_id":null,"evidence_quote":"Supports the use of mobility and syndromic surveillance data as proxies in neural forecasting models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents associations between mobility changes and COVID-19 transmission, the basis for treating mobility as informative."},{"cited_title":"& Liu, Y","cited_arxiv_id":null,"evidence_quote":"Supplies the thresholded Gaussian kernel adjacency matrix construction used for the physical-distance graph."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the linear-layer baseline (DLinear) used in the sensitivity analysis for multivariate time series."},{"cited_title":"& Ning, Y","cited_arxiv_id":null,"evidence_quote":"Exemplifies the cross-location GCN forecasting approach whose contribution this paper tests."}],"review_version":1}