{"id":"29827d45-869b-482b-a991-c710dbecad4d","arxiv_id":"2411.13496","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A GNN that injects GPD-derived tail descriptors into features, adjacency, and loss reports higher heatwave recall and precision than a Li et al. baseline, but its key metric table is arithmetically inconsistent.","lead":"DI-GNN is a graph neural network that adds extreme-value statistics, such as Generalized Pareto Distribution parameters, to the features, graph edges, and loss function to forecast heatwaves. The authors report large gains in balanced accuracy and recall over a single baseline on weather station data from British Columbia, but the supporting numbers are internally inconsistent and no code or data is released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported headline metrics in Tables 1–2 are arithmetically impossible under the paper's own metric definitions, so the claimed DI-GNN improvement has no coherent empirical support.","rationale":"The paper's contribution is empirical: a new architecture plus EVT descriptors should improve heatwave forecasting. The load-bearing condition is that the evaluation tables are trustworthy. The reader's stated weakest assumption was target leakage from GPD features, and that is a legitimate risk, but the more decisive problem is that the reported metrics are internally inconsistent under the equations the paper itself states. The balanced-accuracy identity makes the impossibility independent of class balance, so it cannot be excused by the 12.45% imbalance. This is not a question of consensus or interpretation; it is an arithmetic contradiction. I would not change the reader's REJECT verdict, because the central evidence is unreliable. If the authors supply confusion matrices and the numbers are corrected, the leakage and baseline concerns would still need attention, but the immediate blocker is metric validity. Hence partial agreement with the reader: the rationale flags inconsistency, but the named weakest assumption is leakage rather than the arithmetic failure.","tokens_in":15769,"tokens_out":7972,"duration_ms":86816,"concrete_test":"Request the raw validation-set confusion matrices for both models and both configurations, and recompute every metric in Tables 1–2 from those counts using Eqs. 7–12. For Table 2 (Cout=3), a valid confusion matrix must satisfy TNR = 2×0.8666 − 0.8495 = 0.8837 and Accuracy = p·0.8495 + (1−p)·0.8837 ∈ [0.8495, 0.8837]; if the supplied counts give 0.4377, the table or the metric definitions are wrong. Also verify Table 1 row 1, where the reported BA and Recall force TNR > 1. This single check determines whether the central performance claim is based on real model outputs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the central claim that DI-GNN 'significantly outperforms' Li et al., the results in Tables 1 and 2 must be jointly realizable under the definitions in Section 3.2. They are not. For DI-GNN, Cout=3: BA=0.8666 and Recall=0.8495 imply TNR=2×0.8666−0.8495=0.8837 (Eq. 8). Since Accuracy is a convex combination of Recall and TNR, Accuracy = p·0.8495 + (1−p)·0.8837, so any valid accuracy must lie between 0.8495 and 0.8837; the table reports 0.4377. The same holds for Cout=5: TNR=0.8674, yet reported accuracy is 0.2990. The baseline table is also impossible: row 1 gives TNR = 2×0.6235−0.1286 = 1.1184 > 1. No confusion matrix exists that reproduces either table. At minimum, a metric was computed with a different formula, mislabeled, or generated incorrectly; until this is resolved, the headline performance comparison cannot be evaluated. The GPD-feature leakage risk in Sections 4.1 and 4.4 is a separate concern, but the internal inconsistency is more fundamental.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DI-GNN, a graph attention network for heatwave forecasting from 71 British Columbia weather stations, and claims that injecting Generalized Pareto Distribution (GPD) descriptors into the feature space, adjacency matrix, and loss function substantially improves prediction of rare heatwave days over the Li et al. baseline. The reported evaluation covers 3-day and 5-day horizons and lists balanced accuracy, recall, precision, F1, accuracy, AUC, and average precision. The central claim is that EVT-informed components yield large, stable gains over the baseline on imbalanced heatwave data.","tokens_in":16017,"tokens_out":6509,"duration_ms":68651,"significance":"The underlying idea is timely and potentially interesting: representing tail behavior of temperature distributions inside a GNN is a natural direction for rare-event forecasting. However, as submitted, the empirical evidence is not usable. Tables 1 and 2 violate the metric identities defined in Section 3.2, there is no independent test set or uncertainty quantification, and the GPD descriptors may encode the same threshold that defines the labels. If these issues were corrected with a sound experimental protocol, the approach could merit further study, but the present manuscript does not establish the claimed improvement. The paper is transparent about some limitations (Section 7), such as station-only predictions and the absence of atmospheric drivers, but the problems I identify are more fundamental and require redoing the evaluation.","major_comments":[{"comment":"The reported headline metrics are jointly inconsistent under the paper's own definitions. With positive-class proportion p = 0.1245 (Section 5.1), Eq. (7) implies Accuracy = p*Recall + (1-p)*TNR. For DI-GNN with Cout = 3, BA = 0.8666 and Recall = 0.8495 give TNR = 2*BA - Recall = 0.8837, so Accuracy must lie between 0.8495 and 0.8837; Table 2 reports 0.4377. For Cout = 5, TNR = 0.8674, which forces Accuracy ≈ 0.866, not 0.2990. In addition, Eq. (12) would give F1 ≈ 86.15% from the reported precision and recall, not the tabulated 84.36%. No confusion matrix can reproduce Table 2, so the claimed DI-GNN performance cannot be evaluated.","section":"Tables 1 and 2; Section 3.2"},{"comment":"The baseline table is also internally inconsistent. For Cout = 3, BA = 0.6235 and Recall = 0.1286 imply TNR = 1.1184 > 1, which is impossible. For Cout = 5, BA = 0.3628 and Recall = 0.1200 imply TNR = 0.6056, which yields Accuracy ≈ 0.545 under the stated p = 0.1245, not the reported 0.6901. These inconsistencies make the baseline comparison meaningless as reported.","section":"Table 1; Section 3.2"},{"comment":"The GPD descriptors are fitted to the same Tmax series whose 90th percentile defines the heatwave label, creating a serious risk of circularity. Section 4.4 computes exceedances above the 90th percentile of Tmax, while Section 3 labels heatwave days by Tmax exceeding the 90th percentile under the PKL criteria. If the descriptors are computed over the full record rather than strictly on training folds, the features encode information about the label-generating distribution. Even if the descriptors are computed only on training data, the reported gain over the baseline may reflect per-station climatological tail priors rather than improved temporal forecasting skill. I would want to see temporal-blocked cross-validation with descriptors recomputed per fold, plus a descriptor-only baseline (e.g., logistic regression on ξ_i, σ_i, μ_i, Var_i, q95) to separate prior information from predictive skill.","section":"Sections 4.1, 4.4, 5.2; Eqs. (16) and (21)"},{"comment":"There is no independent test set and no uncertainty quantification. Section 5.2 states that the first 13 years are used for training and the last two years for validation, with early stopping based on validation loss; the results in Tables 1 and 2 therefore appear to be validation-set metrics. The claim that DI-GNN 'significantly outperforms' the baseline is unsupported without a held-out test period, repeated runs with error bars, or a significance test. The instability visible in the baseline training curves (Figures 2 and 3) further suggests run-to-run variability that should be quantified rather than described qualitatively.","section":"Sections 5.2 and 5.4"}],"minor_comments":[{"comment":"Figure numbering and captions are inconsistent: Figure 5 is labeled as Precision-Recall curves for the Li et al. model, but Section 5.4 refers to Figures 5 and 6 as though they are DI-GNN training curves, and the AP values for DI-GNN appear only in Figure 9 with no in-text reference to that figure.","section":"Figures 4, 5, 8, 9"},{"comment":"The loss function introduces a weighting parameter β, but the value of β is never reported in Section 5.2, so the training objective is not fully specified.","section":"Section 4.3, Eq. (20)"},{"comment":"The description should state explicitly whether the GPD threshold and descriptors are recomputed within each training fold or once on all data; Section 5.2 says 'derived from the training data,' but Section 4.4 does not specify any temporal split.","section":"Section 4.4"},{"comment":"The statement 'Code and data is available upon request' is insufficient for reproducibility; particularly because the experimental tables contain impossible entries, the manuscript should provide public code and processed data.","section":"Code Availability"},{"comment":"There is a typo ('increasing frequent') and the sentence about the 2021 heatwave breaking records 'by over 45°C' is ambiguous; presumably it means temperatures exceeded 45°C, not that records were broken by 45°C.","section":"Introduction"}],"recommendation":"reject","confidential_remarks":"This manuscript is not ready for publication. The tables are internally inconsistent under the paper's own metric definitions, and the evaluation protocol is insufficient to support the headline comparison. I recommend rejection rather than revision because the required changes are not local: the experiments would need to be rerun with a coherent evaluation setup, a leakage-aware feature protocol, and an independent test set. If the authors can supply those, a fresh submission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper grafts GPD tail descriptors onto a graph attention network for heatwave forecasting at 3-5 day leads. The specific combination—putting EVT-derived features into the feature vector, adjacency matrix, and loss—is not something I've seen tested, and it is a reasonable thing to try. The authors also choose balanced accuracy, precision-recall, and AP, which is the right instinct for a rare-event problem.\n\nThe trouble is the headline numbers cannot be true. For DI-GNN at Cout=3, recall 84.95% and balanced accuracy 86.66% force TNR = 88.37%; with the stated 12.45% positive rate, accuracy has to be near 88%, not 43.77%. The F1 is also inconsistent: precision 87.38% and recall 84.95% give F1 ≈ 86.1%, not 84.36%. The same arithmetic fails for the Cout=5 row and for both rows of the baseline table. In Table 1, balanced accuracy 62.35% with recall 12.86% implies TNR over 100%, which is impossible. No confusion matrix reproduces any of these tables. So the central claim—that DI-GNN dramatically beats the Li et al. baseline—is currently unsupported by the evidence as printed.\n\nThere are secondary issues. The GPD descriptors are per-station constants fitted on the same Tmax series that defines the heatwave labels. That is not automatically leakage, because the descriptors don't vary day-to-day, but the paper does not rule out that the model is just learning station-specific base rates. The baseline also looks weak: recall around 12-13% is far below what the original Li et al. paper reported, which suggests the adaptation may not be fair. And 'code and data available upon request' is not reproducible; there are no error bars either.\n\nWho gets value from this? Researchers working on EVT-informed ML architectures might find the framework worth borrowing, but nobody should rely on the reported performance numbers. The paper needs corrected metrics, a leakage analysis, and a properly tuned baseline before it can support its claims.\n\nMy advice: don't desk-reject it—the idea is salvageable and the authors clearly know the literature—but send it back for major revision. The arithmetic inconsistency alone is disqualifying for publication as-is.","headline":"The GPD+GNN combination is plausible, but the reported metrics are arithmetically impossible under the paper's own definitions, so the central claim currently has no valid empirical support.","tokens_in":16669,"tokens_out":3871,"would_cite":false,"duration_ms":38654,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Heatwave recall jumps from 13% to 85% with extreme-value GNN","keywords":["heatwave forecasting","extreme value theory","generalized Pareto distribution","graph neural networks","rare event prediction","class imbalance","weather-scale lead times","British Columbia weather stations"],"falsifier":"Fit the GPD descriptors on the training years only, then append a single per-station constant equal to the historical heatwave-day frequency from those same training years to the Li et al. feature set, with no adjacency reweighting and no weighted loss. If this trivial climatology control reaches recall and AUC near DI-GNN's reported values, the GPD machinery is adding nothing beyond station priors; if the control remains near the baseline (recall ~13%, AUC ~0.68), the descriptors are carrying real distributional information.","tokens_in":15512,"feed_emoji":"🔥","tokens_out":16258,"duration_ms":135068,"temperature":0.7,"pith_summary":"The paper claims that feeding a graph neural network with descriptors of the extreme tail of daily maximum temperature — the shape and scale of a Generalized Pareto Distribution fitted above the 90th percentile — makes heatwave forecasting at 3- to 5-day lead times substantially more accurate. The authors embed these descriptors in three places: the node features, the edge weights of the attention graph, and the loss function. On 71 British Columbia stations (2009–2024), the resulting DI-GNN model raises balanced accuracy from roughly 62% to 87% and recall from under 13% to 85% relative to the adapted baseline GNN, with AUC rising from 0.68 to 0.92. The broader claim is that injecting the statistical theory of extremes into the learning architecture, rather than treating extremes as a class-imbalance problem, is a general recipe for rare-event forecasting.","feed_headline":"Heatwave recall jumps from 13% to 85% with extreme-value GNN","feed_subtitle":"Tail statistics from the GPD in features, edges, and loss turn a GNN into a reliable heatwave detector.","key_machinery":"The central machinery is the per-station Generalized Pareto Distribution fit to exceedances of the 90th percentile of daily maximum temperature, summarized by shape $\\xi$, scale $\\sigma$, mean, variance, and 95th percentile $q_{95}$. These five numbers are used three ways: appended to each node's feature vector (Eq. 16); to reweight the Pearson-correlation adjacency matrix as $A_{ij} = \\rho_{ij} w_i w_j$ with $w_i = 1 + |\\xi_i| + \\sigma_i / \\max_j \\sigma_j$ (Eq. 17); and to define a weighted F1 loss that up-weights stations with heavier tails (Eqs. 18–20). The GPD, defined by $H(y|\\xi,\\sigma) = 1 - (1 + \\xi y/\\sigma)^{-1/\\xi}$, models the distribution of threshold excesses, so these descriptors encode how extreme and how variable the local temperature tail is.","core_discovery":"The central discovery is that GPD-derived descriptors — the shape parameter $\\xi$, scale $\\sigma$, mean, variance, and 95th percentile of daily maximum temperature — when computed per station and threaded through the feature vector, the adjacency matrix (via weights $w_i = 1 + |\\xi_i| + \\frac{\\sigma_i}{\\max_j \\sigma_j}$), and a weighted F1 loss, transform the model's ability to catch rare heatwave days. On the BC dataset, DI-GNN achieves balanced accuracy 86.66%, recall 84.95%, precision 87.38%, and AUC 0.919 for the 3-day horizon (Table 2), compared with balanced accuracy 62.35%, recall 12.86%, precision 45.96%, and AUC 0.675 for the adapted Li et al. baseline (Table 1). The authors interpret this as evidence that EVT-informed descriptors give the GNN a principled way to prioritize tail events, replacing heuristic feature engineering with statistically grounded structure at every level of the model.","pith_inferences":["An ablation the paper does not run would isolate which injection point matters: adding only the five GPD features to the Li et al. baseline, without adjacency reweighting or the weighted loss, would show whether the gain is from the extra inputs or from the graph and loss changes.","Because the per-station GPD descriptors are constant and derive from the same Tmax record used to define heatwave labels, they may act as a station-level climatology prior rather than a time-varying forecast signal; a control feature equal to historical heatwave frequency would test this.","DI-GNN's accuracy falls to 43.8% while the baseline achieves 89.2%, meaning the 85% recall likely comes with many extra false alarms; a cost-weighted comparison of missed heatwaves versus false alerts would assess operational value."],"forward_implications":["If the reported gains hold, heatwave warning systems could use DI-GNN to issue reliable alerts at 3- and 5-day lead times with recall around 85%, catching most heatwave days rather than the 13% caught by the baseline.","The approach transfers beyond heatwaves: the authors argue that any rare-event forecasting problem with a well-defined exceedance threshold could benefit from embedding GPD descriptors into features, graph structure, and loss.","DI-GNN's accuracy is actually lower than the baseline's (43.8% vs 89.2% for the 3-day horizon) while its balanced accuracy is much higher, showing an explicit trade-off between raw accuracy and rare-event detection that forecasters must manage.","Because the GPD descriptors are computed per station, the framework adapts to local climate: stations with heavier tails are weighted more heavily in both the graph and the loss, which could make the model usable across diverse climates without retuning."],"supporting_citations":[{"why":"Supplies the spatiotemporal GNN baseline and the PKL-labeled heatwave prediction task that DI-GNN builds on and is compared against.","marker":"[28]"},{"why":"Provides the extreme value theory foundation for using GPD shape and scale parameters to characterize tail exceedances.","marker":"[29, 30]"},{"why":"Demonstrates GPD-based modeling of serially dependent heatwave extremes, motivating the use of GPD descriptors for heatwaves.","marker":"[33]"},{"why":"Supplies theoretical context for extreme value distributions in characterizing heatwave intensity and frequency.","marker":"[34]"},{"why":"Defines the Graph Attention Network mechanism whose attention coefficients DI-GNN reweights via the GPD-informed adjacency matrix.","marker":"[27]"},{"why":"Defines the Perkins-Kirkpatrick-Lewis heatwave criteria (Tmax above the 90th percentile for three or more days) used to label the data.","marker":"[7]"}],"fun_headline_variants":["GPD-tailored GNN lifts heatwave recall to 85%","Extreme-value graph nets forecast heatwaves with 87% precision","DI-GNN: GPD features in edges and loss boost heatwave detection","Tail stats in GNN edges triple heatwave recall","EVT-informed GNN catches heatwaves others miss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the per-station GPD descriptors — fitted on the same Tmax record that defines heatwave labels — are legitimate predictive features rather than a backdoor that leaks the label structure; if they only encode each station's heatwave climatology, the reported forecast improvement is not genuine skill.","fun_headline_variants_meta":{"raw":{"variants":["GPD-tailored GNN lifts heatwave recall to 85%","Extreme-value graph nets forecast heatwaves with 87% precision","DI-GNN: GPD features in edges and loss boost heatwave detection","Tail stats in GNN edges triple heatwave recall","EVT-informed GNN catches heatwaves others miss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1343,"prompt_tokens":1007,"completion_tokens":336,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":248}},"tokens_in":623,"tokens_out":336,"duration_ms":3848,"temperature":1.0,"reasoning_tokens":248,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:20:23.186542+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit the GPD descriptors on the training years only, then append a single per-station constant equal to the historical heatwave-day frequency from those same training years to the Li et al. feature set, with no adjacency reweighting and no weighted loss. If this trivial climatology control reaches recall and AUC near DI-GNN's reported values, the GPD machinery is adding nothing beyond station priors; if the control remains near the baseline (recall ~13%, AUC ~0.68), the descriptors are carrying real distributional information.","supporting_citations":[{"cited_title":"J., Shaby, B","cited_arxiv_id":null,"evidence_quote":"Demonstrates GPD-based modeling of serially dependent heatwave extremes, motivating the use of GPD descriptors for heatwaves."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies theoretical context for extreme value distributions in characterizing heatwave intensity and frequency."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Perkins-Kirkpatrick-Lewis heatwave criteria (Tmax above the 90th percentile for three or more days) used to label the data."}],"review_version":1}