{"id":"f889453f-5070-4158-be55-af86f5bbcb49","arxiv_id":"2508.07410","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Graph-based selection of 20 similar ensemble members from the 80-member MEF forecast improves ENSO prediction skill, especially at long lead times, compared with averaging all members.","lead":"The authors describe a graph-based method to select 20 of 80 ENSO forecast ensemble members and average them into a final forecast. They report improved forecast skill and stability compared with the baseline MEF model, especially at long lead times.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Selection uses test-period observations both to choose and to evaluate the 20-member subset, so the ~10% skill gain is likely circular; no GNN/community detection details are given.","rationale":"The reader's weakest_assumption correctly identifies the load-bearing flaw: the member-selection step uses the observed Niño 3.4 index for the same forecast period that is later used for evaluation. This is evident from the paper's own definitions in §2.2, where RMSE is defined as 'absolute error of prediction with respect to the ground truth,' and from §3.3, where the selected 20-member subset from 'initial 80' is evaluated on the post-2000 period. If the selection were only based on graph structure (inter-member similarity), it might still be legitimate, but the edge weights are explicitly functions of error against the true index, so the graph is just a vehicle for test-label peeking. The paper provides no GNN architecture, no training details, no community detection algorithm, and no code, so there is no way to separate a structural effect from pure post-hoc selection. The internal 40-vs-80 discrepancy further obscures what was actually done. A strict temporal split in member selection is the minimal check that would determine whether the claimed improvement is a true forecast skill gain or an artifact of leakage. Since this concern is decisive and the paper already fails on this ground, the reader's REJECT verdict stands unchanged.","tokens_in":15980,"tokens_out":3094,"duration_ms":33623,"concrete_test":"Re-run the entire selection with a strict temporal leave-one-out protocol: for each target month in 2000-2017, compute each member's RMSE/correlation using only forecasts initialized before that target date (or a rolling window up to the forecast start), build the graph, select 20 members, then evaluate on the target month. If the mean correlation gain over the MEF baseline drops to ~0 or negative, the published gain is due to label leakage. As a second check, compare against an oracle selector that picks the 20 lowest-RMSE members directly on the post-2000 period; if the method matches the oracle, leakage is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (GNN-based selection improves MEF skill by ~10%, especially at >17-month leads) rests on a circular evaluation. In §2.2 the graph edge weights are defined via 'RMSE: absolute error of prediction with respect to the ground truth' and Pearson correlation against observed Niño 3.4. In §3.3 the same post-2000 period is used to select the 20 members and to evaluate the averaged forecast (Fig. S4). Thus the selected subset is chosen as the best 20 on the test labels; any method that picks low-error members on the test set will appear skillful. The GNN/community-detection step is not actually specified (no architecture, training, or algorithm name), so the only operative selection criterion is test-period error. There is also an inconsistency: §2.2 builds the graph from 40 3DCNN runs, while §3.3 says 20 of 'initial 80,' making the pipeline ambiguous. Without a temporal split in member selection, the reported improvement is not a forecast result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a graph-based ensemble post-processing method for the Multimodal ENSO Forecast (MEF) model. The method constructs an undirected weighted graph whose nodes are the 40 (or later, 80) ensemble members and whose edge weights are similarity measures (RMSE and correlation) computed with respect to the observed Niño 3.4 index. A GNN/community-detection step is then used to select 20 members, whose average is the final forecast. The paper claims this selection improves MEF skill by about 10%, especially at lead times beyond 17 months, and that the approach is model-agnostic. The evaluation is performed on the post-2000 period.","tokens_in":16297,"tokens_out":2355,"duration_ms":27239,"significance":"If the reported skill gains were obtained by a genuinely predictive selection rule, the method could be a useful, model-agnostic ensemble post-processing tool for long-lead ENSO forecasting. The idea of selecting a coherent subset of ensemble members based on graph structure is interesting and has potential. However, the manuscript as written does not establish the central claim: the selection rule uses test-period observations to choose the subset, and the same observations are then used to evaluate it. In addition, the GNN/community-detection methodology is not actually specified, and the reported 40/80 member inconsistency makes the pipeline ambiguous. No code, architecture details, or reproducible experimental protocol are provided. These issues are load-bearing, so the paper cannot be accepted in its current form.","major_comments":[{"comment":"The selection rule is circular with respect to the evaluation. Section 2.2 defines edge weights using 'RMSE: absolute error of prediction with respect to the ground truth' and Pearson correlation against observed Niño 3.4. Section 3.3 evaluates the 20 selected ensemble members on the same post-2000 period used to compute these errors (Fig. S4). Thus the selected subset is chosen as the best-performing 20 on the test labels, and the reported ~10% skill gain is largely a direct consequence of test-set fitting rather than an honest forecast improvement. A proper forecast evaluation requires a temporal split in which member selection uses only data before the evaluation period, or a fully cross-validated selection protocol.","section":"§2.2, §3.3"},{"comment":"The manuscript does not actually specify the GNN or community-detection method despite naming it in the title and abstract. Section 2.2 states 'one may use a GNN' and lists optional node features, but no architecture, training objective, number of layers, or community detection algorithm is given. Section 3.3 refers to 'graph-based similarity clustering' without identifying the algorithm. The only concrete selection criterion that appears in the text is the test-period RMSE/correlation edge weight. Thus the central mechanistic claim—that GNN-based structural analysis improves ensemble selection—is unsupported. The paper needs to provide a reproducible description of the method: the actual graph construction, the GNN training procedure, the clustering/community detection algorithm, and the exact rule for choosing 20 members.","section":"§2.2, §3.3"},{"comment":"There is a direct numerical inconsistency in the ensemble size. Section 2.2 and Section 2.3 state the 3DCNN produces 40 runs, and Section 2.3 says the MEF model comprises '40 stochastic runs.' The Abstract and Section 3.3, however, state the graph is built from 'all 80 members of the ensemble' and that 20 are selected 'from within the initial 80.' The pipeline is therefore ambiguous: does the graph contain 40 or 80 nodes? This is not a typo-level issue because the subset-selection statistics depend on the pool size. The authors must reconcile these numbers and specify exactly how the 80 members arise (e.g., 40 3DCNN plus 40 time-series runs).","section":"§2.2, §3.3, Abstract"},{"comment":"The central quantitative claim—'consistently improves performance by approximately 10% over MEF, particularly in extended lead times'—is presented only in a figure caption with no supporting numeric table, confidence intervals, or significance tests. Moreover, Fig. 5 is captioned as comparing 'MEF and GNN-based ensemble selection method,' while the surrounding text and Fig. S4 compare MEF to the CNN baseline. It is unclear whether the improvement is over the original MEF weighted average or over the CNN baseline. The authors need to provide a clear comparison table with skill scores (correlation, RMSE) for MEF, the GNN-selected method, and the CNN baseline, with uncertainties, across lead months.","section":"Fig. 5, §3.2"}],"minor_comments":[{"comment":"The manuscript contains several duplicated or misordered section numbers (two '2.2' sections, '2.3' and '2.4' appear out of order relative to the description of the integrated model). The references are not formatted consistently, and several citations are not relevant to the claims (e.g., refs. 38, 53 in the context of climate forecasting). Please revise thoroughly.","section":"Global"},{"comment":"Equation (1) has formatting issues (subscripts and superscripts are not clearly delineated) and the dimensions of the input tensor are not explicitly defined. The description of the kernel sizes and pooling operations is hard to follow; a clear architecture table would improve reproducibility.","section":"§2.1"},{"comment":"The statement 'MEF captures nearly 50% of all El Niño and La Niña events even at 23-month horizons' is not accompanied by a definition of 'captures' (e.g., phase hit rate, amplitude threshold) or a comparison with the baseline. Please define the metric and provide quantitative evidence.","section":"§3.3"},{"comment":"The abstract claims the method 'produces more stable and consistent outputs,' but no measure of stability or consistency is defined or reported. The conclusion similarly claims superiority 'across almost all seasons' without supporting seasonal skill scores.","section":"Abstract and §4"}],"recommendation":"reject","confidential_remarks":"The manuscript has a fundamental circularity problem in its evaluation and lacks the algorithmic details necessary to reproduce the proposed GNN-based selection. The inconsistency between 40 and 80 ensemble members further undermines confidence. The topic is within the journal's scope, but the current evidence does not support the central claim; a rejection with encouragement to resubmit after a properly temporally separated evaluation and a full specification of the GNN method would be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea—use a similarity graph over ensemble members to pick a coherent, accurate subset instead of MEF's global weighting—is reasonable, and there is a real problem worth working on. The paper honestly notes the method does not always beat MEF and is more stable rather than uniformly more skillful. That is the extent of the good news.\n\nThe central claim is not supported. The graph edges are built using RMSE and correlation with the observed Niño 3.4 index over the post-2000 validation period (Section 2.2), and the same period is then used to select the 20 members and to evaluate the skill of their average (Section 3.3, Fig. S4). Any selection rule that picks the lowest-error members on the test labels will look skillful on those labels. The ~10% improvement is therefore a re-statement of the selection criterion, not a forecast result. The paper does not describe any temporal split in member selection.\n\nThere is also no actual GNN. The text says a GNN is used to learn embeddings and detect clusters, but it never specifies an architecture, a training objective, or a community detection algorithm. The phrase 'GNN-based' appears throughout, but the only operative mechanism described is computing pairwise RMSE/correlation and taking some subset. This is not reproducible and the title overstates the method.\n\nThe manuscript also shows internal inconsistencies: Section 2.2 says 40 3DCNN runs, while Section 3.3 says 20 selected from an initial 80. No code, data, or error bars are provided. The literature citations are sometimes irrelevant (BERT, GANs) and look like padding.\n\nWho is this for? Readers interested in ensemble post-processing for climate forecasts might find the motivation useful as a cautionary example: selection must be trained on past data and evaluated on held-out periods. But the current results are not believable.\n\nMy recommendation: do not send this to peer review in its present form. The circular evaluation is a load-bearing flaw, and the missing methodological details are too large to be fixed by minor revision. A desk reject is appropriate, though the authors could revisit the idea with a proper temporal split and a real GNN description.","headline":"The reported ~10% skill gain is an artifact of selecting the 20-member subset using the same test-period observations used for evaluation, and the paper never actually specifies or implements a GNN.","tokens_in":16718,"tokens_out":2195,"would_cite":false,"duration_ms":26557,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Graph-based subset of 20 ensemble members improves ENSO forecasts by about 10% over the standard blend.","keywords":["ENSO forecasting","graph neural networks","ensemble selection","similarity graph","community detection","multimodal deep learning","long-lead prediction","Niño 3.4 index"],"falsifier":"On the same data, rebuild the similarity graph using only information available before the target season (e.g., errors on a training-period sample) and re-select 20 members, then score on the post-2000 period. If the correlation gain over MEF falls to zero or reverses, the paper's improvement is selection leakage rather than a forecasting advance. A second check is to replace the GNN with simple k-medoids on the same similarity matrix and compare.","tokens_in":15902,"feed_emoji":"🌊","tokens_out":9012,"duration_ms":84115,"temperature":0.7,"pith_summary":"The paper argues that the strongest ENSO forecasts hide inside a large ensemble, and that a graph can find them. It builds an undirected graph whose nodes are the 80 members of the MEF ensemble and whose edge weights measure how similar two forecast outputs are; a GNN and community detection then pick 20 structurally coherent members, and the final forecast is their average. The claim is that this selection raises correlation skill by roughly 10% over the baseline MEF weighting, especially at lead times beyond one year, and keeps correlation above 0.4 through month 23. If true, this would make long-lead ENSO prediction more usable and would transfer to any ensemble-based forecasting system without retraining the base models.","feed_headline":"20 chosen members beat the full ENSO forecast blend","feed_subtitle":"Graph-based selection from 80 runs lifts correlation skill by about 10%, with skill above 0.4 through 23-month leads.","key_machinery":"The similarity graph G(V,E): each of the 80 ensemble outputs is a vertex, and each edge weight encodes pairwise similarity between two runs' forecast outputs, computed from metrics such as RMSE and Pearson correlation. A graph neural network embeds the nodes by aggregating neighbor information, and community detection identifies a dense cluster of coherent, accurate runs from which 20 members are selected; the final prediction is the simple average of those 20. This lets ensemble selection act on structural agreement rather than on individual scalar error alone.","core_discovery":"The authors claim that averaging a graph-selected subset of 20 out of 80 MEF ensemble members gives more skillful and more stable multi-year ENSO forecasts than the original MEF weighting. The selection is made by representing all runs as nodes of a weighted undirected graph, with edge weights derived from RMSE and correlation between run outputs, then applying a GNN and community detection to locate a dense cluster of consistent, high-performing runs. On the post-2000 test period, the graph-selected forecast consistently improves correlation skill by about 10% relative to MEF, is superior to the CNN baseline at all lead times and especially beyond 17 months, and maintains correlation skill","pith_inferences":["If the graph-selection gain generalizes, the practical benefit may come less from the GNN itself than from shifting from averaging everything to averaging a consensus cluster; a simpler cluster-medoid baseline might capture much of the benefit at lower cost, and the paper does not test this.","The paper's similarity metrics are computed against the observed Niño 3.4 index over the same post-2000 window used for scoring, so an out-of-sample re-run with training-period similarity weights would be the decisive check of whether the ~10% is real forecast skill.","The reported statistical fingerprint suggests that spectral smoothness and autocorrelation decay rate—cheaply computed from an output time series—could serve as a surrogate for graph centrality, letting other groups apply the idea without a GNN.","The same recipe could be applied to large physical ensembles such as S2S reforecast archives, where a 20-member consensus subset might also reduce the noise from poorly initialized members."],"forward_implications":["Selecting and averaging 20 graph-coherent members yields roughly 10% higher correlation skill than the full MEF weighting, with the largest gains at lead times beyond one year.","The method keeps the forecast correlation skill above 0.4 for all lead months tested (up to 23 months) and captures roughly half of El Niño and La Niña events at the longest leads.","Because the selection step operates only on the outputs, it can be attached to any ensemble forecast system—statistical, dynamical, or hybrid—without changing the base model.","Selected members tend to be densely interconnected, low-variance, spectrally smooth runs; this fingerprint could be used as a cheap screening criterion.","Forecasts become more stable and consistent, particularly under compound long-lead conditions, even in scenarios where the accuracy gain over the baseline is small."],"supporting_citations":[{"why":"Provides the original deep-learning CNN ENSO forecast model and the CMIP5/GODAS data setup that MEF extends.","marker":"[27]"},{"why":"Provides the improved CNN variant (ACNN) and the validation data used in building the MEF ensemble.","marker":"[29]"},{"why":"Cited as the weighted-fusion MEF baseline whose 80-member ensemble this work post-processes.","marker":"[86]"},{"why":"Supplies the graph neural network formalism for representing ensemble outputs as nodes and edges.","marker":"[25]"},{"why":"Cited as the basis for the RMSE/correlation and output-similarity metrics that define graph edge weights.","marker":"[65]"},{"why":"Supports the premise that large ensemble systems carry predictive skill that can be harvested by selection.","marker":"[26]"}],"fun_headline_variants":["Graph prunes 80 ENSO runs to 20 for better skill","GNN picks 20 best runs from 80 for ENSO forecast","Selected 20 runs beat full ENSO ensemble in skill","Graph-based subset trims ENSO forecasts, gains skill","20 member subset from graph improves ENSO skill"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The similarity graph for selecting the 20 members is built using RMSE and correlation of each forecast against the observed Niño 3.4 index for the same post-2000 period used in the evaluation, so the reported skill gain is only a true forecast improvement if this choice does not amount to picking members by looking at the answer key.","fun_headline_variants_meta":{"raw":{"variants":["Graph prunes 80 ENSO runs to 20 for better skill","GNN picks 20 best runs from 80 for ENSO forecast","Selected 20 runs beat full ENSO ensemble in skill","Graph-based subset trims ENSO forecasts, gains skill","20 member subset from graph improves ENSO skill"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000548,"raw_usage":{"total_tokens":2500,"prompt_tokens":837,"completion_tokens":1663,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":1578}},"tokens_in":581,"tokens_out":1663,"duration_ms":10412,"temperature":1.0,"reasoning_tokens":1578,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:07:36.005224+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same data, rebuild the similarity graph using only information available before the target season (e.g., errors on a training-period sample) and re-select 20 members, then score on the post-2000 period. If the correlation gain over MEF falls to zero or reverses, the paper's improvement is selection leakage rather than a forecasting advance. A second check is to replace the GNN with simple k-medoids on the same similarity matrix and compare.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited as the weighted-fusion MEF baseline whose 80-member ensemble this work post-processes."},{"cited_title":"Long Lead ENSO Forecast Using an Adaptive Graph Convolutional Recurrent Neural Network","cited_arxiv_id":null,"evidence_quote":"Supplies the graph neural network formalism for representing ensemble outputs as nodes and edges."}],"review_version":1}