{"id":"bfd67e7a-4842-41e3-ae97-5c3a9b5c11af","arxiv_id":"2505.10665","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"IceMamba, a Mamba-based deep learning model, reports the lowest average RMSE and a top anomaly correlation among 25 sea ice forecast models, but test-set tuning and benchmark comparability weaken the claim.","lead":"This paper introduces IceMamba, a deep learning model that uses state space and attention techniques to forecast Arctic sea ice coverage one to six months ahead. It claims top skill among 25 benchmark models, but test-set tuning and an external benchmark comparison weaken the evidence.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark comparison is vulnerable to verification-mask mismatch and test-set-driven input selection; a clean re-verification against the published Bushuk protocol is needed.","rationale":"The reader's weakest_assumption is exactly my load-bearing concern: the apples-to-apples comparison against published benchmark scores. I agree with that identification. My concrete_test adds two testable components: (1) verify the evaluation-mask/metric pipeline is identical to Bushuk et al. by recomputing IceMamba scores inside that pipeline, ideally with the published code; and (2) re-run the input-configuration selection on data ending before the benchmark period to remove test-set selection. I did not find a stronger internal inconsistency than the reader noted (e.g., the 'lowest RMSE 11.72% in 2019' in the Discussion conflicts with Table 8's attribution to September 2019 for the short-term model, but that is a presentation inconsistency, not the main load-bearing issue). The numerical ties (e.g., IceMamba-1-only-SIC vs SEAS5 in short-term RMSE, 15.2924% vs 13.8444%) are close but not the decisive issue; the decisive issue is that the comparison protocol is not demonstrated to be matching. The RESSB-vs-VSSB internal comparison is cleaner because it uses identical setups, and the permutation explainability is a genuine independent analysis. Machine-checked verification is absent, but the authors do provide code, so the concrete_test is feasible. Given the paper has real engineering content and honest limitations, the verdict should remain CONDITIONAL, not ACCEPT or REJECT, until the benchmark comparison is verified. I therefore agree with the reader's CONDITIONAL verdict and recommend no change.","tokens_in":23560,"tokens_out":1758,"duration_ms":15399,"concrete_test":"Reproduce the benchmark evaluation by downloading the released IceMamba code and re-running IceMamba-4 and IceMamba-1-only-SIC for 2001-2020, computing RMSE and ACC only over the exact mask used by Bushuk et al. [43] (or, better, running the metrics inside the published benchmark pipeline). Then, re-select the input configuration using only years through 2010 (or 2014) and report benchmark skill for that nested-selection model; if the resulting mean RMSE no longer beats SEAS5's 22.4916% or the mean ACC no longer exceeds 0.6604, the SOTA claim should be downgraded.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central SOTA claim depends on IceMamba-4's lowest mean RMSE (22.0086%) and IceMamba-1-only-SIC's highest mean ACC (0.6604) relative to the 25-model benchmark of [43]. The load-bearing weakness is that IceMamba's scores are computed by the authors while the other models' scores are taken as published, without demonstrating that the verification pipeline is identical. In particular, the paper states that RMSE and ACC focus on 'regions where the standard deviation of September SIC exceeds 10%', but it does not specify the exact SIC climatology period, the land/ocean mask, the handling of the pole hole, the September-mean processing, or the number of grid points used when computing the spatial standard deviation and the metric averages. Any mismatch in this mask or in the metric normalization directly changes the comparison with the published scores. Second, the input configuration for each IceMamba variant was selected by testing multiple data combinations and picking the best on the internal 2016-2022 test period, which overlaps the benchmark years 2016-2020; this selects models on a portion of the very years used to report benchmark skill, so the reported margin over other models is not a fully out-of-sample comparison. The paper's own limitation list (channel fusion, re-gridding bias, fixed Arctic domain) does not address these two verification issues. If the mask or the test-set selection changes the reported differences, the headline SOTA claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces IceMamba, a state-space-model-based encoder-decoder architecture for seasonal pan-Arctic sea ice concentration (SIC) forecasting at 25 km resolution. The core modeling contribution is the Residual Efficient State Space Block (RESSB), which augments a Vision State Space Block with efficient channel attention and a residual 1x1-convolution branch. The authors train three forecast-horizon variants (1, 4, and 6 months) on NSIDC SIC together with ERA5 and ORAS5 reanalysis fields, compare them against anomaly persistence and a 25-model benchmark from Bushuk et al., and claim state-of-the-art seasonal forecast skill: IceMamba-4 achieves the lowest mean RMSE (22.0086%), IceMamba-1-only-SIC achieves the highest mean ACC (0.6604), and IceMamba-1-only-SIC ranks second in mean IIEE. The paper also presents a permutation-based explainability analysis and attributes the model's sensitivity to stratospheric wind to shared anthropogenic trends rather than direct dynamical coupling.","tokens_in":23703,"tokens_out":9066,"duration_ms":84259,"significance":"If the headline claims are supported, this is a valuable contribution: it is, to the authors' knowledge, the first application of state space models to sea ice forecasting; the architecture has linear complexity relative to Transformers; code and data links are provided; the benchmark evaluation uses a temporally separated rolling-window recalibration scheme; and the explainability section includes a detrending control to distinguish trend-driven from mechanistic linkages. However, the state-of-the-art claim is not yet established because the comparison with the 25-model benchmark depends on verification-protocol equivalence that is not demonstrated, and because the input-variable configuration for each IceMamba variant was selected using the same test years that overlap the benchmark evaluation. These issues are load-bearing for the central SOTA claim, and the reported margins over the next-best models are small enough that either issue could change the ranking.","major_comments":[{"comment":"The headline skill claims depend on a comparison in which IceMamba's RMSE, ACC, and IIEE are computed by the authors while the corresponding scores for the 25 benchmark models are taken as published from [43]. The paper states that \"RMSE and ACC metrics focus on regions where the standard deviation of September SIC exceeds 10%\" but does not specify the climatology period used to compute that standard deviation, the land/ocean mask, the treatment of the pole hole, the exact September-mean processing, or the grid-point count used in the spatial averages. Any mismatch between this mask and the mask used in the published benchmark directly changes the reported scores and the ordering of models; the margins in Fig. 5(a) (e.g., IceMamba-4's 22.0086% vs SEAS5's 22.4916%) are small enough that such a mismatch could alter the conclusions. The authors should demonstrate protocol equivalence, ideally by running a single verification code over all models' September-mean forecasts, or at minimum by reproducing the exact Bushuk et al. mask, climatology, and averaging definitions and making the verification code available.","section":"Skill comparison with dynamical and statistical models (Figs. 4-5)"},{"comment":"The per-variant input configuration was selected using test data that overlap the benchmark evaluation. The text states that for each variant \"the data combination with the best forecast performance is selected as the final training set,\" and the Method section defines the internal test set as 2015-2022 (with a corresponding forecast test period of 2016-2022). The benchmark evaluation covers 2001-2020, so the years 2016-2020 appear both in the configuration-selection criterion and in the reported benchmark scores. This makes the reported SOTA margins partly in-sample: IceMamba-4 and IceMamba-1-only-SIC were chosen among candidate input combinations for their performance on a period that includes five of the benchmark years. The authors should redo the variable selection using only data before 2016 (e.g., the 2011-2014 validation period), or use nested cross-validation so that selection and evaluation never touch the same years, and should report whether the benchmark ranking is robust to this choice.","section":"Overall Performance of IceMamba (Table 1) and benchmark setup"}],"minor_comments":[{"comment":"The sentence \"IceMamba-4 achieved the lowest average RMSE ... in all tested models\" appears immediately after Fig. 5, which only shows models with complete 20-year forecasts; please clarify whether this claim refers to all 25 models or to the 20-year subset, and report the mean scores of the 5 incomplete models if they are included in the comparison.","section":"Skill comparison with dynamical and statistical models (Fig. 5)"},{"comment":"The value 11.72% is attributed to IceMamba-1-only-SIC in the Results (\"In September 2019, IceMamba-1-only-SIC recorded an RMSE of 11.72%, the best performance observed across all evaluated models\") but later in the Discussion is attributed to IceMamba-4 (\"IceMamba-4 achieving the lowest RMSE (11.72% in 2019)\"); please correct this inconsistency.","section":"Forecast performance in extreme September sea ice events"},{"comment":"The reported RMSE for IceMamba-1-only-SIC at September 1 initialization is given as 14.8792% in one paragraph and as 15.2924% in the Discussion; please reconcile these numbers.","section":"Skill comparison with dynamical and statistical models"},{"comment":"Supplementary Tables 2-4 are cited both as the input-combination experiments and as the benchmark evaluation results; please disambiguate the table numbering.","section":"Overall Performance of IceMamba"},{"comment":"There is a typo \"January January Y-3\" in the rolling-window description, and \"forecats\" appears in Section 2; both should be corrected.","section":"Method: training scheme"},{"comment":"Table 1 reports metrics over all non-land regions for the 2016-2022 test period, while Figs. 4-5 use the benchmark's >10% standard-deviation mask over 2001-2020; please state this distinction explicitly in the captions to avoid confusion.","section":"Table 1 and Fig. 4/5 captions"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is clearly written and the code/data availability statement is a strength. The main risk is that the benchmark comparison is not yet demonstrated to be apples-to-apples; because the margins over SEAS5 are small, a careful re-verification could either confirm or overturn the headline claim. I would encourage the editor to ask the authors to run the exact Bushuk et al. verification pipeline (or provide a single verification script applied to all models) and to require that input-variable selection be redone on data before 2016. The phrase \"first study to apply state space models to sea ice forecasting\" should also be checked against recent literature, since SSMs are spreading quickly in environmental machine learning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one-line take: IceMamba is a reasonable new architecture and the first SSM applied to seasonal SIC forecasting, but the headline 'beats 25 models' claim is not yet supported as written because the comparison against the Bushuk benchmark is under-specified and the input configurations were selected on part of the same years reported.\n\nWhat's actually new: the RESSB block (VSSB + ECA + 1x1 residual branch) is a sensible assembly, and the paper carefully trains on a rolling window (train up to Y-5, validate Y-4 to Y-1, forecast Y) for the benchmark, which is genuinely time-separated. The ORAS5 subsurface ocean variables (ohc, mld) improving longer-lead forecasts is plausible and internally consistent. The permutation-based explainability section with the detrended u10 control is a thoughtful addition. They publish code and data links, and they list real limitations (re-gridding bias, fixed Arctic domain, no Antarctic test). That is more than many ML climate papers do.\n\nSoft spots, in order of severity. First, the benchmark comparison: IceMamba scores are computed by the authors while the 25-model scores are taken from Bushuk et al. The paper never shows that the verification mask (September SIC std > 10%), land/ocean mask, pole-hole handling, climatology period, or September-mean processing match the published protocol. The '22.0086 vs 22.4916' margin over SEAS5 is about half a percent, so a mask mismatch could easily re-rank the result. That is load-bearing for the SOTA claim. Second, input configurations: for each variant, the paper says it picked the combination with best forecast performance on the internal test period (2016-2022), which overlaps benchmark years 2016-2020. So part of the reported benchmark skill is selected on the test years, not purely out-of-sample. Third, the abstract overstates: no single IceMamba model wins both RMSE and ACC; IceMamba-4 wins RMSE and IceMamba-1-only-SIC wins ACC, and the two are different models with different input complexity. Fourth, there is a numerical inconsistency in the text: IceMamba-1-only-SIC's September-1 RMSE is given as 14.8792% in one sentence and 15.2924% in the next. Minor, but it undermines trust in the tables. No uncertainty intervals on near-tie differences is also a gap, though not fatal.\n\nThe central forecast skill story likely holds for the internal comparison against Anomaly Persistence and the ERA5-only ablation, which use the same pipeline. The external SOTA claim needs a documented re-verification against the Bushuk protocol or a caveat that the comparison is approximate.\n\nWho this is for: sea ice forecasters and ML-for-climate researchers. A serious referee should see it; the architecture and the ORAS5 finding are worth engaging with, but the benchmark claim needs tightening.","headline":"Solid architecture paper whose external SOTA claim is not yet clean: benchmark protocol is under-specified and input selection overlaps test years.","tokens_in":24394,"tokens_out":4794,"would_cite":true,"duration_ms":40770,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IceMamba, a state space deep learning model, is claimed to deliver state-of-the-art seasonal forecasts of pan-Arctic sea ice concentration, beating all 25 models in the September benchmark on average RMSE and ACC.","keywords":["sea ice concentration","seasonal forecasting","state space model","deep learning","pan-Arctic","reanalysis","benchmark comparison","explainability"],"falsifier":"Re-run IceMamba-4 and IceMamba-1-only-SIC through the published benchmark's own evaluation code on the exact verification mask (September SIC standard deviation above 10 percent), climatology, and 2001-2020 forecast years; if the recomputed mean RMSE and ACC differ materially from $22.0086\\%$ and $0.6604$, or fall below the best dynamical baseline and damped persistence, the state-of-the-art claim fails.","tokens_in":23215,"feed_emoji":"🧊","tokens_out":6712,"duration_ms":58738,"temperature":0.7,"pith_summary":"IceMamba, a deep learning architecture built on a state space model with embedded attention, is put forward as a new method for seasonal forecasting of pan-Arctic sea ice concentration at 25 km resolution. The paper claims that, on the September forecast benchmark of 25 dynamical, statistical, and deep learning models over 2001-2020, IceMamba-4 achieves the lowest average RMSE ($22.0086\\%$), IceMamba-1-only-SIC achieves the highest average anomaly correlation ($0.6604$), and the same variant ranks second in integrated ice edge error. It further claims this is the first application of state space models to sea ice forecasting, that the proposed Residual Efficient State Space Block improves on the plain Vision State Space Block, and that adding subsurface ocean reanalysis variables (ocean heat content, mixed-layer depth) improves skill, especially for longer lead times. A sympathetic reader would care because data-driven seasonal sea ice forecasts at useful spatial detail, if they hold up, are much cheaper to run than dynamical systems and could support Arctic operations and climate adaptation decisions.","feed_headline":"State space AI tops 25 models at seasonal sea ice forecasting","feed_subtitle":"IceMamba posts the lowest average RMSE and best anomaly correlation on the September benchmark over 2001-2020.","key_machinery":"The load-bearing component is the Residual Efficient State Space Block (RESSB), a modification of the Vision State Space Block (VSSB). RESSB passes the feature map through an Efficient Channel Attention module, then two VSSB blocks, and adds a SiLU-activated $1\\times1$ convolution branch via residual addition. The VSSB itself uses a 2D Selective Scan (SS2D) that scans image patches in four directions through a cross-scan module and processes each sequence with the S6 selective state space model, giving a global receptive field at linear cost. The ECA channel attention selects the most informative input variables and lag months, and the residual branch helps fuse information across channels. This stack sits inside an encoder-decoder with patch merging and expanding plus skip connections, and the model is retrained under a rolling-window scheme so that forecast years stay out of the training set.","core_discovery":"The paper's central claim is that a state space sequence model, equipped with channel attention and residual connections, can produce the most skillful seasonal pan-Arctic sea ice concentration forecasts among 25 compared models. On the benchmark, IceMamba-4, which predicts the next four monthly mean SIC maps in a single step, has the lowest mean RMSE ($22.0086\\%$), while IceMamba-1-only-SIC, a recurrent variant that updates its input monthly and uses only SIC, has the highest mean ACC ($0.6604$) and the second-lowest mean IIEE ($0.9459\\times 10^{12}\\,\\text{m}^2$) across 2001-2020. The authors argue that the model learns the seasonal cycle implicitly: short lead times lean on the previous month's SIC, longer leads shift to the same calendar month from prior years, and sensitivity to upward surface solar radiation during summer reflects the ice-albedo feedback. They also report that apparent sensitivity to 10 hPa stratospheric wind largely disappears when the input is detrended, indicating a shared anthropogenic trend rather than a causal dynamical coupling. A separate examination of extreme September years (2012, 2016, 2019) supports the claim of robust performance under strong ice loss.","pith_inferences":["If the state space architecture is the reason for the skill, the same RESSB-style block should transfer to other gridded climate forecast problems, such as Antarctic sea ice, sea surface temperature, or vegetation indices, where long-range spatiotemporal dependence matters; the paper does not test this.","Because the model leans on the anthropogenic trend as a form of memory, its forecast skill in the 2030s may degrade if the trend's character changes, for example under a regime shift in Arctic sea ice; a direct test would be to retrain on data ending before 2000 and verify on 2001-2020.","The channel-fusion limitation named in the paper suggests a testable extension: feeding global atmospheric fields or separate per-variable encoders could remove the Arctic-grid bias it describes, and one could compare such variants against IceMamba on the same benchmark.","The permutation result that upward solar radiation matters more than downward radiation points to a concrete probe: an ablation that hides ussr while keeping dssr should hurt summer forecasts more than the reverse, which could be checked with the released model code."],"forward_implications":["If the benchmark comparison is valid, a purely data-driven 25 km SIC forecast can beat all 25 dynamical, statistical, and deep learning models on average September RMSE and ACC, not just match them.","Subsurface ocean variables (ocean heat content and mixed layer depth) become more valuable as the forecast horizon grows, with the largest gains in August-October, so future systems should treat them as standard predictors.","IceMamba-4 retains stable skill under June-initiated forecasts during extreme ice-loss years, suggesting single-step multi-month prediction is more robust to error accumulation than recursive updating.","The implicit seasonal alignment, using last year's same-month SIC at longer leads, works without explicit temporal encoding, indicating the lag ordering alone supplies enough seasonal structure.","The detrending result implies that the model's stratospheric-wind sensitivity is mostly a shared anthropogenic trend, so attribution statements from such models must separate trend from mechanism."],"supporting_citations":[{"why":"Defines the 25-model September sea ice forecast benchmark and its RMSE, ACC, and IIEE evaluation protocol that IceMamba is compared against.","marker":"[43]"},{"why":"Supplies the Mamba selective state space model and S6 block that IceMamba's VSSB and SS2D machinery is built on.","marker":"[41]"},{"why":"Provides the Vision State Space Block (VSSB) and Cross-Scan Module that give IceMamba its 2D global receptive field.","marker":"[89]"},{"why":"Provides the Efficient Channel Attention module used inside RESSB to select important channels and lag months.","marker":"[42]"},{"why":"Source of the parameter selection and climate preprocessing choices that IceMamba adapts for its input design.","marker":"[27]"},{"why":"Supplies the ERA5 atmospheric reanalysis variables used as forecast inputs.","marker":"[39]"},{"why":"Supplies ORAS5 ocean reanalysis variables (ocean heat content, mixed layer depth) that the paper credits with long-lead skill gains.","marker":"[40]"},{"why":"Provides the damped persistence reference forecast that IceMamba must beat in the benchmark.","marker":"[79]"}],"fun_headline_variants":["IceMamba state space net beats 25 rivals on Arctic sea ice","AI ice forecaster: state space tops 25 models in seasonal skill","IceMamba: lowest RMSE and best ACC in pan-Arctic sea ice test","State space model IceMamba wins Arctic seasonal forecast showdown","New AI IceMamba outforecasts 25 models for pan-Arctic sea ice"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that IceMamba beats all 25 models assumes its scores were computed under exactly the same evaluation rules as the published scores of the other models\\u2014the same set of grid cells, the same definition of September mean, the same comparison years\\u2014which the paper does not demonstrate.","fun_headline_variants_meta":{"raw":{"variants":["IceMamba state space net beats 25 rivals on Arctic sea ice","AI ice forecaster: state space tops 25 models in seasonal skill","IceMamba: lowest RMSE and best ACC in pan-Arctic sea ice test","State space model IceMamba wins Arctic seasonal forecast showdown","New AI IceMamba outforecasts 25 models for pan-Arctic sea ice"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000847,"raw_usage":{"total_tokens":3710,"prompt_tokens":991,"completion_tokens":2719,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":2620}},"tokens_in":607,"tokens_out":2719,"duration_ms":17646,"temperature":1.0,"reasoning_tokens":2620,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:06:28.584525+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run IceMamba-4 and IceMamba-1-only-SIC through the published benchmark's own evaluation code on the exact verification mask (September SIC standard deviation above 10 percent), climatology, and 2001-2020 forecast years; if the recomputed mean RMSE and ACC differ materially from $22.0086\\%$ and $0.6604$, or fall below the best dynamical baseline and damped persistence, the state-of-the-art claim fails.","supporting_citations":[{"cited_title":"Bulletin of the American Meteorological Society (2024)","cited_arxiv_id":null,"evidence_quote":"Defines the 25-model September sea ice forecast benchmark and its RMSE, ACC, and IIEE evaluation protocol that IceMamba is compared against."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp","cited_arxiv_id":null,"evidence_quote":"Provides the Efficient Channel Attention module used inside RESSB to select important channels and lag months."},{"cited_title":"Nature communications12(1), 5124 (2021)","cited_arxiv_id":null,"evidence_quote":"Source of the parameter selection and climate preprocessing choices that IceMamba adapts for its input design."},{"cited_title":"Quarterly Journal of the Royal Meteorological Society146(730), 1999–2049 (2020)","cited_arxiv_id":null,"evidence_quote":"Supplies the ERA5 atmospheric reanalysis variables used as forecast inputs."},{"cited_title":"Ocean science15(3), 779–808 (2019)","cited_arxiv_id":null,"evidence_quote":"Supplies ORAS5 ocean reanalysis variables (ocean heat content, mixed layer depth) that the paper credits with long-lead skill gains."}],"review_version":1}