{"id":"f6ab0b45-0e02-4e99-9baf-03c58ac6a13b","arxiv_id":"2411.19031","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A benchmark of CNN, ConvLSTM, Transformer and physics-informed Transformer for Arabian Sea SST prediction, whose headline claim that the physics-informed model wins at 30 days is contradicted by its own Table 2.","lead":"This paper compares four machine learning models for predicting sea surface temperature in the Arabian Sea at 7, 15, and 30 day lead times. It claims the physics-informed transformer works best at 30 days, but its own summary table shows the plain transformer performs better on most metrics.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central 30-day claim is contradicted by its own Table 2: the plain Transformer beats the PINN-Transformer on all four threshold metrics, so 'PINN-Transformer excels' is internally unsupported.","rationale":"The reader's strongest claim, that the abstract is contradicted by Table 2 at 30 days, is exactly the load-bearing problem I identify. I did not adopt the reader's weakest assumption as the primary attack because it concerns the mechanism (whether the physics loss is correctly specified) rather than the direct empirical comparison. If the 30-day numbers in Table 2 are reliable, the paper's headline fails regardless of how the physics loss is formulated; the Table 2 contradiction is therefore the most decisive test. I partly agree with the reader's weakest assumption as an additional serious flaw: Section 3 never states the PDE, the form of the physics loss, or the weighting coefficient, so the attribution to 'physical principles' is also ungrounded. Together these justify the reader's REJECT verdict; my read does not change it.","tokens_in":15963,"tokens_out":6027,"duration_ms":50442,"concrete_test":"Regenerate the 30-day row of Table 2 from the raw test-set predictions, or from the grid-level maps in Figure 8 if raw outputs are unavailable, using the paper's standard metric definitions (anomaly correlation, Nash-Sutcliffe efficiency, normalized RMSE, MAE) and the same threshold counts. Then apply a paired bootstrap over the 504 grids to the 30-day differences between Transformer and PINN-Transformer on each metric. If the regenerated table reproduces the published percentages, the abstract and conclusions overstate the PINN-Transformer's 30-day skill; if the regenerated table reverses them, the published table likely contains a typo and the central claim could be repaired, which would change this verdict to CONDITIONAL.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the PINN-Transformer excels at 30-day SST prediction and that this demonstrates the value of physical constraints. The decisive evidence is the paper's own Table 2, because the abstract's claim is specifically about comparative skill. In the 30-day row, the plain Transformer outperforms the PINN-Transformer on every reported threshold: ACC>0.8 (58.13% vs 47.02% of grids), NSE>0.7 (35.91% vs 19.25%), NRMSE<0.1 (20.04% vs 0.20%), and MAE<0.5 (65.48% vs 55.16%). The metric ranges in the same row also show the Transformer is not worse on any range (ACC 0.72-0.88 vs 0.68-0.87; NSE 0.43-0.77 vs 0.41-0.74; MAE 0.37-0.68 vs 0.38-0.70). This is not a matter of external consensus; it is an internal inconsistency. Section 4.3's text that the PINN-Transformer 'stands out with the highest ACC and NSE values... lowest NRMSE and MAE' directly contradicts Table 2, as does the Figure 9 paragraph giving PINN-Transformer NSE 0.46-0.58 against Transformer 0.50-0.59. The physics-informed loss is indeed unspecified in Section 3 (only 'terms related to the divergence and Laplacian of the velocity components'), but that is a secondary weakness: even a perfectly specified physics loss would not make Table 2 support a claim it contradicts.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks four deep-learning models—CNN, ConvLSTM (referred to as LSTM), Transformer, and a physics-informed Transformer (PINN-Transformer)—for predicting sea surface temperature (SST) in the Arabian Sea at 7-, 15-, and 30-day lead times. Inputs are ERA5 reanalysis fields (six atmospheric predictors plus SST) at 0.5° resolution over 2000–2023, with 2000–2016 used for training and 2017–2023 for testing. Performance is assessed with anomaly correlation coefficient (ACC), Nash–Sutcliffe efficiency (NSE), normalized root mean square error (NRMSE), and mean absolute error (MAE). The abstract and conclusions claim that the PINN-Transformer excels at 30-day predictions, indicating that physical constraints improve long-term SST forecasting.","tokens_in":16376,"tokens_out":4243,"duration_ms":36603,"significance":"A well-executed comparison of modern ML architectures for SST forecasting would be of practical value, and the paper has a reasonable evaluation design: a held-out test period, four established metrics, and grid-level threshold percentages that are reported in full in Table 2. If the central claim were supported, it would suggest that physics-informed losses stabilize forecasts at longer lead times, a result of interest to the ocean forecasting community. However, the paper's headline claim is contradicted by its own Table 2, and the physics-informed component is too underspecified to support any attribution. The paper therefore does not deliver on its stated contribution, though the underlying benchmark data could be useful if the narrative were corrected and the methodology expanded.","major_comments":[{"comment":"The central claim that 'the PINN-Transformer excels in 30-day predictions' is directly contradicted by Table 2. At the 30-day lead time, the plain Transformer beats the PINN-Transformer on every reported threshold: ACC>0.8 on 58.13% vs 47.02% of grids, NSE>0.7 on 35.91% vs 19.25%, NRMSE<0.1 on 20.04% vs 0.20%, and MAE<0.5 on 65.48% vs 55.16%. The metric ranges in the same row also show no compensating advantage (e.g., ACC 0.72–0.88 vs 0.68–0.87). Section 4.3's statement that the PINN-Transformer 'stands out with the highest ACC and NSE values' and Figure 9's claim that the PINN-Transformer shows the 'highest NSE values' (0.46–0.58, versus 0.50–0.59 for the Transformer) are inconsistent with the numerical results. The conclusion in §5 that physical integration yields a 'distinct advantage' at longer lead times is unsupported by the paper's own data.","section":"Abstract; §4.3; Table 2"},{"comment":"The physics-informed loss is never specified. The text states only that the PINN-Transformer's loss 'includes terms related to the divergence and Laplacian of the velocity components (u and v)', without giving the governing equation, the exact form of the additional loss terms, the weighting coefficient that balances data loss against physics loss, or the discretization used. No ablation is provided to isolate the effect of this term. Consequently, the attribution of any observed behavior to 'physical principles' cannot be tested or reproduced, and the reader cannot evaluate whether the constraint correctly represents ocean dynamics or is merely an arbitrary regularizer.","section":"§3 (Methodology)"},{"comment":"The Conclusions state that the analysis 'was restricted to 504 grids at a 1-degree spatial resolution,' but §2.2 says the data were bilinearly interpolated to 0.5° × 0.5°, and §4.1 states that this resolution 'results in 504 grid points.' Over the stated domain (7.25°–18.75°N, 62°–72°E), 504 grid points are consistent with 0.5° spacing, not 1°. This internal inconsistency must be corrected, as the spatial resolution directly affects the interpretation of all results and the stated limitation.","section":"§5 (Conclusions) vs §2.2 and §4.1"},{"comment":"The study provides no reference baseline (persistence, climatology, or an operational numerical forecast) against which the reported skill can be judged. Given the paper's title and introduction frame the work as assessing 'state-of-the-art' ML and PIML, absolute skill levels such as 'ACC>0.8 on 58% of grids at 30 days' need context. The four-model comparison can stand on its own, but the broader claim about the 'potential' of these methods requires a baseline, especially since all models show large declines at 30 days.","section":"§4.4; Table 2"}],"minor_comments":[{"comment":"The text in §4.3 says the PINN-Transformer has 'the highest NSE values, ranging from 0.46 to 0.58,' immediately after reporting the Transformer's NSE range of 0.50–0.59; the numerical ranges contradict the qualitative claim and should be fixed.","section":"§4.3; Figure 9"},{"comment":"The paper switches between 'ConvLSTM' and 'LSTM' without consistency: §3 introduces ConvLSTM, §4.1 says 'the ConvLSTM model will be referred to as LSTM from now on,' but the abstract and §1 use both names. This is confusing and should be standardized.","section":"§4.1 and §1"},{"comment":"The caption contains a typo ('Tabel 2'); figure captions also inconsistently refer to 'CC' rather than 'ACC' (e.g., Figures 4, 6, 8).","section":"Table 2 caption"},{"comment":"The conclusion that 'CNN and Transformer models exhibited robust performance in the 30-day forecast' is overstated: Table 2 shows CNN achieving NSE>0.7 on only 11.90% of grids and NRMSE<0.1 on 0% of grids at 30 days, and the Transformer also shows substantial degradation.","section":"§5 and Table 2"},{"comment":"The statement that source code is 'available from the corresponding author upon reasonable request' falls short of modern reproducibility standards; given that the final text also lacks hyperparameters and training details, making code and data publicly available would be necessary for verification.","section":"Data availability"}],"recommendation":"reject","confidential_remarks":"The stress-test concern lands: Table 2 contradicts the abstract and §4.3, so the central claim fails internally, not merely against external consensus. The underspecified physics loss compounds the problem, making the paper's key attribution untestable. The internal inconsistencies in the limitations section and the absence of baselines further weaken the submission. In principle, the benchmark data—especially the detailed grid-level threshold percentages—could support a revised paper that reports the plain Transformer as the best model at 15- and 30-day leads and treats the PINN-Transformer as a negative result, but that would be a substantial reframing of the manuscript's contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know before anything else: the abstract and Section 4.3 claim the PINN-Transformer excels at 30-day prediction, but Table 2 of the same paper shows the plain Transformer beating it on all four threshold metrics in that row. ACC > 0.8 on 58.13% of grids versus 47.02%, NSE > 0.7 on 35.91% versus 19.25%, NRMSE < 0.1 on 20.04% versus 0.20%, MAE < 0.5 on 65.48% versus 55.16%. The metric ranges also favor the Transformer. This is not an external disagreement; it is an internal inconsistency. The sentence in Section 4.3 about the PINN-Transformer standing out with the highest ACC and NSE and lowest NRMSE and MAE is simply false relative to the paper's own table.\n\nWhat the paper does well is modest but real. It sets up a clean regional benchmark for Arabian Sea SST at 7-, 15-, and 30-day leads using four architectures (CNN, ConvLSTM, Transformer, PINN-Transformer), six ERA5 predictors, and four standard skill metrics with grid-wise threshold percentages. That kind of comparison is useful for operational model selection, and the authors cite relevant prior work on physics-informed SST prediction (Yuan et al., 2023).\n\nThe soft spots are proportional to the central flaw. The physics-informed loss is unspecified: Section 3 only mentions \"terms related to the divergence and Laplacian of the velocity components,\" with no equation, no weighting coefficient, no ablation. So even if the claim were true, attributing the improvement to physical principles would be ungrounded. There is also a spatial-resolution inconsistency: the methods say bilinear interpolation to 0.5°, but the conclusion says 1-degree resolution, and 504 grids at 0.5° does not match the declared domain. No code or data are released, and there are no baselines like persistence or climatology, so the absolute skill numbers are hard to interpret.\n\nIn short, the main conclusion is contradicted by the paper's own evidence, and the modeling details needed to evaluate the PINN component are absent. The benchmark comparison in Table 2 may be salvageable if the authors fix the narrative, specify the loss, and clarify the resolution. As written, I would not send this to peer review; it needs major revision first.","headline":"The paper's own Table 2 contradicts its central claim that the PINN-Transformer excels at 30-day SST prediction; the regional benchmark may have value after major fixes, but the current conclusions are unsupported.","tokens_in":16863,"tokens_out":2086,"would_cite":false,"duration_ms":35672,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Transformer with an added physics-informed loss is claimed to give the best 30-day sea surface temperature forecasts in this benchmark.","keywords":["machine learning","sea surface temperature","physics-informed neural network","Transformer","ConvLSTM","Arabian Sea","ERA5","forecast lead time"],"falsifier":"Recompute the 30-day testing metrics on the same 504-grid test set and compare the plain Transformer with the PINN-Transformer; the paper's own Table 2 already shows the plain Transformer ahead on all four thresholds ($ACC>0.8$: 58.13% vs 47.02%; $NSE>0.7$: 35.91% vs 19.25%; $NRMSE<0.1$: 20.04% vs 0.20%; $MAE<0.5$: 65.48% vs 55.16%), so any analysis that reproduces these numbers without the PINN-Transformer winning would falsify the headline claim.","tokens_in":15810,"feed_emoji":"🌊","tokens_out":10970,"duration_ms":112339,"temperature":0.7,"pith_summary":"The paper benchmarks four deep-learning models—CNN, ConvLSTM, Transformer, and a Transformer with an added physics-informed loss—for forecasting sea surface temperature in the Arabian Sea at 7-, 15-, and 30-day leads. Its central claim is that the PINN-Transformer, which adds divergence and Laplacian terms of the wind components to the loss, becomes the best model at 30 days, while the pure CNN and ConvLSTM are best at 7 days and the Transformer at 15 days. The authors interpret this as evidence that physical constraints help generalization at longer lead times. The paper's own Table 2, however, lists the plain Transformer ahead of the PINN-Transformer on all four metrics at 30 days, a discrepancy the text does not resolve.","feed_headline":"Physics-informed Transformer beats pure ML at 30-day SST","feed_subtitle":"If right, hybrid deep learning would extend reliable ocean forecasts to monthly lead times.","key_machinery":"The load-bearing mechanism is the physics-informed loss term added to the Transformer, described only as 'terms related to the divergence and Laplacian of the velocity components ($u$ and $v$).' The paper invokes this term as the reason the PINN-Transformer generalizes better at 30 days, because it constrains predictions to respect ocean-heat dynamics. The comparison design—four architectures across three lead times with four skill metrics (ACC, NSE, NRMSE, MAE) computed grid-by-grid at 504 grid points—is what the claim rests on.","core_discovery":"On the paper's own terms, the discovery is that incorporating physics-informed loss terms into a Transformer architecture—specifically terms tied to the divergence and Laplacian of the $u$ and $v$ wind components—lets the model sustain predictive skill at a 30-day lead time, where purely data-driven models degrade sharply. The authors argue that the hybrid model's ability to remain physically consistent is what makes it more robust and less prone to overfitting as the forecast horizon extends, and that this advantage only appears at longer lead times, making physics integration essential for medium-range SST prediction. In the same narrative, short-term forecasts (7–15 days) are adequately served by pure ML models, so the contribution is framed as a lead-time-dependent trade-off. The paper's own results table, however, shows the plain Transformer outperforming the PINN-Transformer at 30 days on $ACC>0.8$ (58.13% vs 47.02%), $NSE>0.7$ (35.91% vs 19.25%), $NRMSE<0.1$ (20.04% vs 0.20%), and $MAE<0.5$ (65.48% vs 55.16%), which is not mentioned in the abstract.","pith_inferences":["The paper's own Table 2 implies the opposite of the abstract's headline: at 30 days the plain Transformer beats the PINN-Transformer on every threshold, so the attribution of long-lead skill to physical constraints is not supported by the presented numbers.","Because the governing PDE, exact form of the physics loss, and weighting coefficient are never specified, the experiment cannot be reproduced or independently verified without contacting the authors.","A natural next test would be to sweep the physics-loss weight, including setting it to zero, on the same data; if skill at 30 days stays flat or improves as the weight goes to zero, the claimed benefit vanishes.","Comparing the same architectures on a different basin (e.g., Bay of Bengal or tropical Pacific) would show whether the reported 30-day ordering is region-specific."],"forward_implications":["If the PINN-Transformer's 30-day advantage is real, adding physics-based loss terms to attention-based models could extend reliable SST forecasts to monthly lead times.","Short-term SST forecasting (7–15 days) would remain best served by purely data-driven CNNs and LSTMs, saving computation.","The lead-time-dependent ranking suggests hybrid models should be evaluated separately at each forecast horizon rather than with a single aggregate score.","Operational ocean and monsoon forecasting could adopt physics-informed Transformers for medium-range outlooks if the result replicates.","The reported grid-level thresholds give a concrete benchmark for future SST-prediction studies."],"supporting_citations":[{"why":"Supplies the physics-informed neural network framework that justifies adding PDE-based loss terms to the model.","marker":"Raissi et al. (2019)"},{"why":"Review of physics-informed machine learning used to argue that physical constraints improve model robustness.","marker":"Karniadakis et al. (2021)"},{"why":"Prior physics-guided neural network for SST prediction that the PINN-Transformer approach builds on.","marker":"Yuan et al. (2023)"},{"why":"Provides the ConvLSTM-Transformer architecture from which the plain Transformer implementation is adapted.","marker":"Choudhury et al. (2023)"},{"why":"Establishes the LSTM baseline for SST prediction and motivates temporal-dependency modeling.","marker":"Zhang et al. (2017)"},{"why":"Cited for the idea that physics integration may add unnecessary complexity in data-rich short-term forecasting.","marker":"Willard et al. (2022)"}],"fun_headline_variants":["SST forecasting: pure Transformer edges out physics-informed at 30 days","Physics-informed ML promises, but plain Transformer delivers at 30-day SST","Does physics-injection help SST? Table says no at 30 days","30-day SST: plain Transformer outperforms PINN, study shows"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the unspecified physics-informed loss terms (divergence and Laplacian of the wind components) actually encode the ocean dynamics that determine sea surface temperature, and that any 30-day performance difference is caused by those terms rather than by architecture details or training randomness.","fun_headline_variants_meta":{"raw":{"variants":["SST forecasting: pure Transformer edges out physics-informed at 30 days","Physics-informed ML promises, but plain Transformer delivers at 30-day SST","Does physics-injection help SST? Table says no at 30 days","30-day SST: plain Transformer outperforms PINN, study shows"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000436,"raw_usage":{"total_tokens":2190,"prompt_tokens":889,"completion_tokens":1301,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":1223}},"tokens_in":505,"tokens_out":1301,"duration_ms":18350,"temperature":1.0,"reasoning_tokens":1223,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:35:56.242936+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the 30-day testing metrics on the same 504-grid test set and compare the plain Transformer with the PINN-Transformer; the paper's own Table 2 already shows the plain Transformer ahead on all four thresholds ($ACC>0.8$: 58.13% vs 47.02%; $NSE>0.7$: 35.91% vs 19.25%; $NRMSE<0.1$: 20.04% vs 0.20%; $MAE<0.5$: 65.48% vs 55.16%), so any analysis that reproduces these numbers without the PINN-Transformer winning would falsify the headline claim.","supporting_citations":[],"review_version":1}