REVIEW 4 major objections 5 minor 34 references
Assessing the potential of state-of-the-art machine learning and physics-informed machine learning in predicting sea surface temperature
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A Transformer with an added physics-informed loss is claimed to give the best 30-day sea surface temperature forecasts in this benchmark.
desk verdict The paper's own Table 2 contradicts its central claim that the PINN-Transformer excels at 30-day SST prediction; the regional benchmark may have value after major fixes, but the current conclusions are unsupported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the physics-informed loss term added to the Transformer, described only as 'terms related to the divergence and Laplacian of the velocity components ($u$ and $v$).' The paper invokes this term as the reason the PINN-Transformer generalizes better at 30 days, because it constrains predictions to respect ocean-heat dynamics. The comparison design—four architectures across three lead times with four skill metrics (ACC, NSE, NRMSE, MAE) computed grid-by-grid at 504 grid points—is what the claim rests on.
What would settle it
Recompute the 30-day testing metrics on the same 504-grid test set and compare the plain Transformer with the PINN-Transformer; the paper's own Table 2 already shows the plain Transformer ahead on all four thresholds ($ACC>0.8$: 58.13% vs 47.02%; $NSE>0.7$: 35.91% vs 19.25%; $NRMSE<0.1$: 20.04% vs 0.20%; $MAE<0.5$: 65.48% vs 55.16%), so any analysis that reproduces these numbers without the PINN-Transformer winning would falsify the headline claim.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that incorporating physics-informed loss terms into a Transformer architecture—specifically terms tied to the divergence and Laplacian of the $u$ and $v$ wind components—lets the model sustain predictive skill at a 30-day lead time, where purely data-driven models degrade sharply. The authors argue that the hybrid model's ability to remain physically consistent is what makes it more robust and less prone to overfitting as the forecast horizon extends, and that this advantage only appears at longer lead times, making physics integration essential for medium-range SST prediction. In the same narrative, short-term forecasts (7–15 days) are adequately served by pure ML models, so the contribution is framed as a lead-time-dependent trade-off. The paper's own results table, however, shows the plain Transformer outperforming the PINN-Transformer at 30 days on $ACC>0.8$ (58.13% vs 47.02%), $NSE>0.7$ (35.91% vs 19.25%), $NRMSE<0.1$ (20.04% vs 0.20%), and $MAE<0.5$ (65.48% vs 55.16%), which is not mentioned in the abstract.
Load-bearing premise
The paper assumes that the unspecified physics-informed loss terms (divergence and Laplacian of the wind components) actually encode the ocean dynamics that determine sea surface temperature, and that any 30-day performance difference is caused by those terms rather than by architecture details or training randomness.
Editorial extensions
If this is right
- If the PINN-Transformer's 30-day advantage is real, adding physics-based loss terms to attention-based models could extend reliable SST forecasts to monthly lead times.
- Short-term SST forecasting (7–15 days) would remain best served by purely data-driven CNNs and LSTMs, saving computation.
- The lead-time-dependent ranking suggests hybrid models should be evaluated separately at each forecast horizon rather than with a single aggregate score.
- Operational ocean and monsoon forecasting could adopt physics-informed Transformers for medium-range outlooks if the result replicates.
- The reported grid-level thresholds give a concrete benchmark for future SST-prediction studies.
Reading between the lines
- The paper's own Table 2 implies the opposite of the abstract's headline: at 30 days the plain Transformer beats the PINN-Transformer on every threshold, so the attribution of long-lead skill to physical constraints is not supported by the presented numbers.
- Because the governing PDE, exact form of the physics loss, and weighting coefficient are never specified, the experiment cannot be reproduced or independently verified without contacting the authors.
- A natural next test would be to sweep the physics-loss weight, including setting it to zero, on the same data; if skill at 30 days stays flat or improves as the weight goes to zero, the claimed benefit vanishes.
- Comparing the same architectures on a different basin (e.g., Bay of Bengal or tropical Pacific) would show whether the reported 30-day ordering is region-specific.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper benchmarks four deep-learning models—CNN, ConvLSTM (referred to as LSTM), Transformer, and a physics-informed Transformer (PINN-Transformer)—for predicting sea surface temperature (SST) in the Arabian Sea at 7-, 15-, and 30-day lead times. Inputs are ERA5 reanalysis fields (six atmospheric predictors plus SST) at 0.5° resolution over 2000–2023, with 2000–2016 used for training and 2017–2023 for testing. Performance is assessed with anomaly correlation coefficient (ACC), Nash–Sutcliffe efficiency (NSE), normalized root mean square error (NRMSE), and mean absolute error (MAE). The abstract and conclusions claim that the PINN-Transformer excels at 30-day predictions, indicating that physical constraints improve long-term SST forecasting.
Significance. A well-executed comparison of modern ML architectures for SST forecasting would be of practical value, and the paper has a reasonable evaluation design: a held-out test period, four established metrics, and grid-level threshold percentages that are reported in full in Table 2. If the central claim were supported, it would suggest that physics-informed losses stabilize forecasts at longer lead times, a result of interest to the ocean forecasting community. However, the paper's headline claim is contradicted by its own Table 2, and the physics-informed component is too underspecified to support any attribution. The paper therefore does not deliver on its stated contribution, though the underlying benchmark data could be useful if the narrative were corrected and the methodology expanded.
major comments (4)
- [Abstract; §4.3; Table 2] The central claim that 'the PINN-Transformer excels in 30-day predictions' is directly contradicted by Table 2. At the 30-day lead time, the plain Transformer beats the PINN-Transformer on every reported threshold: ACC>0.8 on 58.13% vs 47.02% of grids, NSE>0.7 on 35.91% vs 19.25%, NRMSE<0.1 on 20.04% vs 0.20%, and MAE<0.5 on 65.48% vs 55.16%. The metric ranges in the same row also show no compensating advantage (e.g., ACC 0.72–0.88 vs 0.68–0.87). Section 4.3's statement that the PINN-Transformer 'stands out with the highest ACC and NSE values' and Figure 9's claim that the PINN-Transformer shows the 'highest NSE values' (0.46–0.58, versus 0.50–0.59 for the Transformer) are inconsistent with the numerical results. The conclusion in §5 that physical integration yields a 'distinct advantage' at longer lead times is unsupported by the paper's own data.
- [§3 (Methodology)] The physics-informed loss is never specified. The text states only that the PINN-Transformer's loss 'includes terms related to the divergence and Laplacian of the velocity components (u and v)', without giving the governing equation, the exact form of the additional loss terms, the weighting coefficient that balances data loss against physics loss, or the discretization used. No ablation is provided to isolate the effect of this term. Consequently, the attribution of any observed behavior to 'physical principles' cannot be tested or reproduced, and the reader cannot evaluate whether the constraint correctly represents ocean dynamics or is merely an arbitrary regularizer.
- [§5 (Conclusions) vs §2.2 and §4.1] The Conclusions state that the analysis 'was restricted to 504 grids at a 1-degree spatial resolution,' but §2.2 says the data were bilinearly interpolated to 0.5° × 0.5°, and §4.1 states that this resolution 'results in 504 grid points.' Over the stated domain (7.25°–18.75°N, 62°–72°E), 504 grid points are consistent with 0.5° spacing, not 1°. This internal inconsistency must be corrected, as the spatial resolution directly affects the interpretation of all results and the stated limitation.
- [§4.4; Table 2] The study provides no reference baseline (persistence, climatology, or an operational numerical forecast) against which the reported skill can be judged. Given the paper's title and introduction frame the work as assessing 'state-of-the-art' ML and PIML, absolute skill levels such as 'ACC>0.8 on 58% of grids at 30 days' need context. The four-model comparison can stand on its own, but the broader claim about the 'potential' of these methods requires a baseline, especially since all models show large declines at 30 days.
minor comments (5)
- [§4.3; Figure 9] The text in §4.3 says the PINN-Transformer has 'the highest NSE values, ranging from 0.46 to 0.58,' immediately after reporting the Transformer's NSE range of 0.50–0.59; the numerical ranges contradict the qualitative claim and should be fixed.
- [§4.1 and §1] The paper switches between 'ConvLSTM' and 'LSTM' without consistency: §3 introduces ConvLSTM, §4.1 says 'the ConvLSTM model will be referred to as LSTM from now on,' but the abstract and §1 use both names. This is confusing and should be standardized.
- [Table 2 caption] The caption contains a typo ('Tabel 2'); figure captions also inconsistently refer to 'CC' rather than 'ACC' (e.g., Figures 4, 6, 8).
- [§5 and Table 2] The conclusion that 'CNN and Transformer models exhibited robust performance in the 30-day forecast' is overstated: Table 2 shows CNN achieving NSE>0.7 on only 11.90% of grids and NRMSE<0.1 on 0% of grids at 30 days, and the Transformer also shows substantial degradation.
- [Data availability] The statement that source code is 'available from the corresponding author upon reasonable request' falls short of modern reproducibility standards; given that the final text also lacks hyperparameters and training details, making code and data publicly available would be necessary for verification.
Circularity Check
No circularity found: the paper is an empirical model benchmark whose central physical-improvement claim is contradicted by its own Table 2, but that is an internal-consistency problem, not an equation-level circularity.
full rationale
Walking the paper's derivation chain: the inputs are ERA5 atmospheric variables and SST; the models are trained on 2000-2016 and tested on 2017-2023; the reported ACC, NSE, NRMSE, and MAE values are computed from held-out test predictions. No equation in the paper defines a target quantity in terms of a fitted parameter, and no predicted result is equivalent by construction to an input or to a fitted quantity. The physics-informed loss is described only as 'terms related to the divergence and Laplacian of the velocity components (u and v)', with no explicit PDE or weighting coefficient, so there is no equation to exhibit as circular. The claim that the PINN-Transformer 'excels in 30-day predictions' is unsupported by the paper's own Table 2, where the plain Transformer outperforms the PINN-Transformer on all four 30-day thresholds (ACC>0.8, NSE>0.7, NRMSE<0.1, MAE<0.5); however, this is an internal inconsistency or correctness failure, not a circularity pattern. No load-bearing self-citations are present: the cited PINN references (Raissi et al. 2019, Karniadakis et al. 2021) are external background works, not prior papers by these authors. No uniqueness theorem, ansatz smuggled via citation, or renaming of a known result occurs. The paper is self-contained as an empirical comparison, so under the given circularity criteria it receives a score of 0.
Assumptions & free parameters
free parameters (3)
- physics-informed loss weighting coefficient =
not reported
- model hyperparameters (number of filters, kernel sizes, layers, dropout, learning rate) =
not reported
- input sequence length and lead-time construction =
not reported
assumptions (5)
- domain assumption ERA5 reanalysis SST and atmospheric variables are accurate enough to serve as ground truth for training and testing.
- domain assumption The six selected atmospheric predictors plus SST are sufficient to explain SST variability in the Arabian Sea.
- ad hoc to paper The physics-informed loss based on divergence and Laplacian of velocity components correctly represents SST-relevant ocean dynamics.
- domain assumption The chronological split (2000-2016 train, 2017-2023 test) provides a representative evaluation without non-stationarity issues.
- standard math Standard backpropagation with Adam optimization converges to useful minima for the reported architectures.
Cite this review
Pith. "Pith review of Assessing the potential of state-of-the-art machine learning and physics-informed machine learning in predicting sea surface temperature." pith.science (2026). https://pith.science/paper/WURHNBDO
@misc{pith2026241119031,
author = {Pith},
title = {Pith review of: Assessing the potential of state-of-the-art machine learning and physics-informed machine learning in predicting sea surface temperature},
year = {2026},
howpublished = {\url{https://pith.science/paper/WURHNBDO}},
note = {Machine review of arXiv:2411.19031}
}
read the original abstract
The growing adoption of machine learning (ML) in modelling atmospheric and oceanic processes offers a promising alternative to traditional numerical methods. It is essential to benchmark the performance of both ML and physics-informed ML (PINN) models to evaluate their predictive skill, particularly for short- to medium-term forecasting. In this study, we utilize gridded sea surface temperature (SST) data and six atmospheric predictors (cloud cover, relative humidity, solar radiation, surface pressure, u-component of velocity, and v-component of velocity) to capture both spatial and temporal patterns in SST predictions.
Figures
Reference graph
Works this paper leans on
-
[12]
Nature Clim Change, 5, 107–113
Tales of future weather. Nature Clim Change, 5, 107–113. https://doi.org/10.1038/nclimate2450 Hess, P., Boers, N.,
-
[15]
Nature Reviews Physics , 3(6), pp
Physics- informed machine learning. Nature Reviews Physics , 3(6), pp. 422 -440. https://doi.org/10.1038/s42254-021-00314-5. Kraus, E.B., Businger, J.A.,
-
[17]
Nature Clim Change, 7, 885 –889
Increasing frequency of extremely severe cyclonic storms over the Arabian Sea. Nature Clim Change, 7, 885 –889. https://doi.org/10.1038/s41558-017-0008-6 Nguyen, T., Brandstetter, J., Kapoor, A., Gupta, J. K., Grover, A.,
-
[18]
arXiv preprint arXiv:2301.10343
ClimaX: A foundation model for weather and climate. arXiv preprint arXiv:2301.10343. O’Carroll, A.G., Armstrong, E.M., Beggs, H.M., Bouali, M., Casey, K.S., et al.,
-
[20]
A review on regional convection -permitting climate modeling: Demonstrations, prospects, and challenges. Rev. Geophys., 53,323 –361, https://doi.org/10.1002/2014RG000475 Qiao, B., Wu, Z., Tang, Z., Wu, G.,
-
[21]
In: 2022 24th international conference on advanced communication technology (ICACT)
Sea surface temperature prediction approach based on 3d CNN and LSTM with attention mechanism. In: 2022 24th international conference on advanced communication technology (ICACT). IEEE, pp. 342–347 Raissi, M., Perdikaris, P., Karniadakis, G. E.,
work page 2022
-
[22]
Journal of Computational Physics, 378, 686 –707
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686 –707. https://doi.org/10.1016/j.jcp.2018.10.045 Rasp, S., Dueben, P. D., Scher, S., Weyn, J. A., Mouatadid, S., Thuerey, N .,
-
[23]
Journal of Advances in Modeling Earth Systems,12, e2020MS002203
WeatherBench: A benchmark data set for data-driven weather forecasting. Journal of Advances in Modeling Earth Systems,12, e2020MS002203. https://doi.org/10.1029/2020MS002203 Rasp, S., Lerch, S.,
Show all 34 references
-
[24]
Neural networks for postprocessing ensemble weather forecasts. Mon. Weather Rev, 146, 3885–3900. https://doi.org/10.1175/mwr-d-18-0187.1 Rodwell, M. J., Palmer, T. N.,
-
[25]
In Proceedings of the 2015 international conference on advanced computer science and information systems, Depok, Indonesia, 10–11 October 2015; pp
Weather Forecasting Using Deep Learning Techniques. In Proceedings of the 2015 international conference on advanced computer science and information systems, Depok, Indonesia, 10–11 October 2015; pp. 281–285 Scher, S.,
2015
-
[26]
Geophysical Research Letters, 45, 12,616 – 12,622
Toward data-driven weather and climate forecasting: Approximating a simple general circulation model with deep learning. Geophysical Research Letters, 45, 12,616 – 12,622. https://doi.org/10.1029/2018GL080704 Schultz, M. G., Betancourt, C., Gong, B., Kleinert, F., Langguth, M....
-
[27]
Philosophical Transactions of the Royal Society A, 379(2194), 20200097
Can deep learning beat numerical weather prediction?. Philosophical Transactions of the Royal Society A, 379(2194), 20200097. Shen, C., Appling, A.P., Gentine, P., et al., 2023 . Differentiable modelling to unify machine learning and physical models for geosciences. Nat Rev Ea...
2023 doi
-
[31]
Journal of Advances in Modeling Earth Systems, 11,2680 –2693
Can machines learn to predict weather? Using deeplearning to predict gridded 500 -hPageopotential height from historical weather data. Journal of Advances in Modeling Earth Systems, 11,2680 –2693. https://doi.org/10.1029/2019MS001705 Willard, J., Jia, X., Xu, S., Steinbach, M....
-
[34]
IEEE Geosci
Prediction of 3 -D ocean temperature by multilayer convolutional LSTM. IEEE Geosci. Remote Sens. Lett, 17, (8), 1303 –1307. https://doi.org/10.1109/LGRS.2019.2947170. Zhang, Q., Wang, H., Dong, J., Zhong, G., Sun, X.,
2019
-
[55]
https://doi.org/10.1145/3514228 Xiao, C., Chen, N., Hu, C.,Wang, K., Xu, Z., Cai, Y ., Xu, L., Chen, Z., Gong, J.,
-
[68]
https://doi.org/10.1007/978-3-540-79881-1_2 Solomatine, D.P., Ostfeld, A.,
Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-79881-1_2 Solomatine, D.P., Ostfeld, A.,
-
[588]
https://doi.org/10.1038/s43017-021-00210-7 Solomatine, D., See, L., Abrahart, R.,
-
[1317]
Analog forecasting of extreme - causing weather patterns using deep learning
https://doi.org/10.1038/s41598-020-57897-9 Chattopadhyay, A., Nabizadeh, E., Hassanzadeh, P., 2020a. Analog forecasting of extreme - causing weather patterns using deep learning. Journal of Advances in Modeling Earth Systems, 12, e2019MS001958. https://doi.org/10.1029/2019MS00...
-
[1656]
https://doi.org/ 10.3390/rs15061656 Hartmann, D. L.,
-
[1993]
Impact of clouds on the surface radiation balance of the Arctic Ocean. Meteorl. Atmos. Phys, 51,197–217. https://doi.org/10.1007/BF01030494 de Burgh-Day, C. O., Leeuwenburg, T.,
-
[2004]
Neural Comput and Applic, 13, 112 –122
An ensemble of neural networks for weather forecasting. Neural Comput and Applic, 13, 112 –122. https://doi.org/10.1007/s00521-004- 0413-4 Michalakes, J.,
-
[2008]
Data -driven modelling: some past experiences and new approaches. J. Hydroinformatics 10, 3–22. https://doi.org/10.2166/hydro. 2008.015 Steppeler, J., Li, J.,
2008 doi
-
[2010]
Sea surface temperature variability: Patterns and mechanisms, Annu. Rev. Mar. Sci., 2, 115 –143, https://doi.org/10.1146/annurev-marine-120408-151453 Evan, A. T., Camargo, S. J.,
-
[2014]
Geophys. Res. Lett., 42,1894–1902, https://doi/org/10.1002/2015GL063083. Hazeleger, W., van den Hurk, B., Min, E. et al.,
1902 doi
-
[2015]
379–386 Hao, P., Li, S., Song, J., Gao, Y .,
deep hybrid model for weather forecasting, in: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2015, pp. 379–386 Hao, P., Li, S., Song, J., Gao, Y .,
2015
-
[2017]
Remote Sens
A multi-scale high-resolution analysis of global sea surface temperature. Remote Sens. Environ, 200, 154 –169. https://doi.org/10.1016/j.rse.2017.07.029 Choudhury, B., Chakraborty, K., Singha, U. P., Kuri, D., Handique, M., Sharma, N.,
2017 doi
-
[2018]
Oceanography, 31, 162 –173
Categorizing and naming marine heatwaves. Oceanography, 31, 162 –173. https://doi.org/10.5670/oceanog.2018.205 Holton, J. R.,
2018 doi
-
[2019]
Front Mar Sci 6:420
Observational needs of sea surface temperature. Front Mar Sci 6:420. https://doi.org/10.3389/fmars.2019.00420 Palmer, T. N.,
2019
-
[2020]
IEEE Access 2020, 8, 180544–180557
Hybrid CNN -LSTM model for short -term individual household load forecasting. IEEE Access 2020, 8, 180544–180557. Bakun, A.,
2020
-
[2021]
Physics -informed neural networks (PINNs) for fluid mechanics: a review. Acta. Mech. Sin. 37(12), 1727 –1738. https://doi.org/10.1007/s10409- 021-01148-1 Cai, W., Ng, B., Wang, G. et al.,
-
[2022]
Increased ENSO Sea surface temperature variability under four IPCC emission scenarios. Nat. Clim. Chang , 12, 228 –231. https://doi.org/10.1038/s41558-022-01282-z Chattopadhyay, A., Hassanzadeh, P., Pasha, S., 2020b. Predicting clustered weather patterns: A test case for appli...
-
[2023]
TENCON 2023 - 2023 IEEE Region 10 Conference (TENCON), Chiang Mai, Thailand, 495-500, https://doi.org/10.1109/TENCON58879.2023.10322451
Study of Sea Surface Temperature Prediction and Oceanographic Exploration Using Deep Learning. TENCON 2023 - 2023 IEEE Region 10 Conference (TENCON), Chiang Mai, Thailand, 495-500, https://doi.org/10.1109/TENCON58879.2023.10322451. Curry, J.A., Schramm, J.L., Ebert, E.E.,
2023
- [3498]
- [4678]
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.