REVIEW 2 major objections 6 minor
Short-Term Turbulence Prediction for Seeing Using Machine Learning
T0 review · 2 major / 6 minor · reviewed 2026-07-13 · grok-4.5
Pith's one-line read A normalizing-flow model forecasts atmospheric seeing two hours ahead with the best balance of accuracy and calibrated uncertainty.
desk verdict Solid applied ML comparison for two-hour seeing forecasts; FloTS is a useful probabilistic addition, but the random split of overlapping windows is a real soft spot that leaves absolute numbers and the ranking claim provisional. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
FloTS: a masked autoregressive normalizing flow whose context encoder (an LSTM) maps the past seeing window into the parameters of a base Gaussian, which is then transformed into a flexible conditional distribution over the future two-hour window.
What would settle it
Re-split the same Maunakea DIMM series into strictly chronological blocks with no temporal overlap between train and test, retrain all four models, and check whether FloTS still shows the lowest RMSE and the best-calibrated coverage at the two-hour horizon.
Extended reading notes
Core claim
When only historical seeing measurements are available, a normalizing-flow model for time series (FloTS) yields the best overall balance of point accuracy and well-calibrated predictive uncertainty for two-hour-ahead forecasts, outperforming Gaussian processes and matching the deterministic accuracy of LSTMs while supplying richer non-Gaussian uncertainty estimates.
Load-bearing premise
Randomly shuffling overlapping two-hour windows into train, validation and test sets does not leak future information across the split and therefore does not inflate the reported forecast skill.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript develops short-term (up to 2 h) forecasts of atmospheric seeing from Maunakea DIMM data (2009–2024), resampled to 10 min. It compares deterministic RNN and LSTM encoder–decoder models with two probabilistic approaches: a Gaussian process with an exponential kernel and a novel masked-autoregressive normalizing-flow model (FloTS) conditioned by an LSTM context encoder. All models are trained only on historical seeing, with a 2 h input window selected by RMSE. Probabilistic forecasts are post-calibrated via temperature scaling of coverage (PP plots). On a held-out set the authors report RMSE ~0.20″ and competitive Pearson r for LSTM and FloTS at 2 h, with FloTS providing the best accuracy–uncertainty trade-off after calibration; two qualitative case studies illustrate the predictive densities.
Significance. If the ranking is robust under a leakage-free evaluation protocol, the work would be a useful contribution to operational astronomy and free-space optical communications: a practical, seeing-only probabilistic nowcast with explicit non-Gaussian uncertainty, calibrated coverage, and a clear comparison of statistical versus deep generative models. The introduction of FloTS for this domain, the systematic input-length study, and the transparent calibration analysis are genuine strengths. The result is incremental rather than transformative, but it is well motivated and of clear applied interest.
major comments (2)
- §2.2: After constructing 71 906 sliding-window pairs with a single-timestep stride (so consecutive pairs share 11/12 input and 11/12 target steps), the pairs are randomly shuffled into 81.5/8.5/10 % train/val/test. This protocol admits substantial temporal leakage: a test window can have nearly identical history and future already present (or nearly present) in training. Absolute RMSE, correlation, and PP-plot calibration for FloTS (and the LSTM/GP baselines) may therefore be inflated by partial memorization rather than genuine two-hour skill. The abstract and §5 claim that FloTS achieves the best accuracy–uncertainty balance rests on these metrics. A chronological or blocked temporal split (or at least a sensitivity check with non-overlapping windows / multi-day gaps) is required before the ranking can be considered secured.
- §3.3 and §4: GP hyperparameters are fit on random batches of only 500 pairs and then averaged; the same overlapping-window construction is used for evaluation. Combined with the Gaussian assumption that the authors themselves show is imperfect (Fig. 4 and Appendix A, where even piecewise-linear temperature scaling fails to reach the ideal PP line), the GP baseline is under-powered relative to FloTS. A fairer comparison would use a larger inducing-point or sparse GP (or a multi-scale kernel) under the same leakage-free split; otherwise the claimed superiority of FloTS over GP is only partially supported.
minor comments (6)
- Section numbering: the introduction states that Section 4 presents both the statistical evaluation and the case studies, then ‘Finally, Section 5 summarizes’; the body has a free-standing CASE STUDY block after §4. Renumber for clarity.
- Table 1 is referenced for RNN/LSTM hyperparameters but is empty/placeholder in the draft; supply the actual numbers (hidden units, dropout, lr, etc.).
- Fig. 1 caption and text use both ‘FloTS’ and ‘FLOW’; standardize nomenclature.
- Eqs. (1)–(3) and (7)–(12) mix encoder/decoder notation (h_t vs g_t, c_t vs d_t); a short clarifying sentence would help readers follow the autoregressive decoder.
- No code or data-release statement is given; for a methods paper in astro-ph.IM a public repository (even if only the trained models and split indices) would strengthen reproducibility.
- Minor typos: ‘Short-T erm’, ‘T urbulence’, ‘Universit´ e’, and duplicated Cherubini et al. 2021 entries in the reference list.
Circularity Check
No derivation circularity: FloTS ranking is empirical supervised learning on held-out windows; calibration is standard post-hoc temperature scaling, not a self-definition of the target.
-
self citation load bearing
[Section 3.6 (Calibration of Probability Distributions)]
"Building on the calibration prescription of Srinivasan et al. (2025), a generalization of the temperature scaling in Guo et al. (2017), we look at the mismatch between the empirical and nominal coverage to characterize the fidelity of the modeled probability distribution with that of the data samples."
The temperature-based recalibration that produces the final PP plots and case-study densities for FloTS (and GP) is taken from a prior paper whose author list overlaps the present one. The step is not load-bearing for the ranking claim itself (which rests on uncalibrated RMSE/correlation as well), and the method is a standard post-hoc density adjustment rather than a definition of the seeing target; hence only a minor self-citation, not a circular reduction of the scientific result.
full rationale
The paper's central claim (FloTS best accuracy–uncertainty trade-off for 2-hour seeing forecasts) is an empirical comparison of four models trained only on historical DIMM seeing, evaluated by RMSE, Pearson r, and PP-plot coverage on a held-out 10% test set of sliding-window pairs. There is no first-principles derivation that reduces a predicted quantity to a fitted constant by construction, no uniqueness theorem imported from the authors, and no renaming of a known result as a new prediction. The sole self-reference is the calibration procedure of Srinivasan et al. (2025), used only as a post-processing temperature rescaling of already-trained predictive densities; it does not define the seeing target or force the FloTS ranking. Minor methodological concerns (random shuffle of overlapping windows) affect correctness/leakage risk, not circularity of the derivation chain. Score 1 reflects only that light self-citation of the calibration method; the scientific claim itself is self-contained against external data.
Assumptions & free parameters
free parameters (5)
- RNN/LSTM hidden size, dropout, learning rate, scheduler patience
- GP exponential-kernel length-scale, signal variance, noise variance
- FloTS MAF architecture + LSTM context-encoder weights
- Calibration temperature polynomial coefficients T1…T5
- Input window length (chosen 2 h)
assumptions (4)
- domain assumption Seeing time series can be treated as regularly sampled after local linear interpolation inside continuous segments; gaps >10 min start new segments.
- domain assumption A 2-hour history is sufficient; longer temporal dependencies are weak or unhelpful for a 2-hour forecast.
- ad hoc to paper Random shuffle of overlapping (input, target) pairs yields an unbiased estimate of generalization.
- domain assumption Gaussian process prior with exponential kernel adequately models residual fluctuations after per-window demeaning.
invented entities (1)
-
FloTS (normalizing-flow model for time series)
Cite this review
Pith. "Pith review of Short-Term Turbulence Prediction for Seeing Using Machine Learning." pith.science (2026). https://pith.science/paper/7OI7A7TI
@misc{pith2026260324466,
author = {Pith},
title = {Pith review of: Short-Term Turbulence Prediction for Seeing Using Machine Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/7OI7A7TI}},
note = {Machine review of arXiv:2603.24466}
}
read the original abstract
Optical turbulence, driven by fluctuations of the atmospheric refractive index, poses a significant challenge to ground-based optical systems, as it distorts the propagation of light. This degradation affects both astronomical observations and free-space optical communications. While adaptive optics systems correct turbulence effects in real-time, their reactive nature limits their effectiveness under rapidly changing conditions, underscoring the need for predictive solutions. In this study, we address the problem of short-term turbulence forecasting by leveraging machine learning models to predict the atmospheric seeing parameter up to two hours in advance. We compare statistical and deep learning approaches, with a particular focus on probabilistic models that not only produce accurate forecasts but also quantify predictive uncertainty, crucial for robust decision-making in dynamic environments. Our evaluation includes Gaussian processes (GPs) for statistical modeling, recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) as deterministic baselines, and our novel implementation of a normalizing flow for time series (FloTS) as a flexible probabilistic deep learning method. All models are trained exclusively on historical seeing data, allowing for a fair performance comparison. We show that FloTS achieves the best overall balance between predictive accuracy and well-calibrated uncertainty.
Reviewed July 13, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.