REVIEW 3 major objections 7 minor 18 references
With every other part of a 10-minute solar forecast frozen, modern sky-image encoders all beat smart persistence on Folsom, and two nearly tie near 65 W/m² RMSE.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-30 17:10 UTC pith:VYYB5BDC
load-bearing objection Honest fixed-pipeline backbone bake-off for 10-min sky-image GHI; useful hygiene and operating points, not a resolved architecture ranking. the 3 major comments →
A Controlled Visual-Backbone Benchmark for Multimodal Short-Term Solar Irradiance Forecasting
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Under one fixed multimodal 10-minute-ahead clear-sky-index pipeline, all evaluated visual backbones improve on clear-sky-index smart persistence on the Folsom strict split, with single-seed VMamba Small and Swin Base reaching matched RMSE of 65.39 and 65.50 W/m² versus persistence at 81.37 W/m²; larger encoders do not consistently help, compact CNN/Transformer variants keep most of the skill at far higher throughput, and the same protocol on a 313-sample NREL split shows persistence remaining strongest.
What carries the argument
The strict_v1 fixed comparison protocol: identical 224×224 masked sky-dome inputs, four-stage projectors to a 1024-D visual descriptor, single-layer LSTM on a 40-step weather history, concatenation fusion, lightweight regression head, Huber loss, AdamW schedule, seed 42, and chronological splits—so only backbone family and scale change.
Load-bearing premise
The shared lightweight fusion head, fixed short training schedule, and reduced backbone learning rate are neutral enough that RMSE differences mainly reflect encoder quality rather than mismatch or under-training of larger models.
What would settle it
Re-run the top Folsom models with multiple independent seeds and compute paired confidence intervals over test days; if the visual rows no longer beat smart persistence or the near-tie between VMamba Small and Swin Base reverses beyond the interval, the single-seed ranking claim fails.
If this is right
- Encoder choice for sky-image irradiance forecasting should be reported together with smart-persistence skill, a temporal-only diagnostic, and throughput, not as a lone architecture ranking.
- Compact ConvNeXt Tiny or Swin Tiny can retain nearly the Folsom skill of heavier models at several times the frame rate under the same head.
- A ranking obtained on a large site like Folsom should not be transferred to a low-matched-sample site without re-checking persistence and sample counts.
- Extra visual capacity does not automatically lower error for a scalar 10-minute clear-sky-index target when fusion and training budget stay fixed.
Where Pith is reading between the lines
- The protocol could be reused as a standard “encoder bake-off” contract for other multimodal nowcasting tasks (e.g., wind or precipitation from camera feeds) where fusion heads are often re-tuned per backbone.
- If repeated-seed intervals confirm the Folsom near-ties, deployment decisions can safely prioritize FPS and parameter count over chasing the single lowest RMSE backbone.
- The NREL collapse under long physical history windows suggests that history length in real time, not step count, should be an explicit axis in any follow-on controlled benchmark.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a controlled benchmark protocol ("strict_v1") for comparing visual backbones in multimodal 10-min-ahead solar irradiance forecasting: preprocessing, clear-sky-index target, LSTM weather branch, fusion, regression head, loss, optimizer schedule, seed, and chronological splits are all fixed, and only the visual backbone (ConvNeXt, Swin, VMamba, Spatial Mamba, MambaVision; 19 configurations) is varied. On the Folsom strict split (224,022 test samples, 2016 held out chronologically), every visual-backbone run beats clear-sky-index smart persistence (81.37 W/m²), with single-seed VMamba S and Swin B at 65.39 and 65.50 W/m²; a temporal-only diagnostic reaches 69.51. On a 313-sample strict NREL split, persistence wins and all visual rows show negative skill, which the authors correctly treat as a low-data stress test. The manuscript explicitly disclaims architecture-level dominance and statistically resolved ranking.
Significance. If the numbers hold, this is a useful and unusually disciplined contribution to a literature where encoder, temporal model, fusion, and training recipe typically change together, rendering cross-paper comparisons uninterpretable. Specific strengths worth naming: (i) a genuinely controlled protocol with chronological held-out years and matched samples across all rows; (ii) evaluation against clear-sky-index smart persistence, the correct operational baseline, rather than raw persistence; (iii) a temporal-only diagnostic reported as a diagnostic rather than silently folded into the ranking; (iv) public code; and (v) conservative scoping — the abstract and §V.E explicitly refuse top-1 and dominance claims. The robust central finding (all visual rows beat smart persistence on Folsom by 15–20%) is unlikely to reverse under reseeding and is a meaningful, if modest, statement about the value of sky-image features under a shared head. The NREL negative result is also valuable as a caution against transferring single-site rankings.
major comments (3)
- [Table III; Abstract; §V.E] All runs use seed 42 with a single training pass (Table II), yet the abstract reports specific operating points (65.39 vs 65.50 W/m²) and Table III marks gold/silver/bronze cells, which constitutes a de-facto top-3 ranking over a spread of 0.60 W/m². Seed-to-seed test-RMSE fluctuation of 0.3–1.0 W/m² at this error level is typical for end-to-end fine-tuning with a 0.1× backbone LR and 8-epoch budget, so the medal ordering plausibly does not survive reseeding. §V.E concedes this, but the abstract and table presentation still carry the ranking. Either (a) run 3–5 seeds for at least the top-3 Folsom models plus the temporal-only row and report mean±std with a paired day-blocked test, or (b) remove the medal shading and present the abstract numbers as an unordered band. Given the paper's stated refusal of a top-1 claim, option (b) is cheap and consistent; option (a) is the higher-value fix.
- [§V.D; §VI; Table II] The conclusion that 'larger encoders do not consistently improve' rests on the assumption that the shared head, fixed 8-epoch AdamW schedule, and 0.1× backbone LR ratio are a neutral interface. For the largest models (ConvNeXt L 197M, MambaVision L 229M), 8 epochs at 5e-6 effective backbone LR may simply under-train the encoder, making the observed non-monotonic scaling a protocol artifact rather than an encoder property — the manuscript admits this possibility in §V.D/§V.E but the conclusion is stated affirmatively in §VI. A minimal load-bearing check: for one small and one large backbone of the same family (e.g., ConvNeXt T vs L), run a small sweep over backbone LR ratio (e.g., 0.1×, 0.3×, 1.0×) or doubled epochs, and report whether the ordering flips. Without this, conclusion 2 in §VI should be softened to match the §V.D hedging.
- [§V.A; Table III] The temporal-only diagnostic reaches 69.51 W/m², only ~4 W/m² (≈6%) behind the best visual row, so the marginal value of the image branch under this head is itself a near-tie-scale quantity of unknown seed stability. Additionally, the 224,022 test samples at 60 s cadence are strongly autocorrelated, so any sample-level uncertainty would be misleadingly tight; the effective sample size is closer to the number of distinct test days. If uncertainty quantification is added (per major comment 1), it should be block-bootstrapped or paired at the day level, and the visual-vs-temporal-only gap should be among the quantities tested, since it bears directly on how much the image encoder contributes under the fixed head.
minor comments (7)
- [§III.C] The temporal-only diagnostic is constructed by zeroing z_v while keeping the full 1152-D head (§III.C). The head's visual-side weights were therefore trained against a zero input; a temporal-only model trained from scratch with a 128-D input head would be a slightly different (arguably stronger) diagnostic. This is a defensible choice for interface preservation, but one sentence acknowledging the distinction would help readers interpret the 69.51 figure.
- [§III.A] The choice to clip k to [0,1.2] and to exclude zenith angles above 85° is stated without justification or sensitivity check. A brief rationale (and the fraction of samples removed by each filter, beyond Table I's aggregate counts) would improve reproducibility.
- [§III.B] The sky-dome mask radius is given as '250/512 of the image width' applied after resizing to 224×224 — it is unclear whether the radius is applied in the original 512-px coordinate frame before resizing or scaled to the 224-px frame. Please clarify.
- [§IV.B; Fig. 2] FPS is measured with batch size 4 and random inputs (§IV.B), which excludes data-loading and preprocessing costs; for a deployment-oriented Pareto claim (Fig. 2), note explicitly that these are compute-only numbers. Also, Fig. 2's axis tick labels (40–600 non-uniform) and the unlabeled 'T/BS/S' annotations are hard to read; consider log-x or a cleaner annotation scheme.
- [§V.C; Fig. 3] The ERF overlays (Fig. 3) are qualitative and, as the authors note, non-causal. Consider either moving them to an appendix or adding one quantitative summary (e.g., fraction of ERF mass within the central disk) to make the comparison across the four panels more than visual.
- [Table IV] The NREL strict split (313 test samples) is well handled, but Table IV shows R² = 0.069 (Swin T) and −0.378 (MambaVision L) — worth one sentence noting that with n=313 and seasonal concentration these R² values are near-noise, to preempt readers over-interpreting them.
- [Abstract; References] Abstract typos from typesetting: 'for10min-ahead', missing spaces around numbers and units ('65.39W m−2', 'the313-sample'). Reference [17] is a 2026 arXiv preprint by an overlapping author group; it is used only for motivation, which is fine, but its status (preprint vs published) should be stated.
Circularity Check
No circularity: empirical backbone bake-off against external CSI smart persistence on held-out chronological splits.
full rationale
The paper is a controlled empirical comparison, not a derivation of a first-principles prediction. The load-bearing claims are measured RMSE/skill of end-to-end trained models versus clear-sky-index smart persistence (Eq. 4–5) on chronological test years (Folsom 2016; NREL 2020). Smart persistence ˆk_sp = k_t and the CSI target k = GHI/GHI_cs are standard external baselines, not quantities defined from the model outputs. Train/val-only standardization and inverse-transform to W/m² are ordinary ML evaluation hygiene, not fitted-input-as-prediction. The sole overlapping-author citation [17] supplies only methodological motivation for a fixed-protocol encoder comparison; none of the Folsom/NREL numbers, rankings, or skill scores reduce to it. Temporal-only rows are explicitly labeled diagnostics, not ranking criteria. No self-definitional loop, uniqueness import, or renamed known result appears in the claim chain. Score 0 is the honest finding.
Axiom & Free-Parameter Ledger
free parameters (6)
- backbone_lr_ratio =
0.1
- AdamW_base_lr_and_schedule =
lr=5e-5, wd=0.05, epochs=8, seed=42
- weather_history_length_steps =
40
- sky_dome_mask_radius_fraction =
250/512
- visual_projector_and_head_widths =
256×4 / LSTM128 / MLP256 / drop0.3 / δ=1.0
- k_clip_and_zenith_filter =
k∈[0,1.2], z≤85°
axioms (5)
- domain assumption Ineichen clear-sky model yields a valid GHI_cs for defining k and smart persistence in physical units.
- domain assumption Chronological year splits (Folsom 2014–15/2016, NREL 2018–19/2020) prevent leakage and are a fair operational test.
- domain assumption ImageNet-pretrained hierarchical four-stage features remain meaningful after sky-dome masking and end-to-end fine-tuning for scalar CSI regression.
- ad hoc to paper Holding fusion, loss, and optimization fixed isolates backbone effects enough for comparative (not absolute optimal) ranking.
- domain assumption Smart persistence in CSI space is the appropriate skill denominator for short-horizon GHI.
invented entities (1)
-
strict_v1 fixed multimodal comparison protocol
independent evidence
Cite this review
Pith. "Pith review of A Controlled Visual-Backbone Benchmark for Multimodal Short-Term Solar Irradiance Forecasting." pith.science (2026). https://pith.science/paper/VYYB5BDC
@misc{pith2026260723633,
author = {Pith},
title = {Pith review of: A Controlled Visual-Backbone Benchmark for Multimodal Short-Term Solar Irradiance Forecasting},
year = {2026},
howpublished = {\url{https://pith.science/paper/VYYB5BDC}},
note = {Machine review of arXiv:2607.23633}
}
read the original abstract
Sky-image irradiance studies often compare forecasting systems in which the image encoder, temporal model, fusion block, target definition, and training recipe all change together. We use a narrower protocol: the multimodal forecasting pipeline is fixed, and only the visual backbone is varied. The shared setup keeps preprocessing, clear-sky-index normalization, weather-history encoding, fusion, regression head, loss, optimizer schedule, seed, and chronological split policy unchanged. We compare ConvNeXt, Swin Transformer, VMamba, Spatial Mamba, and MambaVision backbones for 10min-ahead forecasting on Folsom and a strict matched NREL split. Forecast skill is measured against clear-sky-index smart persistence, and temporal-only rows are reported as weather-history diagnostics rather than as the main ranking criterion. On the Folsom strict split, all evaluated visual-backbone runs improve over smart persistence. In the evaluated single-seed strict runs, VMamba Small and Swin Base reach matched Folsom RMSE values of 65.39 W/m^2 and 65.50 W/m^2; the temporal-only diagnostic reaches 69.51 W/m^2. On the 313-sample NREL strict split, smart persistence remains strongest at 17.48 W/m^2, while the lowest visual RMSE is obtained by Swin Tiny at 23.76 W/m^2. These results provide a reproducible encoder comparison under one fixed multimodal operating point rather than establishing architecture-level dominance, statistically resolved ranking, or fully optimized forecasting performance. Code available here: https://github.com/Oshadha345/irradiance_benchmark
Figures
Reference graph
Works this paper leans on
-
[1]
Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” 2022. [Online]. Available: https://arxiv.org/abs/2201.03545
Pith/arXiv arXiv 2022
-
[2]
Swin transformer: Hierarchical vision transformer using shifted windows,
Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” 2021. [Online]. Available: https://arxiv.org/abs/2103.14030
Pith/arXiv arXiv 2021
-
[3]
Vmamba: Visual state space model,
Y . Liu, Y . Tian, Y . Zhao, H. Yu, L. Xie, Y . Wang, Q. Ye, J. Jiao, and Y . Liu, “Vmamba: Visual state space model,” 2024. [Online]. Available: https://arxiv.org/abs/2401.10166
Pith/arXiv arXiv 2024
-
[4]
Spatial-mamba: Effective visual state space models via structure-aware state fusion,
C. Xiao, M. Li, Z. Zhang, D. Meng, and L. Zhang, “Spatial-mamba: Effective visual state space models via structure-aware state fusion,”
-
[5]
Mambavision: A hybrid mamba- transformer vision backbone,
A. Hatamizadeh and J. Kautz, “Mambavision: A hybrid mamba- transformer vision backbone,” 2025. [Online]. Available: https: //arxiv.org/abs/2407.08083
Pith/arXiv arXiv 2025
-
[6]
Open- source sky image datasets for solar forecasting with deep learning: A comprehensive survey,
Y . Nie, X. Li, Q. Paletta, M. Aragon, A. Scott, and A. Brandt, “Open- source sky image datasets for solar forecasting with deep learning: A comprehensive survey,”Renewable and Sustainable Energy Reviews, vol. 189, p. 113977, 2024
2024
-
[7]
Very short-term solar irradiance forecast using all-sky imaging and real-time irradiance measurements,
M. Caldas and R. Alonso-Su ´arez, “Very short-term solar irradiance forecast using all-sky imaging and real-time irradiance measurements,” Renewable Energy, vol. 143, pp. 1643–1658, 2019
2019
-
[8]
Short term solar irradiance forecasting using sky images based on a hybrid cnn–mlp model,
O. El Alani, M. Abraim, H. Ghennioui, A. Ghennioui, I. Ikenbi, and F.-E. Dahr, “Short term solar irradiance forecasting using sky images based on a hybrid cnn–mlp model,”Energy Reports, vol. 7, pp. 888–900, 2021, technologies and Materials for Renewable Energy, Environment and Sustainability
2021
-
[9]
All sky imaging-based short-term solar irradiance forecasting with long short-term memory networks,
N. Hendrikx, K. Barhmi, L. Visser, T. de Bruin, M. P ´o, A. Salah, and W. van Sark, “All sky imaging-based short-term solar irradiance forecasting with long short-term memory networks,”Solar Energy, vol. 272, p. 112463, 2024
2024
-
[10]
A transformer- based multimodal-learning framework using sky images for ultra-short- term solar irradiance forecasting,
J. Liu, H. Zang, L. Cheng, T. Ding, Z. Wei, and G. Sun, “A transformer- based multimodal-learning framework using sky images for ultra-short- term solar irradiance forecasting,”Applied Energy, vol. 342, p. 121160, 2023
2023
-
[11]
Sky image-based solar forecasting using deep learning with heterogeneous multi-location data: Dataset fusion versus transfer learning,
Y . Nie, Q. Paletta, A. Scott, L. M. Pomares, G. Arbod, S. Sgouridis, J. Lasenby, and A. Brandt, “Sky image-based solar forecasting using deep learning with heterogeneous multi-location data: Dataset fusion versus transfer learning,”Applied Energy, vol. 369, p. 123467, 2024
2024
-
[12]
Solar forecasts based on the clear sky index or the clearness index: Which is better?
P. Lauret, R. Alonso-Su ´arez, J. Le Gal La Salle, and M. David, “Solar forecasts based on the clear sky index or the clearness index: Which is better?”Solar, vol. 2, no. 4, pp. 432–444, 2022
2022
-
[13]
Blending of a novel all sky imager model with persistence and a satellite based model for high-resolution irradiance nowcasting,
N. Straub, W. Herzberg, A. Dittmann, and E. Lorenz, “Blending of a novel all sky imager model with persistence and a satellite based model for high-resolution irradiance nowcasting,”Solar Energy, vol. 269, p. 112319, 2024
2024
-
[14]
Using sky-classification to improve the short- term prediction of irradiance with sky images and convolutional neural networks,
V . A. Martinez Lopez, G. van Urk, P. J. Doodkorte, M. Zeman, O. Isabella, and H. Ziar, “Using sky-classification to improve the short- term prediction of irradiance with sky images and convolutional neural networks,”Solar Energy, vol. 269, p. 112320, 2024
2024
-
[15]
Benchmarking deep learning methods for irradiance estimation from sky images with applications to video prediction-based irradiance nowcasting,
L. F. Varaschin and D. Silva, “Benchmarking deep learning methods for irradiance estimation from sky images with applications to video prediction-based irradiance nowcasting,”Solar Energy, vol. 302, p. 114000, Dec. 2025
2025
-
[16]
Mamba: Linear-time sequence modeling with selective state spaces,
A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” 2024. [Online]. Available: https://arxiv.org/abs/ 2312.00752
Pith/arXiv arXiv 2024
-
[17]
N. Wasalathilaka, D. Perera, O. Samarakoon, B. Wijenayake, R. Godaliyadda, V . Herath, and P. Ekanayake, “A controlled benchmark of visual state-space backbones with domain-shift and boundary analysis for remote-sensing segmentation,” 2026. [Online]. Available: https://arxiv.org/abs/2604.18721
Pith/arXiv arXiv 2026
-
[2025]
Available: https://arxiv.org/abs/2410.15091
[Online]. Available: https://arxiv.org/abs/2410.15091
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.