Pith. sign in

REVIEW 1 major objections 2 minor 3 references

A two-stage cascade predicts aurora visibility by multiplying occurrence probability with clear-observation probability, reaching ROC-AUC 0.937.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 16:20 UTC pith:UDS3JCHE

load-bearing objection The two-stage cascade reports solid AUC gains on held-out and independent sites, but all-sky image labels for occurrence already embed clear-sky information, so the claimed decoupling of physics from local conditions is likely compromised. the 1 major comments →

arxiv 2605.24038 v1 pith:UDS3JCHE submitted 2026-05-21 physics.space-ph astro-ph.EPastro-ph.IMcs.LG

Aurora Hunter: A Two-Stage Framework for Probabilistic Visibility Forecasting

Zongyuan Ge , Chenwaner Zhang , Haoyang Li , Hantai Zhang , Wenxin Gu , Wei Zhou , Zhaoming Wang This is my paper
classification physics.space-ph astro-ph.EPastro-ph.IMcs.LG
keywords aurora visibilityprobabilistic forecastingtwo-stage cascadeXGBoostspace weathercloud coverlunar illuminationROC-AUC
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Aurora visibility at a site depends on whether the aurora is physically occurring due to solar wind activity and whether clouds or moonlight allow naked-eye detection. The paper decouples these into two stages: one model predicts occurrence using physics features, and a second predicts clear conditions using cloud and lunar features. Their product gives the overall visibility probability. This separation improves performance over training a single model on all features at once. The result supports better planning for space weather studies and aurora viewing.

Core claim

Aurora Hunter predicts P(visible) as P(occurring) times P(clear|occurring). Stage 1 uses XGBoost on 51 solar-wind features trained on joint Tromso-Kiruna hourly samples with image-classifier labels. Stage 2 uses logistic regression on 21 cloud and lunar features trained only on occurring hours. The cascade reaches ROC-AUC 0.937 on Tromso 2019-2020 test data and 0.905 on independent Kiruna 2024 data, improving the single-stage baseline by 0.087, with generalization shown on held-out Skibotn data.

What carries the argument

The cascade formula P(visible) = P(occurring) × P(clear|occurring) that trains separate models on occurrence features versus observational-condition features.

Load-bearing premise

The product of the two stage outputs equals the true joint visibility probability, which holds only if the occurrence and clear stages are conditionally independent or any dependence is captured by the separate feature sets.

What would settle it

On a new set of hourly site observations, a single model trained on the combined feature set achieving higher ROC-AUC than the cascade product would falsify the claimed benefit of decoupling.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Reaches ROC-AUC 0.937 on Tromso test data from 2019-2020
  • Reaches ROC-AUC 0.905 on independent Kiruna data from 2024
  • Generalizes to held-out Skibotn data from 2022-2025
  • SHAP analysis identifies Kp x nightside interaction, MLT position, and auroral oval distance as dominant predictors accounting for 39 percent combined importance

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same separation of event probability from detection probability could apply to visibility forecasts for other transient phenomena such as meteor showers.
  • Separate stages allow one model to be updated with new solar data without retraining the cloud model when weather data changes.
  • The cascade output could feed into decision tools that weigh visibility probability against travel costs for observers.
  • Adding real-time satellite cloud imagery to the second stage would test whether the current 21 features are sufficient for the clear-observation task.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 2 minor

Summary. The paper introduces Aurora Hunter, a two-stage probabilistic cascade for aurora visibility forecasting. Stage 1 uses XGBoost on 51 solar-wind and geomagnetic features to predict P(occurring) from Tromsø AI all-sky image labels (∼16,600 hourly samples, 2015–2023). Stage 2 uses logistic regression on 21 cloud/lunar features to predict P(clear|occurring) only on aurora-occurring hours. The product P(visible) = P(occurring) × P(clear|occurring) is reported to reach ROC-AUC 0.937 on the Tromsø 2019–2020 test set and 0.905 on independent Kiruna 2024 data, outperforming a single-stage baseline by +0.087; cross-site generalization is checked on Skibotn data and SHAP analysis highlights Kp × nightside, MLT, and oval distance as key drivers.

Significance. If the claimed separation of physical occurrence from local observing conditions holds, the cascade offers improved interpretability over black-box single-stage models and supports site-specific forecasting for research and tourism. Strengths include temporal hold-out splits, an independent geographic test site, and explicit SHAP attribution; these elements allow falsifiable evaluation of the decoupling hypothesis.

major comments (1)
  1. [Abstract and §2] Abstract and §2 (label construction): the Tromsø AI all-sky image classifier labels used for Stage 1 can only register aurora when skies are sufficiently clear for detection. Consequently the Stage-1 target encodes both physical occurrence and observability, violating the conditional-independence assumption required for the product formula to isolate the two physical drivers. This label contamination directly undermines the central claim that the cascade decouples solar-wind physics from cloud/lunar effects; the reported AUC gains may therefore reflect the shared information rather than genuine separation.
minor comments (2)
  1. [Results] No error bars, confidence intervals, or hyperparameter-search protocol are reported for the AUC values or the logistic-regression coefficients; these should be added to allow assessment of statistical significance of the +0.087 improvement.
  2. [Methods] The manuscript does not state whether the product formula was validated against joint (occurrence AND clear) labels on a held-out set; such a calibration check would strengthen the modeling claim.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive review and for raising this important point about label construction. We respond to the major comment below.

read point-by-point responses

  1. Referee: [Abstract and §2] Abstract and §2 (label construction): the Tromsø AI all-sky image classifier labels used for Stage 1 can only register aurora when skies are sufficiently clear for detection. Consequently the Stage-1 target encodes both physical occurrence and observability, violating the conditional-independence assumption required for the product formula to isolate the two physical drivers. This label contamination directly undermines the central claim that the cascade decouples solar-wind physics from cloud/lunar effects; the reported AUC gains may therefore reflect the shared information rather than genuine separation.

    Authors: We agree that the Tromsø AI all-sky image labels used as targets for Stage 1 are positive only when aurora is both physically occurring and sufficiently clear for the classifier to register it in the images. This introduces some dependence between the Stage 1 target and local observing conditions, which means the strict conditional-independence assumption required for the product formula to fully isolate the two drivers does not hold perfectly. The manuscript's central claim of decoupling is therefore qualified by this label property, and the reported AUC improvement may partly reflect shared information rather than pure separation. We will revise the abstract and §2 to explicitly acknowledge this limitation in label construction, discuss its implications for the product formula, and moderate the language around complete decoupling of solar-wind physics from cloud/lunar effects while retaining the empirical performance and interpretability results. revision: yes

Circularity Check

0 steps flagged

No circularity: cascade uses independent stages and external validation

full rationale

The paper defines the cascade P(visible)=P(occurring)*P(clear|occurring) as a modeling choice with separate feature sets and training subsets (Stage 1 on all samples with image labels, Stage 2 only on occurring hours). No equation reduces to its input by construction, no parameter is fitted then renamed as a prediction, and no self-citation chain supports a load-bearing claim. Performance is measured on held-out temporal splits (Tromso 2019-2020) and an independent site (Kiruna 2024), so reported AUC values are not forced by the training procedure. The label source for Stage 1 raises a potential correctness issue about confounding but does not create a self-referential reduction.

Axiom & Free-Parameter Ledger

2 free parameters · 2 axioms · 0 invented entities

The central performance numbers rest on standard supervised-learning assumptions plus the specific modeling choice of multiplying the two stage outputs.

free parameters (2)
  • XGBoost hyperparameters
    Tree depth, learning rate, and regularization parameters are chosen during training on the 16,600 samples.
  • Logistic regression coefficients
    21 coefficients plus intercept are fitted on the aurora-occurring subset.
axioms (2)
  • domain assumption The Tromso AI all-sky image classifier supplies accurate binary labels for aurora occurrence.
    These labels are used as ground truth for Stage 1 training.
  • domain assumption The product of the two stage probabilities equals the joint visibility probability.
    This is the explicit cascade formula used to obtain the final ROC-AUC numbers.

pith-pipeline@v0.9.1-grok · 5820 in / 1667 out tokens · 47024 ms · 2026-06-30T16:20:52.134338+00:00 · methodology

0 comments
read the original abstract

Forecasting aurora borealis visibility matters for space weather research and aurora tourism. Visibility at a site and night depends on two distinct factors: (1) whether aurora is physically occurring, driven by solar wind-magnetosphere coupling, and (2) whether observing conditions allow naked-eye detection, mainly cloud cover and lunar illumination. We present Aurora Hunter, a two-stage cascade that decouples these factors. Stage 1 predicts P(occurring) with XGBoost using 51 physics-driven features trained on joint Tromso+Kiruna data (about 16,600 hourly samples, 2015-2023) with labels from the Tromso AI all-sky image classifier. Stage 2 predicts P(clear observation given occurring) with logistic regression using 21 cloud-cover and lunar-illumination features trained only on aurora-occurring hours. The cascade P(visible)=P(occurring)*P(clear|occurring) reaches ROC-AUC 0.937 (Tromso test, 2019-2020) and 0.905 (independent Kiruna, 2024), improving a single-stage baseline by +0.087. Held-out Skibotn data (2022-2025) confirm cross-site generalization. SHAP identifies the Kp x nightside interaction, MLT position, and auroral oval distance as dominant predictors (39% combined). Prototype: https://aurora-hunter.onrender.com.

Figures

Figures reproduced from arXiv: 2605.24038 by Chenwaner Zhang, Hantai Zhang, Haoyang Li, Wei Zhou, Wenxin Gu, Zhaoming Wang, Zongyuan Ge.

Figure 1
Figure 1. Figure 1: Representative all-sky camera images from the Tromsø AI dataset (Nanjo et al. 2022) illustrating the four ground-truth label categories. (a) Aurora arc (arc): clear sky with visible aurora (aurora visible = 1). (b) Aurora-cloudy (ac): aurora present but cloud-obscured. (c) Diffuse aurora (diffuse): broad luminous glow. (d) Clear sky, no aurora (clear): true negative (aurora occurring = 0). Stage 1 separate… view at source ↗
Figure 2
Figure 2. Figure 2: AACGM-v2 magnetic coordinate grid at 110 km altitude (2020 January 15, 22:00 UT), generated using the aacgmv2 Python package (Shepherd 2014). Dashed contours show constant MLAT (50◦ –80◦ , 5◦ intervals); colored meridians show MLT (0h, 6h, 12h, 18h). Colored stars mark the three observation stations: Tromsø (red, MLAT 67.2◦ ), Skibotn (blue, 66.7◦ ), and Kiruna (green, 65.0◦ ). The geographic–magnetic long… view at source ↗
Figure 3
Figure 3. Figure 3: Total cloud cover fraction from the Open-Meteo ERA5-based reanalysis archive (Open-Meteo 2024) (2024 January 15, 22:00 UTC), shown on a north polar stereographic projection. Red regions indicate favorable clear-sky conditions; blue regions indicate heavy cloud cover. The spatially varying cloud field drives the location-dependent Stage 2 observing-condition factor in our global visibility maps. Colored sta… view at source ↗
Figure 4
Figure 4. Figure 4: Real all-sky camera observations for Cases A–C described in Section 5.1, illustrating the three fundamental outcome regimes of the cascade framework. (a) Case A: bright discrete aurora under clear skies (2019 October 26, 19:00 UT; Pvis = 0.47). (b) Case B: aurora glow partially obscured by cloud cover (2020 January 22, 21:00 UT; Pvis = 0.25). (c) Case C: overcast with no geomagnetic activity (2020 October … view at source ↗
Figure 5
Figure 5. Figure 5: Test-set precision (blue), recall (red), and F1 score (green) as functions of the probability threshold. Dashed line: F1-optimal threshold (0.267); dotted line: F0.5-optimal threshold (0.444). As the threshold increases, precision improves modestly while recall drops sharply, reflecting the class imbalance (8.4% positive rate in the test set). 5.3. Stage 1: Feature Importance and SHAP Analysis Full feature… view at source ↗
Figure 6
Figure 6. Figure 6: Effect of lunar illumination on aurora observation success rate (Tromsø, aurora-occurring hours 2015–2020). Green bars: favorable conditions (< 50% illumination); orange: transitional; red (> 70%): severely degraded. The dashed vertical line marks the 70% threshold used as a binary feature in Stage 2. At illumination > 70%, observation success drops sharply: the 70–80% bin has 21.7% success, and the >80% a… view at source ↗
Figure 7
Figure 7. Figure 7: Two-layer cascade decomposition (Kp = 3.5, 2024 January 15, 22:00 UTC, Moon = 17%), produced by the authors’ trained cascade model. Left: Stage 1 aurora occurrence probability Pocc, evaluated on an AACGM-v2 magnetic coordinate grid (Shepherd 2014) using OMNI space weather inputs (King & Papitashvili 2005). The nightside auroral oval is clearly visible as a crescent of elevated probability. Center : Stage 2… view at source ↗
Figure 8
Figure 8. Figure 8: Global aurora occurrence probability maps (north polar stereographic projection, 45◦ –90◦N) for three synthetic geomagnetic activity scenarios, computed via the hybrid physics–ML formula (Equation 8) using the authors’ trained Stage 1 XGBoost model with AACGM-v2 magnetic coordinates (Shepherd 2014). Left: Quiet (Kp ≈ 1, Bz = −2 nT). Center : Moderate (Kp ≈ 3.5, Bz = −3.5 nT). Right: G1 Storm (Kp ≈ 6, Bz = … view at source ↗
Figure 9
Figure 9. Figure 9: Data-driven Kp response at the nightside reference point (MLAT = 67◦ , MLT = 0h). Black circles: observed aurora occurrence rates with 95% binomial confidence intervals. Blue triangles: raw XGBoost output. Green: logistic fit (a = 1.96, b = 1.19, c = 0.34). Magenta dashed: power-law fit (exponent = 0.67). Red: old power-law extrapolation (exponent = 1.5). Gray bars: training sample counts (right axis). The… view at source ↗
Figure 10
Figure 10. Figure 10: Differential aurora visibility between Kiruna and Skibotn on 2023 October 20–21. (a) Shared space weather conditions: IMF Bz (olive, shading below zero), Kp (green steps), and Newell coupling (magenta). (b) Cloud cover contrast over the displayed window. The gray band marks the five-hour differential interval (23:00 UTC October 20 to 04:00 UTC October 21) during which Kiruna was nearly clear while Skibotn… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

3 extracted references · 3 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter doi edition editor eprint howpublished institution journal key month number organization pages publisher school series title misctitle type volume year version url label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts ...

    work page

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION format.url url empty "" new.block "" url * "" * if FUNCTION format.eprint eprint empty "" archivePrefix empty "" archivePrefix "arXiv" = new.block " " eprint * " " * new.block " " eprint * " " * if if if FUNCTION format.doi doi empty "" " " doi * " " * if FUNCTION format.pid doi empty eprint empty ur...

    work page

  3. [3]

    Z(3qIW 'Qޟ\9p] @ D a PR CB a PR CB a PR CB a PR CB a PR CB _ 8J jkkS[[Z[[cTJHDx ԱX,&: ٶ `RXaKD

    thebibliography [1] 20pt to REFERENCES 6pt =0pt \@twocolumntrue 12pt -12pt 10pt plus 3pt =0pt =0pt =1pt plus 1pt =0pt =0pt -12pt =13pt plus 1pt =20pt =13pt plus 1pt \@M =10000 =-1.0em =0pt =0pt 0pt =0pt =1.0em @enumiv\@empty 10000 10000 `\.\@m \@noitemerr \@latex@warning Empty `thebibliography' environment \@ifnextchar \@reference \@latexerr Missing key o...

    work page doi:10.1002/2015sw001320 2017