{"id":"caa4d7ff-6f85-4e43-b76d-b517360f5872","arxiv_id":"2605.24038","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"A cascade of XGBoost for aurora occurrence and logistic regression for clear conditions produces visibility probabilities with ROC-AUC 0.937 on Tromso test data and 0.905 on independent Kiruna data, beating a single-stage baseline by 0.087.","lead":"Aurora Hunter is a two-stage ML model that first predicts the chance an aurora is occurring using solar wind features, then predicts the chance of clear skies given occurrence using cloud and moon data, and multiplies the two for a visibility probability. A generalist might read it because it offers a practical way to improve forecasts for aurora tourism and space weather alerts by separating physical drivers from local viewing conditions.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Occurrence labels from all-sky images confound physical occurrence with observability","rationale":"The reader's weakest assumption correctly flags the product formula but mis-locates the risk: the equality P(visible) = P(occurring) * P(clear|occurring) is definitionally true. The load-bearing issue is upstream, in whether the occurrence labels are uncontaminated by the clear-sky requirement. This is visible directly from the abstract's description of the label source and does not require the full text. The empirical AUC numbers could still hold as a black-box predictor, but the central modeling claim of a cleanly decoupled two-stage framework would be weakened.","tokens_in":1802,"tokens_out":422,"duration_ms":40337,"concrete_test":"Re-label a random 20% subset of the Tromso training hours (2015-2023) using an independent non-optical proxy (e.g., Kp > 4 or AE index threshold) as the occurrence target, retrain only Stage 1 while keeping Stage 2 unchanged, and recompute cascade ROC-AUC on the 2019-2020 Tromso hold-out; a drop >0.05 would indicate the original image-derived labels were inflating performance via confounding.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Stage 1 is trained to predict P(occurring) using labels from the Tromso AI all-sky image classifier. All-sky images can only register aurora when skies are clear enough for detection, so these labels actually encode 'occurring AND sufficiently clear.' Stage 2 is then trained only on those same 'occurring' hours to predict P(clear|occurring). Because the first-stage target already embeds information about the second factor, the claimed decoupling is compromised; the product P(occurring) * P(clear|occurring) therefore does not isolate the two physical drivers as asserted. The reported ROC-AUC gains on Tromso and Kiruna test sets may reflect this label contamination rather than genuine separation of solar-wind physics from cloud/lunar effects.","agreement_with_reader":"disagree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces Aurora Hunter, a two-stage probabilistic cascade for aurora visibility forecasting. Stage 1 uses XGBoost on 51 solar-wind and geomagnetic features to predict P(occurring) from Tromsø AI all-sky image labels (∼16,600 hourly samples, 2015–2023). Stage 2 uses logistic regression on 21 cloud/lunar features to predict P(clear|occurring) only on aurora-occurring hours. The product P(visible) = P(occurring) × P(clear|occurring) is reported to reach ROC-AUC 0.937 on the Tromsø 2019–2020 test set and 0.905 on independent Kiruna 2024 data, outperforming a single-stage baseline by +0.087; cross-site generalization is checked on Skibotn data and SHAP analysis highlights Kp × nightside, MLT, and oval distance as key drivers.","tokens_in":1988,"tokens_out":538,"duration_ms":19898,"significance":"If the claimed separation of physical occurrence from local observing conditions holds, the cascade offers improved interpretability over black-box single-stage models and supports site-specific forecasting for research and tourism. Strengths include temporal hold-out splits, an independent geographic test site, and explicit SHAP attribution; these elements allow falsifiable evaluation of the decoupling hypothesis.","major_comments":[{"comment":"Abstract and §2 (label construction): the Tromsø AI all-sky image classifier labels used for Stage 1 can only register aurora when skies are sufficiently clear for detection. Consequently the Stage-1 target encodes both physical occurrence and observability, violating the conditional-independence assumption required for the product formula to isolate the two physical drivers. This label contamination directly undermines the central claim that the cascade decouples solar-wind physics from cloud/lunar effects; the reported AUC gains may therefore reflect the shared information rather than genuine separation.","section":"Abstract and §2"}],"minor_comments":[{"comment":"No error bars, confidence intervals, or hyperparameter-search protocol are reported for the AUC values or the logistic-regression coefficients; these should be added to allow assessment of statistical significance of the +0.087 improvement.","section":"Results"},{"comment":"The manuscript does not state whether the product formula was validated against joint (occurrence AND clear) labels on a held-out set; such a calibration check would strengthen the modeling claim.","section":"Methods"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive review and for raising this important point about label construction. We respond to the major comment below.","responses":[{"response":"We agree that the Tromsø AI all-sky image labels used as targets for Stage 1 are positive only when aurora is both physically occurring and sufficiently clear for the classifier to register it in the images. This introduces some dependence between the Stage 1 target and local observing conditions, which means the strict conditional-independence assumption required for the product formula to fully isolate the two drivers does not hold perfectly. The manuscript's central claim of decoupling is therefore qualified by this label property, and the reported AUC improvement may partly reflect shared information rather than pure separation. We will revise the abstract and §2 to explicitly acknowledge this limitation in label construction, discuss its implications for the product formula, and moderate the language around complete decoupling of solar-wind physics from cloud/lunar effects while retaining the empirical performance and interpretability results.","revision_made":"yes","referee_comment":"[Abstract and §2] Abstract and §2 (label construction): the Tromsø AI all-sky image classifier labels used for Stage 1 can only register aurora when skies are sufficiently clear for detection. Consequently the Stage-1 target encodes both physical occurrence and observability, violating the conditional-independence assumption required for the product formula to isolate the two physical drivers. This label contamination directly undermines the central claim that the cascade decouples solar-wind physics from cloud/lunar effects; the reported AUC gains may therefore reflect the shared information rather than genuine separation."}],"tokens_in":1494,"tokens_out":346,"duration_ms":37278,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's core idea is a product of two probabilities: P(occurring) from XGBoost on 51 solar-wind and magnetosphere features, times P(clear|occurring) from logistic regression on 21 cloud and lunar features. It gets ROC-AUC 0.937 on the Tromso 2019-2020 hold-out and 0.905 on the independent Kiruna 2024 set, beating a single-stage baseline by 0.087. Cross-site Skibotn data and SHAP rankings for Kp-nightside interaction and oval distance are also shown.\n\nWhat works is the operational framing for aurora tourism and the use of a real independent geographic test set rather than just temporal splits. The prototype link suggests they actually built something usable.\n\nThe main soft spot is the labeling. Occurrence labels come from the Tromso AI all-sky image classifier. Those images only flag aurora when skies are already clear enough to see it, so the stage-1 target is effectively 'occurring and observable,' not pure physical occurrence. Training stage 2 only on those same hours then multiplies two quantities that are not cleanly separated. The reported gains could partly reflect that contamination instead of genuine isolation of the two drivers. No error bars, no joint-label validation of the product formula, and no discussion of this dependence appear in the abstract.\n\nThis is for people building site-specific visibility forecasts in space weather or similar environmental ML tasks. A reader who needs the exact numbers or the feature list could extract value, but anyone expecting a robust separation of physical drivers should treat the central claim with caution.\n\nIt is worth sending to peer review so the label issue can be checked against the full methods and any additional ground-truth data.","headline":"The two-stage cascade reports solid AUC gains on held-out and independent sites, but all-sky image labels for occurrence already embed clear-sky information, so the claimed decoupling of physics from local conditions is likely compromised.","tokens_in":2507,"tokens_out":443,"would_cite":false,"duration_ms":21801,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A two-stage cascade predicts aurora visibility by multiplying occurrence probability with clear-observation probability, reaching ROC-AUC 0.937.","keywords":["aurora visibility","probabilistic forecasting","two-stage cascade","XGBoost","space weather","cloud cover","lunar illumination","ROC-AUC"],"falsifier":"On a new set of hourly site observations, a single model trained on the combined feature set achieving higher ROC-AUC than the cascade product would falsify the claimed benefit of decoupling.","tokens_in":2710,"feed_emoji":"🌌","tokens_out":715,"duration_ms":41250,"temperature":0.7,"pith_summary":"Aurora visibility at a site depends on whether the aurora is physically occurring due to solar wind activity and whether clouds or moonlight allow naked-eye detection. The paper decouples these into two stages: one model predicts occurrence using physics features, and a second predicts clear conditions using cloud and lunar features. Their product gives the overall visibility probability. This separation improves performance over training a single model on all features at once. The result supports better planning for space weather studies and aurora viewing.","feed_headline":"Two-stage cascade forecasts aurora visibility at 0.937 ROC-AUC","feed_subtitle":"Separating occurrence from cloud and moonlight effects improves single-stage baseline by 0.087 on test and independent sites.","key_machinery":"The cascade formula P(visible) = P(occurring) × P(clear|occurring) that trains separate models on occurrence features versus observational-condition features.","core_discovery":"Aurora Hunter predicts P(visible) as P(occurring) times P(clear|occurring). Stage 1 uses XGBoost on 51 solar-wind features trained on joint Tromso-Kiruna hourly samples with image-classifier labels. Stage 2 uses logistic regression on 21 cloud and lunar features trained only on occurring hours. The cascade reaches ROC-AUC 0.937 on Tromso 2019-2020 test data and 0.905 on independent Kiruna 2024 data, improving the single-stage baseline by 0.087, with generalization shown on held-out Skibotn data.","pith_inferences":["The same separation of event probability from detection probability could apply to visibility forecasts for other transient phenomena such as meteor showers.","Separate stages allow one model to be updated with new solar data without retraining the cloud model when weather data changes.","The cascade output could feed into decision tools that weigh visibility probability against travel costs for observers.","Adding real-time satellite cloud imagery to the second stage would test whether the current 21 features are sufficient for the clear-observation task."],"forward_implications":["Reaches ROC-AUC 0.937 on Tromso test data from 2019-2020","Reaches ROC-AUC 0.905 on independent Kiruna data from 2024","Generalizes to held-out Skibotn data from 2022-2025","SHAP analysis identifies Kp x nightside interaction, MLT position, and auroral oval distance as dominant predictors accounting for 39 percent combined importance"],"fun_headline_variants":["Two-stage model multiplies aurora occurrence and clear observation probs","Aurora Hunter cascade hits 0.937 ROC-AUC on Tromso 2019-2020 test","Separating occurrence from clear-sky conditions boosts AUC by 0.087","XGBoost stage one predicts aurora occurrence from 51 solar wind features","Logistic regression stage two models clear observation given occurrence"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The product of the two stage outputs equals the true joint visibility probability, which holds only if the occurrence and clear stages are conditionally independent or any dependence is captured by the separate feature sets.","fun_headline_variants_meta":{"raw":{"variants":["Two-stage model multiplies aurora occurrence and clear observation probs","Aurora Hunter cascade hits 0.937 ROC-AUC on Tromso 2019-2020 test","Separating occurrence from clear-sky conditions boosts AUC by 0.087","XGBoost stage one predicts aurora occurrence from 51 solar wind features","Logistic regression stage two models clear observation given occurrence"]},"model":"grok-4.3","cost_usd":0.004597,"raw_usage":{"total_tokens":2331,"prompt_tokens":769,"num_sources_used":0,"completion_tokens":95,"cost_in_usd_ticks":45974500,"prompt_tokens_details":{"text_tokens":769,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1467,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":769,"tokens_out":95,"duration_ms":19513,"temperature":1.0,"reasoning_tokens":1467,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T16:20:52.134338+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"On a new set of hourly site observations, a single model trained on the combined feature set achieving higher ROC-AUC than the cascade product would falsify the claimed benefit of decoupling.","supporting_citations":[],"review_version":1}