{"id":"b2817570-472e-45af-aa55-e76830023246","arxiv_id":"2411.18070","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A new automated proximity-based framework scores how well solar flare forecasting models' explanations align with active regions, and finds model M2 aligns better than M1.","lead":"The authors propose an automated proximity-based framework that scores how well an AI model's explanation highlights the active regions responsible for solar flares. Applied to two full-disk forecasting models, the framework finds one model's explanations sit closer to active regions, which the paper interprets as better spatial alignment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proximity Score rewards diffuse over-coverage; without a specificity penalty or random baseline, the M1/M2 PS gap is not evidence of explanation quality.","rationale":"The reader correctly identified that the evaluation assumes proximity to active regions is a faithful proxy for explanation quality and that hand-set parameters may drive the conclusions. My stress-test sharpens this into a concrete structural failure mode: because PS is a distance-to-hull average and ACR is a containment fraction, both metrics can be trivially improved by making attribution regions larger and more diffuse, even when the explanation becomes less informative. This is not merely an unvalidated proxy; it is a metric that rewards the opposite of specificity unless a separate penalty or control is introduced. The observed M2 advantage could therefore be an artifact of attribution-map breadth rather than evidence of better alignment. This concern is load-bearing because the paper's headline contribution is an automated quantitative evaluation, and the M1/M2 comparison is the primary demonstration. However, the issue is addressable: adding a specificity term, benchmarking against uniform/blurred baselines, and performing a sensitivity sweep over Table I parameters could salvage the framework. For that reason, I do not move the verdict from the reader's CONDITIONAL; I would require these additions before the quantitative claims are accepted.","tokens_in":7843,"tokens_out":3902,"duration_ms":39432,"concrete_test":"Run PEPS-SF unchanged on a random subset of, say, 200 magnetograms using synthetic attribution maps: (a) a uniform full-disk map scaled to the same maximum intensity as the real GGCAM maps, and (b) a Gaussian-blurred version of each real map (e.g., sigma = 20 pixels). If either baseline attains an average PS at or below M2's 29.4 or an ACR near 0.55, the metric cannot distinguish meaningful explanations from uninformative diffuse activations, and the M1-vs-M2 comparison would need to be reinterpreted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Eq. (4) and (5): PS is the average Euclidean distance from each active region to the closest attribution convex hull, and ACR is the fraction of active regions falling inside bounding regions. Because the metric is purely distance-based, it has no penalty for over-coverage: a large, diffuse attribution region that spans the whole disk will produce dmin = 0 for every AR, giving PS = 0 and ACR = 1. The pipeline actively enlarges regions by adding 5/40-pixel buffers and re-clustering (Fig. 3e-f), making this degenerate behavior easier to reach. Therefore M2's lower average PS (29.4 vs 61.4) may simply reflect broader, less specific attribution maps rather than more accurate localization. The paper reports no measure of attribution spread, no random or uniform-attribution baseline, and no sensitivity analysis for the hand-set Canny thresholds, DBSCAN parameters, or buffer widths. Without such controls, the claim that PEPS-SF 'quantitatively assesses the accuracy and relevance of local explanations' is not supported by the presented evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PEPS-SF, an automated framework for evaluating post hoc explanations in full-disk solar flare forecasting. The framework derives attribution maps with Guided Grad-CAM, processes them through edge detection, density-based clustering, convex hull extraction, buffering, and re-clustering, and then compares the resulting regions to NOAA active-region locations using two new metrics: Proximity Score (Eq. 4) and Attribution Colocation Ratio (Eq. 5). The method is applied to two previously published CNN models, M1 and M2, over a dataset of 5,923 full-disk magnetograms from 2020 to 2023. The paper reports that M2 has a lower average Proximity Score (29.4 vs. 61.4 pixels) while both models have similar ACR values (about 0.55), and it interprets this as evidence that M2's explanations are more accurately aligned with active regions. The contribution is framed as a fully automated, quantitative evaluation of explanation quality for operational use.","tokens_in":8040,"tokens_out":3819,"duration_ms":34501,"significance":"If the proximity metrics are valid measures of explanation quality, the framework would be a useful step toward automated interpretability evaluation in a high-stakes operational domain. The paper has real strengths: an explicitly specified algorithm, a large evaluation dataset, a concrete comparison between two models, and an open-source code repository. The fundamental validity of the metric, however, is not yet established. The paper provides no random or uniformity baseline, no validation against human judgments or alternative attribution methods, and no sensitivity analysis for the hand-set parameters. Because the Proximity Score has no specificity or coverage penalty, a trivial diffuse attribution map can achieve a perfect score, so the reported M1/M2 difference may reflect attribution breadth rather than localization quality. These issues are fixable with additional controls and analysis, but they currently prevent the central claim from being supported.","major_comments":[{"comment":"The Proximity Score is the average Euclidean distance from each active region to the closest attribution convex hull, with no penalty for attribution regions that cover large portions of the solar disk. A spatially diffuse attribution map that covers the full disk will have dmin = 0 for every AR, yielding PS = 0 and ACR = 1. The preprocessing pipeline actively enlarges regions through eastward/westward buffers and re-clustering (Fig. 3e-f), so the smaller average PS of M2 in Table II (29.4 vs. 61.4) is indistinguishable from M2 having broader, less specific attribution maps. The paper reports no measure of attribution spread or coverage, so the claim that lower PS reflects more accurate localization is not established.","section":"Section II.B, Eqs. (4) and (5)"},{"comment":"The central assumption that proximity of attribution hulls to NOAA active-region centroids is a valid proxy for explanation quality is introduced without validation. The manuscript gives no random or uniform-attribution baseline, no comparison with human judgments (despite the prior human-in-the-loop evaluation in [6]), and no comparison with alternative attribution methods such as Integrated Gradients or DeepLIFT. A minimal control would be to run the same PEPS-SF pipeline on random or uniform attribution maps and on a second attribution method; if those baselines produce similar PS/ACR values to the real models, the metric would not be informative about explanation quality. Without such controls, the abstract's claim that the framework 'quantitatively assesses the accuracy and relevance' of explanations is unsupported.","section":"Section II.B and Table I"},{"comment":"The conclusion that M2 is 'more accurate and consistent' than M1 is based on raw mean and standard deviation values, but no statistical tests or confidence intervals are provided. The standard deviations are very large relative to the mean differences (e.g., FN: 72.49 vs. 38.02 with SDs 71.89 and 46.96; TP: 59.43 vs. 52.03 with SDs 68.05 and 48.68), so the reported differences may not be statistically significant. Since the four contingency categories are not independent across images, a paired bootstrap or permutation test over the 5,923 images is needed to support the comparative claims.","section":"Section IV.A, Table III"},{"comment":"The six hand-set parameters (Canny thresholds 30 and 50, DBSCAN min_samples 2 and max_dist 10, eastward buffer 5, and westward buffer 40) are used without any sensitivity analysis. Because these parameters control the merging and enlargement of attribution regions, the relative ordering of M1 and M2 in Table II could change over plausible parameter ranges. The authors should vary each parameter over a reasonable grid and report whether the qualitative comparison is stable. Without this analysis, the reported M1/M2 difference cannot be distinguished from an artifact of the particular parameter values chosen.","section":"Section III, Table I"}],"minor_comments":[{"comment":"The wording 'quantitatively assesses the accuracy and relevance of local explanations' overstates what the geometric metrics measure; consider replacing 'accuracy and relevance' with 'spatial alignment' until the proxy is validated.","section":"Abstract and Section II.B"},{"comment":"The caption contains the phrase 'An schematic overview'; it should read 'A schematic overview'.","section":"Figure 2 caption"},{"comment":"The sentence 'In Step (a) , the Guided-GradCAM image is resized...' contains an awkward space before the comma and inconsistent naming (Guided-GradCAM vs. GGCAM); please standardize the terminology.","section":"Section II.A"},{"comment":"Equation (5) defines ACR as a percentage by multiplying by 100, but Table II reports values such as 0.552943 without a percent sign; clarify the units and formatting consistently.","section":"Eq. (5) and Table II"},{"comment":"The text states that M2 'outperforms M1 in false negatives, false positives, and true negatives,' but Table III shows FP means of 52.95 (M1) and 55.63 (M2), so M1 has the lower FP score; the sentence should be corrected to match the table.","section":"Section IV.A"},{"comment":"The text states that M1 'generally exhibits higher mean ACRs across most categories,' but Table IV shows FN ACR of 0.514 for M1 and 0.521 for M2, so the claim needs qualification or correction.","section":"Section IV.B"}],"recommendation":"major_revision","confidential_remarks":"The paper is a case study of two models from the same research group, and the open-source code and large dataset are genuine assets. The central methodological gap—absence of baselines, sensitivity analysis, and validation of the proximity proxy—is fixable within the scope of the manuscript, so I recommend major revision rather than rejection. I would also ask the authors to make the statistical analysis more rigorous, as the large standard deviations in Tables III and IV substantially weaken the comparative claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The genuinely new thing is PEPS-SF: an automated pipeline that takes GGCAM attribution maps, runs Canny edge detection, DBSCAN clustering, convex hulls, buffers, and then measures each active region's distance to the nearest hull (PS) and the fraction of ARs inside the buffered regions (ACR). It is applied to 5,923 full-disk magnetograms and two CNN models. Code is in a public repo. That is a useful, reproducible contribution to the space weather explainability field, which really does lack automated evaluation of explanations.\n\nThe weaknesses are not in the implementation but in the interpretation. The metrics have no baseline. A uniform attribution map covering the whole disk would give PS = 0 and ACR = 1, and the pipeline's buffers and re-clustering only push it in that direction. The paper reports no random or uniform-attribution baseline, no measure of attribution spread, and no sensitivity analysis for the hand-set parameters (Canny thresholds 30/50, DBSCAN min_samples 2, max_dist 10, buffers 5/40). So the headline finding — M2's PS of 29.4 vs M1's 61.4 — could simply mean M2's attribution maps are more diffuse, not better localized. The paper also reports means and standard deviations but no statistical test for the M1 vs M2 difference. Given those gaps, the claim that PS/ACR 'quantitatively assess the accuracy and relevance of local explanations' goes beyond what the evidence shows. What they measure is spatial proximity to NOAA active regions, which is a reasonable proxy for operational usefulness but not validated as a true measure of explanation quality.\n\nThe paper is still worth a serious referee. The pipeline is clean, the domain gap is real, and the missing controls are very addressable: add a uniform/random baseline, a specificity penalty or region-size report, and a sensitivity sweep over the preprocessing parameters. With those, the M1/M2 comparison could be meaningful. As it stands, the conclusions are conditional.","headline":"A useful automated pipeline for checking explanation alignment to active regions, but the proximity metrics lack baselines and sensitivity analysis, so the M1/M2 comparison is not yet evidence of explanation quality.","tokens_in":8572,"tokens_out":2551,"would_cite":false,"duration_ms":22113,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces an automated proximity-based framework, PEPS-SF, for scoring whether post hoc explanations of full-disk solar flare models point near active regions, and reports that one of two compared models sits substantially…","keywords":["solar flare prediction","explainable AI","Guided Grad-CAM","attribution maps","proximity analysis","full-disk magnetograms","deep learning interpretability","space weather forecasting"],"falsifier":"Replace the real Guided Grad-CAM attribution maps with randomly placed hulls of matched size and run the same PEPS-SF pipeline; if the randomized hulls achieve an average Proximity Score at or below M2's 29.4 pixels or an Attribution Colocation Ratio near 0.55, then the metrics are rewarding diffuse coverage rather than explanation fidelity.","tokens_in":7597,"feed_emoji":"☀️","tokens_out":6538,"duration_ms":55095,"temperature":0.7,"pith_summary":"This paper proposes PEPS-SF, a fully automated pipeline for checking whether the explanations produced by full-disk solar flare forecasting models point at the solar regions that actually produce flares. It applies the pipeline to two convolutional models, M1 and M2, that predict M-class and larger flares from line-of-sight magnetograms within a 24-hour window. Using Guided Grad-CAM attribution maps, the pipeline detects edges, clusters them, builds buffered convex hulls, and measures how far each NOAA active region lies from the nearest hull, producing a Proximity Score and an Attribution Colocation Ratio. The central finding is that M2's explanations are substantially closer to active regions, with an average Proximity Score of 29.4 pixels versus M1's 61.4, while both models colocate with active regions at nearly the same rate of about 55 percent. The paper argues that this kind of automated, near-real-time scoring can make post hoc explanations auditable in operational space weather forecasting.","feed_headline":"Flare AI explanation metric finds one model twice as close","feed_subtitle":"Automated proximity scores check whether solar-flare AI explanations align with active regions, without human raters.","key_machinery":"The load-bearing object is PEPS-SF, a proximity-based evaluation pipeline defined in Algorithm 1. It takes a magnetogram and the coordinates of active regions and flare locations, generates a Guided Grad-CAM attribution map, scales intensities to 0-255, applies Canny edge detection with thresholds 30 and 50, clusters edge pixels with DBSCAN using a minimum of 2 samples and a maximum distance of 10 pixels, derives convex hulls, pads them with a 5-pixel eastward and a 40-pixel westward buffer to account for solar rotation over the 24-hour window, merges nearby hulls, masks to the solar disk, and converts heliographic coordinates to pixel coordinates. The two output metrics are the Proximity Score, the mean minimum Euclidean distance from each active region to the nearest buffered hull, and the Attribution Colocation Ratio, the fraction of active regions inside those bounding regions. These metrics carry the argument because the entire comparison of M1 and M2 reduces to the numbers they produce.","core_discovery":"The paper's central claim is that the reliability of post hoc explanations in solar flare forecasting can be scored automatically once the regions of interest, namely NOAA active regions and flare locations, are known. To that end it defines two metrics. The Proximity Score is the average Euclidean distance between each active region and the closest buffered convex hull built from the model's Guided Grad-CAM attribution map; a lower score means the explanation highlights pixels nearer the physically relevant region. The Attribution Colocation Ratio is the percentage of active regions that fall inside those buffered bounding regions. Applied to 5,923 full-disk magnetograms from 2020 to 2023, the framework reports that M2 is closer on average (29.4 versus 61.4 pixels) and more consistent across the true-positive, false-positive, true-negative, and false-negative categories, while both models place only about 55 percent of active regions inside their attribution hulls. The conclusion drawn is that one model's explanations align better with active-region structure and that the framework is a step toward operational, automated evaluation of explanation fidelity.","pith_inferences":["A check the paper leaves implicit: replacing active-region centroids with the actual NOAA flare coordinates in Equation (4) could re-rank the models, since the stated fidelity target is the active region rather than the flare itself.","The reported ordering is meaningful only if it is stable under preprocessing choices; a grid search over Canny thresholds, DBSCAN parameters, and buffer widths would show whether M2's 32-pixel advantage is robust or an artifact of a few parameter values.","The same pipeline could be run on other attribution methods, such as Integrated Gradients or DeepLIFT, to rank explanation algorithms rather than models, turning Proximity Score and Attribution Colocation Ratio into general-purpose selection criteria.","Without a baseline of randomized attribution hulls matched in size and location distribution, a low Proximity Score might reflect diffuse or centrally concentrated explanations rather than true alignment; adding such a baseline would establish significance."],"forward_implications":["A forecasting model's explanations can be graded without human raters whenever known active-region locations are available, making routine audits of deployed flare models possible.","M2's lower average Proximity Score (29.4 versus 61.4 pixels) is evidence that its Guided Grad-CAM explanations concentrate closer to active regions than M1's explanations do.","An Attribution Colocation Ratio near 0.55 for both models means roughly 45 percent of active regions are not covered by the attribution hulls, so neither model's explanations fully account for all flaring regions in a given image.","Because the Proximity Score is continuous and the Attribution Colocation Ratio is binary, they answer different audit questions: how far off is the explanation, and does it cover the region at all.","Applied per prediction category, the framework can flag systematic failures such as false negatives whose explanations point far from the actual active regions."],"supporting_citations":[{"why":"Supplies the Grad-CAM localization method that Guided Grad-CAM builds on, and is therefore the attribution-generation method for every explanation scored in the paper.","marker":"[7]"},{"why":"Supplies guided backpropagation, the pixel-level component that Guided Grad-CAM combines with Grad-CAM to produce the attribution maps.","marker":"[8]"},{"why":"Provides model M1, the full-disk explainable solar flare prediction model whose explanations are evaluated here, and whose earlier human-in-the-loop study is extended into a fully automated framework.","marker":"[6]"},{"why":"Provides model M2, the second full-disk model compared in the study, and the near-limb prediction setup that M2's evaluation is based on.","marker":"[18]"},{"why":"Establishes active regions as the primary physical initiators of solar flares, which is the justification for treating NOAA active-region locations as the regions of interest in the proximity analysis.","marker":"[3]"}],"fun_headline_variants":["Proximity metric ranks solar flare AI explanations","One flare AI model's explanations twice as close to active regions","Automated proximity scoring checks solar flare AI interpretability","New framework scores AI explanation fidelity for solar flare prediction","Flare AI: proximity analysis reveals one model more aligned"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise, stated in Section II.B and partly conceded in Section V's note on preprocessing noise, is that an explanation counts as good exactly when its highlighted hulls sit near NOAA active-region centroids, and that the hand-set Canny, DBSCAN, and buffer parameters do not alter the model ranking; if either condition fails, the M1-versus-M2 comparison does not measure explanation quality.","fun_headline_variants_meta":{"raw":{"variants":["Proximity metric ranks solar flare AI explanations","One flare AI model's explanations twice as close to active regions","Automated proximity scoring checks solar flare AI interpretability","New framework scores AI explanation fidelity for solar flare prediction","Flare AI: proximity analysis reveals one model more aligned"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1416,"prompt_tokens":986,"completion_tokens":430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":352}},"tokens_in":602,"tokens_out":430,"duration_ms":4605,"temperature":1.0,"reasoning_tokens":352,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:32:25.766890+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the real Guided Grad-CAM attribution maps with randomly placed hulls of matched size and run the same PEPS-SF pipeline; if the randomized hulls achieve an average Proximity Score at or below M2's 29.4 pixels or an Attribution Colocation Ratio near 0.55, then the metrics are rewarding diffuse coverage rather than explanation fidelity.","supporting_citations":[{"cited_title":"Grad-cam: Visual explanations from deep networks via gradient-based localization,","cited_arxiv_id":null,"evidence_quote":"Supplies the Grad-CAM localization method that Guided Grad-CAM builds on, and is therefore the attribution-generation method for every explanation scored in the paper."},{"cited_title":"Striving for simplicity: The all convolutional net,","cited_arxiv_id":null,"evidence_quote":"Supplies guided backpropagation, the pixel-level component that Guided Grad-CAM combines with Grad-CAM to produce the attribution maps."},{"cited_title":"Ex- ploring deep learning for full-disk solar flare prediction with empirical insights from guided grad-cam explanations,","cited_arxiv_id":null,"evidence_quote":"Provides model M1, the full-disk explainable solar flare prediction model whose explanations are evaluated here, and whose earlier human-in-the-loop study is extended into a fully automated framework."},{"cited_title":"Unveiling the potential of deep learning models for solar flare prediction in near-limb regions,","cited_arxiv_id":null,"evidence_quote":"Provides model M2, the second full-disk model compared in the study, and the near-limb prediction setup that M2's evaluation is based on."},{"cited_title":"Flare-productive active regions,","cited_arxiv_id":null,"evidence_quote":"Establishes active regions as the primary physical initiators of solar flares, which is the justification for treating NOAA active-region locations as the regions of interest in the proximity analysis."}],"review_version":1}