{"id":"5a0041d3-de6d-4b24-b19b-54f8659f9401","arxiv_id":"2412.08589","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A YOLO-based detector, SPACE-SUIT, finds sunspots, plages, filaments, and off-limb structures in SUIT Mg II k images with a validation MAP of 0.874 on mock data.","lead":"SPACE-SUIT is an AI tool that detects and classifies solar features, such as sunspots, plages, filaments, and off-limb structures, in images from the SUIT telescope on India's Aditya-L1 mission. It was trained on mock images made from IRIS data and reports good detection scores on those mock images, but its performance on real SUIT data is only checked visually.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation metrics may be inflated by train/validation leakage: Section 2.2 augments ~217 source images into 3056 training images but never states the 123 FITS validation images come from disjoint IRIS mosaics, so the headline 0.788/0.863/0.874 numbers may reflect near-duplicate memorization.","rationale":"The paper's contribution is a YOLOv8-obb detector trained on synthesized SUIT-like images; the only quantitative evidence for the central claim is Table 3 on mock validation FITS. The most load-bearing condition is that those validation images genuinely test generalization. That condition is not established: the dataset construction section describes augmentation of ~217 original images into 3056 training images and lists a separate validation FITS set, but never documents a source-level split. The presence of both PSF-convolved and non-PSF-convolved versions of the same originals makes same-source leakage especially easy. I do not allege the authors deliberately leaked; the absence of documentation is enough to make the headline metrics unverifiable. The reader flagged this in the rationale but made the transfer assumption the weakest point; I would put leakage first because it attacks the only quantitative headline and can be settled by a manifest check. The real-data transfer issue is real and the authors admit it, so it is less load-bearing for the stated numbers. Hence the reader's CONDITIONAL verdict is appropriate; no verdict change, but the condition should explicitly include source-disjoint split verification and data/code release.","tokens_in":19374,"tokens_out":4377,"duration_ms":44278,"concrete_test":"Request the dataset manifest and training/validation assignment; group images by source IRIS mosaic observation date. Retrain YOLOv8-obb under a strict temporal split (e.g., all mosaics after a cutoff are validation only) and recompute Table 3. Additionally, compute pairwise normalized cross-correlation or perceptual hashes between each of the 123 validation FITS images and all 3056 training images; report the maximum similarity and drop any near-duplicate pairs that cross the split. If the headline metrics decrease materially or if near-duplicate pairs are found, the claim must be restated as mock-data performance on a source-disjoint split and the real-SUIT deployment claims need quantitative support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2/Table 1: the pre-augmentation set is 217 images, the augmented training set is 3056, and the validation FITS set is 123; the paper says the original ~200 images 'contain both psf-convolved and non psf-convolved images.' No statement anywhere says the 123 validation FITS images come from IRIS mosaics that are disjoint from the training mosaics, nor that the source-level split is temporal or by observation. The training pipeline converts images to lossless JPG and applies flips, HSV, gray, gamma; these are exactly the transformations that make near-duplicates hard to detect visually while leaving the detector with a shortcut. If any validation image is an augmented version of a training source (or the PSF-convolved/non-PSF-convolved pair of the same mosaic straddles the split), the reported precision 0.788 / recall 0.863 / MAP 0.874 in Table 3 measure memorization, not generalization. The self-validation histograms (Figs. 7-8) compare ground-truth and predicted boxes on the same split and cannot detect this. The transfer failure on real SUIT (rescaling factor 8, Section 5.1.2) is consistent with the model having learned dataset-specific intensity statistics rather than robust feature geometry, which makes the leakage question more acute.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SPACE-SUIT, a YOLOv8-obb-based object detector for identifying plages, sunspots, filaments, and off-limb structures in SUIT Mg II k (NB3) images. Because few real SUIT images are available, the authors construct mock-SUIT images from IRIS Mg II k full-disk mosaics by convolving with the SUIT NB3 effective area and PSF, manually label bounding boxes, and train three YOLO variants (v8, v8-obb, v9). The v8-obb model is selected as best, with reported validation metrics on mock FITS data of precision 0.788, recall 0.863, and MAP@0.5 0.874 (Table 3). The model is then applied to real SUIT Level-1 images using an ad hoc rescaling factor of 8, with qualitative comparisons to AIA 1700 Å sunspots. The paper also introduces a 'self-validation' scheme based on statistical moments and Tamura texture features computed inside detected and ground-truth bounding boxes, arguing that consistency between these distributions supports the detections on real data. The paper concludes that entropy best discriminates the four feature classes and that the method demonstrates 'zero-shot' capability on SUIT data, while acknowledging that fine-tuning on real SUIT data would be needed.","tokens_in":19605,"tokens_out":3866,"duration_ms":43021,"significance":"If the mock-data metrics are trustworthy, this is a practically useful contribution: it provides a ready-to-use detector for a new solar mission (SUIT) and demonstrates a transfer pipeline from IRIS mosaics to SUIT-like images. The comparative evaluation of YOLO v8/v8-obb/v9 and the public description of the mock-image generation are useful for future instrument-specific detectors. The self-validation idea using statistical and Tamura features is interesting as a qualitative diagnostic when no ground truth exists, though it is not a substitute for an independent test set. The main weakness is that the central quantitative claim (the Table 3 numbers) depends on a train/validation split that is not described, and the real-SUIT validation is only qualitative and involves a manually tuned rescaling. These issues are fixable, so the work is potentially acceptable after substantial revision.","major_comments":[{"comment":"The manuscript never describes how the 217 pre-augmentation images were split into training and validation sets. Because the training set is augmented to 3056 images and the validation set includes 123 FITS images, it is unknown whether any validation image is an augmented or PSF-convolved variant of a training mosaic. The headline metrics in Table 3 (precision 0.788, recall 0.863, MAP 0.874) are only meaningful if the validation images come from IRIS mosaics that are disjoint from all training mosaics. The authors must specify the source-level split (e.g., temporal, by observation date, or by random mosaic assignment) and ideally report performance on a held-out set of IRIS observations that were never used in training.","section":"Section 2.2, Table 1"},{"comment":"The real-SUIT evaluation is not quantitative and the preprocessing is tuned on the same images: the rescaling factor of 8 is chosen by 'trial and error' on the SUIT images themselves (Section 5.1.2). The later statement in Section 6 that the distributions give 'confidence in the performance of our model on unseen data' and support 'zero-shot learning' is therefore not supported. The authors should either provide a quantitative comparison of sunspot detections against AIA 1700 Å (e.g., overlap, centroid error, or detection rate) using a fixed, pre-registered preprocessing rule, or explicitly restrict the claims to a qualitative demonstration and present the rescaling factor as a limitation requiring fine-tuning.","section":"Section 5.1.2, Figure 6"},{"comment":"The self-validation scheme computes distributions of statistical and Tamura features on the same training and validation splits used to train the model. This procedure can only show that detected boxes resemble typical training boxes; it cannot detect missed detections, localization errors, or biases that preserve texture statistics. The paper partially acknowledges this in Section 6 ('It does not... provide a measure of the fraction of all regions captured correctly'), but the abstract and the phrase 'validated with the observed SUIT images' overstate the evidence. The authors should soften these claims and make clear that the self-validation is a consistency check, not an independent validation.","section":"Sections 4, 5.2, and 6"},{"comment":"The gamma augmentation is selected by 'visual inspection' with only the 'best looking' values used, and the same subsection states that the full range of gamma is not used. This introduces a manually chosen, data-dependent free parameter. The authors should either justify this selection more rigorously (e.g., by a sensitivity analysis of model performance versus gamma range) or otherwise treat it as a hyperparameter that is reported in the final model configuration.","section":"Section 2.2, Eq. (1)"}],"minor_comments":[{"comment":"The text says the data are saved as 'lossless JPG images,' but JPEG is a lossy format. This is also inconsistent with the later statement in Section 5.1.1 that JPG data are 'lossy' and quantized. Please reword to 'high-quality JPEG' or use a lossless format such as PNG.","section":"Section 2.2"},{"comment":"The text says the original images are 'approximately 200' while Table 1 lists 217 pre-augmentation images; please harmonize these numbers.","section":"Section 2.2, Table 1"},{"comment":"The manuscript refers to 'Level 1' data in the abstract and Table 1 but mentions 'level 1.1 SUIT data' in Section 5.1.2; please clarify which data level is used and whether the calibration steps differ.","section":"Section 5.1.2 and Section 2"},{"comment":"The captions say the rows correspond to the listed measures, but the text in Section 5.2 refers to 'first row,' 'second row,' etc.; consider numbering the subplots or explicitly referring to panels to improve readability.","section":"Figures 7 and 8"},{"comment":"The row 'Test (SUIT Data)' shows 'NA' for filaments; please explain why no filaments were detected in the real SUIT images, as this is relevant to the claim of detecting all four feature classes.","section":"Table 4"},{"comment":"The reference for Schuh and Angryk (2014) contains '???' in the venue field; please provide the full bibliographic information.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of Solar Physics and addresses a timely need for automated feature extraction for SUIT. The main concern is the unspecified train/validation split, which directly affects the credibility of the headline numbers; if the authors can demonstrate a clean source-level split, the quantitative claim becomes much stronger. The real-SUIT evaluation is currently a qualitative demonstration rather than a validation, and the 'self-validation' terminology should be toned down. I would not recommend rejection because the core methodology and dataset construction are sound as far as described, and the identified issues are addressable in revision. However, the authors should be pushed to provide the split details and either improve the real-data evaluation or clearly label it as exploratory."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid engineering paper for a real instrument, but the headline accuracy numbers are not yet trustworthy for real SUIT data, and the paper's own caveats (manual rescaling, fine-tuning needed) support that.\n\nWhat's genuinely useful: the mock-SUIT pipeline (folding IRIS Mg II k mosaics through SUIT's effective area and PSF, inpainting stitching artifacts, rebinning to SUIT plate scale) is a sensible way to bootstrap training before enough real data exists. The YOLOv8-obb comparison and the decision to use rotated boxes for off-limb and filament structures is reasonable. The authors are also honest about the gap between mock and real data; Section 5.1.2 shows they needed a manually chosen rescaling factor of 8, and Section 6 states the model would need fine-tuning on real SUIT data.\n\nThe soft spots are mostly about validation. The mock-FITS metrics (P=0.788, R=0.863, MAP=0.874) could be inflated by train/validation leakage: the paper says ~217 pre-augmentation images were augmented into 3056 training images, but it never states that the 123 FITS validation images come from IRIS mosaics disjoint from training at the source-image level. With PSF-convolved and non-PSF-convolved versions of the same mosaic possibly straddling the split, I can't rule out near-duplicate memorization. That needs to be fixed by describing a source-level split or releasing the split metadata.\n\nThe real-SUIT evaluation is qualitative, and the rescaling factor was tuned on the same SUIT images where it's applied, so that part is a demonstration, not a measured result. The self-validation via Tamura/statistical features is an interesting idea for future work, but as implemented it is a visual histogram comparison; it doesn't provide a quantitative accuracy measure on real data. The paper says as much: this scheme doesn't measure the fraction of all regions captured correctly.\n\nThe paper doesn't appear to ship code or data, which makes the central reproducibility question harder.\n\nWho it's for: solar physicists working with SUIT/Aditya-L1 and anyone building ML detectors for new solar instruments. It deserves a serious referee: the mock-image construction is careful enough to be worth reviewing, and the instrumentation community needs this kind of development work published. I'd recommend accepting for review, with referees required to pin down the data split and demand code/data release as a condition of publication.","headline":"Useful engineering for SUIT, but the headline mock-data numbers need a documented source-level split before they can be trusted, and the real-data claim is qualitative.","tokens_in":20243,"tokens_out":2548,"would_cite":false,"duration_ms":26272,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AI detector finds four chromospheric features in SUIT images","keywords":["chromosphere","sunspot detection","plage regions","solar filaments","off-limb structures","object detection","YOLO","Tamura features"],"falsifier":"Manually label a few hundred real SUIT NB3 full-disc images, run SPACE on them, and compute the per-class precision and recall; if the numbers fall well below the mock-SUIT validation values (precision ≈0.788, recall ≈0.863), the mock-training transfer assumption is false.","tokens_in":19133,"feed_emoji":"☀️","tokens_out":9485,"duration_ms":82646,"temperature":0.7,"pith_summary":"SPACE-SUIT is an automatic detector that classifies four chromospheric structures—plage regions, sunspots, filaments, and off-limb structures—in images from SUIT, the ultraviolet telescope on India's Aditya-L1 observatory. Because SUIT had not accumulated enough labeled images, the authors train the detector on 'mock-SUIT' images synthesized from ten years of IRIS Mg II k full-disk mosaics, folded through SUIT's effective area and point-spread function. On the mock validation data the best model, a YOLOv8 oriented-bounding-box network, reaches a precision of about 0.788, a recall of about 0.863, and a mean average precision of about 0.874; on real SUIT Level-1 images it detects most structures of interest after a rescaling adjustment, but this evaluation is qualitative. The paper also introduces a label-free 'self-validation' scheme that compares statistical and Tamura texture features of ground-truth and predicted regions, arguing this can stand in for human labels when none exist.","feed_headline":"AI detector finds four chromospheric features in SUIT images","feed_subtitle":"Trained on synthetic IRIS mosaics, it scores 0.874 mean average precision and flags features in real Aditya-L1 images.","key_machinery":"The central mechanism is a supervised object-detection pipeline built on YOLOv8 with oriented bounding boxes (OBB), which predicts rotated rectangles for features of arbitrary orientation. The training data is mock-SUIT: roughly ten years of IRIS Mg II k full-disk mosaics are wavelength-summed over SUIT's NB3 effective area, cleaned to remove cosmic rays and mosaic stitching artifacts, convolved with SUIT's NB3 point-spread function, and rebinned to SUIT plate scale, producing images that stand in for real SUIT observations. The validation mechanism is a set of ten scalar descriptors per bounding box—Shannon entropy, standard deviation, skewness, kurtosis, and five gray-level co-occurrence matrix statistics (contrast, homogeneity, dissimilarity, energy, correlation)—whose distributions are compared across ground truth, predictions, and real SUIT detections.","core_discovery":"The paper's central claim is that SPACE—a YOLOv8-obb network trained on mock-SUIT images—can detect and classify plage regions, sunspots, filaments, and off-limb structures in SUIT's Mg II k images with a precision of ≈0.788, a recall of ≈0.863, and a mean average precision of ≈0.874 on the mock validation FITS dataset, and can identify many of these structures in real SUIT Level-1 data. A second claim is that distributions of statistical and Tamura features computed inside detected bounding boxes agree between ground truth and predictions on mock data and differ across the four feature classes, providing a way to self-validate detections where no labels exist.","pith_inferences":["A testable extension would be to train on a small set of hand-labeled real SUIT images and measure how much the 0.788/0.863 validation numbers change; if the gap closes, the mock-to-real transfer can be solved by fine-tuning rather than by a deeper domain model.","The self-validation distributions could double as an instrument-health monitor: since SUIT's throughput drifts with contaminant deposition and baking cycles, tracking entropy or contrast statistics over time may flag when a detector retrain is needed.","If the mock-image generator is updated to match SUIT's actual PSF, flat-field, and stray-light residuals, the same training recipe could be transferred to other Mg II k imagers (or other SUIT bandpasses) without re-architecting the detector.","The off-limb recall deficit (0.542) suggests that elongated, limb-crossing structures are underrepresented or too heterogeneous in the OBB labels; augmenting with more off-limb examples or a segmentation head might be needed for complete catalogs."],"forward_implications":["SPACE itself is a candidate Level 2+ data product for SUIT: it can produce per-image catalogs of plages, sunspots, filaments, and off-limb structures automatically and in near real time.","Because the model is trained on synthetic data, the same recipe can be rerun as SUIT accumulates real images, with the labeled real data replacing or augmenting the mock set.","The reported per-class scores imply that plages and filaments are detected with high recall (≈0.98 and ≈0.96 on FITS validation) while off-limb structures are the weak class (recall ≈0.54), so the model's practical use should start with the well-detected classes.","The self-validation descriptors separate the four classes (for example, sunspots have low entropy and high dissimilarity, filaments high entropy), so they can flag predictions whose texture does not match the training distribution, as happens for off-limb regions on real SUIT data.","Deploying SPACE on real data currently requires a manual rescaling of the image intensities by a factor of about 8, and the authors state that fine-tuning on real SUIT data is needed for effective identification."],"supporting_citations":[{"why":"Provides the IRIS Mg II k full-disk mosaics that are the raw material for every mock-SUIT training image.","marker":"De Pontieu et al. (2014)"},{"why":"Defines SUIT and its NB3 passband, the instrument and channel the detector is built for.","marker":"Tripathi et al. (2017)"},{"why":"Supplies the YOLOv8-obb architecture and the oriented-bounding-box labeling format used by the best-performing model.","marker":"Jocher, Chaurasia, and Qiu (2023)"},{"why":"Provides YOLOv9, the alternative architecture compared against YOLOv8 in model selection.","marker":"Wang, Yeh, and Liao (2024)"},{"why":"Documents the YOLO family architecture and loss functions that SPACE's detection head is based on.","marker":"Terven, Córdova-Esparza, and Romero-González (2023)"},{"why":"Otsu thresholding is used to build the masks for removing mosaic stitching artifacts in mock-SUIT construction.","marker":"Otsu (1979)"},{"why":"Navier-Stokes inpainting is used to fill dark stitching lines after thresholding.","marker":"Bertalmio, Bertozzi, and Sapiro (2001)"},{"why":"AIA 1700 Å images serve as the independent reference for judging sunspot detections on real SUIT data.","marker":"Lemen et al. (2012)"},{"why":"Describes the Level-1 calibration of SUIT data on which SPACE is deployed.","marker":"Sarkar et al. (2025)"}],"fun_headline_variants":["AI spots plage, spots, filaments, limb on Sun's chromosphere","YOLO network classifies solar features in Aditya-L1 SUIT data","AI solar feature detector hits 87% mean average precision","Self-validating AI identifies four solar chromospheric features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that mock-SUIT images—IRIS Mg II k mosaics convolved with SUIT's effective area and PSF—are close enough to real SUIT NB3 observations that a detector trained on them works without retraining; the paper's own rescaling and fine-tuning caveats show this premise holds only approximately.","fun_headline_variants_meta":{"raw":{"variants":["AI spots plage, spots, filaments, limb on Sun's chromosphere","YOLO network classifies solar features in Aditya-L1 SUIT data","AI solar feature detector hits 87% mean average precision","Self-validating AI identifies four solar chromospheric features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000598,"raw_usage":{"total_tokens":2843,"prompt_tokens":1036,"completion_tokens":1807,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":1731}},"tokens_in":652,"tokens_out":1807,"duration_ms":14791,"temperature":1.0,"reasoning_tokens":1731,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:44:00.451140+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually label a few hundred real SUIT NB3 full-disc images, run SPACE on them, and compute the per-class precision and recall; if the numbers fall well below the mock-SUIT validation values (precision ≈0.788, recall ≈0.863), the mock-training transfer assumption is false.","supporting_citations":[{"cited_title":"https://github.com/ultralytics/ ultralytics","cited_arxiv_id":null,"evidence_quote":"Supplies the YOLOv8-obb architecture and the oriented-bounding-box labeling format used by the best-performing model."}],"review_version":1}