{"id":"300c2d06-919d-488e-82b1-26c90a8a5592","arxiv_id":"2411.14078","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Self-supervised pretraining on radio images beats ImageNet features for radio source classification in linear evaluation, but fine-tuning erases most of the advantage.","lead":"This paper compares six self-supervised learning methods for classifying radio astronomy images, against models pretrained on regular photos, across four source classification tasks. It reports that radio-specific self-supervised pretraining clearly beats ImageNet features in linear evaluation, but the advantage mostly disappears when the whole network is fine-tuned.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline linear-evaluation gap may partly be a preprocessing artifact: ImageNet baselines are probed with per-cutout min-max normalization rather than their native ImageNet mean/std normalization, and MiraBest—the largest gap—is unaffected by resampling leakage.","rationale":"Good-faith reading: the paper is a benchmark whose core evidence is Table 1, claiming radio SSL models beat ImageNet-pretrained models in linear evaluation. For that claim to hold, the baseline must be evaluated in a way that isolates pretraining domain rather than input preprocessing. Section 4.1 applies per-cutout min-max normalization to all inputs. ImageNet BYOL was pretrained with standard mean/std normalization. A frozen ImageNet backbone evaluated on min-max-normalized radio cutouts operates out-of-distribution; its low linear-probe accuracy may therefore indicate preprocessing mismatch, not inferior natural-image features. Table 2 supports this reading: after fine-tuning adapts the backbone, ImageNet baselines reach parity or better. MiraBest, the largest Table 1 gap (91.0 vs 67.5), is explicitly not resampled, so the resampling-order issue identified by the reader cannot explain the flagship number; this makes the preprocessing confound the most load-bearing concern. Secondary issues (resampling order, MSRS/Uncurated overlap, ImageNet-100 baseline for ResNet-18) should also be checked, but they are either orthogonal to MiraBest or affect all methods. The proposed test is one experiment: extract ImageNet baseline features with native normalization and re-run the linear heads. If the gap persists, the central claim is strengthened; if it collapses, the paper's headline overstates the value of radio-specific SSL.","tokens_in":12606,"tokens_out":13176,"duration_ms":135694,"concrete_test":"Re-run the linear-evaluation rows for the ImageNet-100 (ResNet-18) and ImageNet-1k (ResNet-50) BYOL checkpoints using the released code, but replace the per-cutout min-max normalization with the standard ImageNet per-channel normalization (mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) after resizing to 224 and replicating grayscale to three channels; keep the same 3-fold protocol and linear-head training. If MiraBest accuracy rises from 67.5 toward the 85–91 range, the reported SSL advantage is largely an artifact of preprocessing. For completeness, also report the same baseline under min-max normalization to isolate the effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states that all input images are normalized using the minimum and maximum values within a single cutout and then resized to 224×224. The ImageNet-pretrained BYOL baselines, however, were trained with the standard per-channel mean/std normalization (mean 0.485, 0.456, 0.406; std 0.229, 0.224, 0.225). In linear evaluation the backbone is frozen, so a shifted input distribution can severely degrade the extracted features; the baseline cannot adapt. Table 2 is consistent with this: once the full backbone is fine-tuned and can adapt to the min-max input distribution, ImageNet baselines match or exceed the radio-pretrained models. The largest advertised gap in Table 1 is MiraBest, 91.0 vs 67.5, and MiraBest is explicitly not resampled (§3.4), so the reader's resampling-order concern cannot explain that flagship number. A preprocessing mismatch can. Thus the central claim—that radio SSL features are substantially better for linear transfer—is not yet established; the comparison needs an ImageNet-normalized baseline.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a benchmark of six self-supervised learning methods (SimCLR, BYOL, DINO, WMSE, SwAV, All4One) pretrained on two radio-continuum datasets (a curated SMGPS set and an uncurated SMGPS/EMU set), evaluated by linear probing and fine-tuning on four radio classification tasks (MiraBest, RGZ, MSRS, VLASS), with ImageNet-pretrained BYOL as a baseline. The authors report that radio-pretrained SSL models substantially outperform the ImageNet baseline in linear evaluation, that curated pretraining helps more than dataset size, and that All4One is the strongest linear-evaluation method; they also introduce the MSRS dataset. The headline claim is that SSL pretraining on radio data yields transferable representations for SKA-era analysis.","tokens_in":12875,"tokens_out":7888,"duration_ms":73929,"significance":"The benchmark is timely and useful: it compares six SSL methods in a consistent solo-learn framework, uses public downstream tasks, releases code and models, and introduces MSRS, a multi-survey labeled dataset. The linear-evaluation results are clearly tabulated and the experimental design is largely reproducible. If the main advantage over ImageNet is confirmed under a fair preprocessing protocol, the result would be a valuable practical guide for SKA preparation. At present, however, the headline claim is not fully established because of the normalization mismatch with the ImageNet baseline, an ambiguous resampling protocol, and a confounded curation comparison.","major_comments":[{"comment":"Section 4.1 states that all input images are normalized with per-cutout min-max scaling and then resized to 224x224, but the ImageNet-pretrained BYOL baselines were trained with the standard per-channel mean/std normalization of ImageNet. Under linear evaluation the backbone is frozen, so presenting min-max-normalized inputs to an ImageNet model is a distribution shift that can substantially degrade the extracted features; the model cannot adapt. This is consistent with Table 2, where fine-tuning (which can adapt to the input distribution) removes most of the gap between ImageNet and radio-pretrained models. Consequently the headline linear-evaluation advantage in Table 1, including the flagship MiraBest comparison (SimCLR 91.0 vs ImageNet 67.5), is not a controlled comparison. Please add an ImageNet-pretrained baseline evaluated with ImageNet-style normalization, or otherwise show that the preprocessing mismatch does not change the ranking.","section":"Section 4.1"},{"comment":"Section 3.4 resamples each downstream dataset to balance classes by undersampling or duplicating samples, but it does not state whether this is done before or after the 3-fold cross-validation described in Section 4.1. If resampling precedes the split, duplicated cutouts from the same minority source can appear in both training and test folds, inflating the reported accuracies and potentially changing the SSL-versus-ImageNet ranking. MiraBest is not resampled, so this does not explain that result, but it affects RGZ, MSRS, and VLASS. Please clarify the order, and if resampling was performed before splitting, rerun the evaluation with resampling nested inside each training fold.","section":"Section 3.4"},{"comment":"Section 4.1 trains the Uncurated models for 100 epochs and the Curated models for 600 epochs, with the same batch size. The curated-vs-uncurated comparison therefore varies curation, dataset size (285,585 vs 17,062), and total optimizer steps simultaneously, so Section 4.2's conclusion that 'dataset curation positively impacts results' and matters more than dataset size is not supported by this design. The claim that 'all SSL methods benefit' is also contradicted by Table 1: DINO on MiraBest drops from 83.6 uncurated to 64.2 curated (and from 69.0 to 66.6 on VLASS). Please match training budgets, or explicitly treat the epoch difference as a confound and weaken the curation claim accordingly.","section":"Sections 4.1 and 4.2"},{"comment":"The abstract states that when the entire backbone is fine-tuned, 'the benefits of SSL are less evident but still outperform pretraining,' and Section 5 repeats the claim. Table 2 does not support it: the ImageNet baselines match or exceed the radio-pretrained models in most configurations, e.g., ImageNet-100 BYOL ResNet-18 reaches 96.2 on MiraBest, 81.0 on RGZ, 75.4 on MSRS, and 84.1 on VLASS, compared with best curated ResNet-18 values of 99.2, 81.6, 77.0, and 83.5, respectively; ImageNet-1k BYOL ResNet-50 reaches 98.5 on MiraBest. Please revise the abstract and conclusions to state that fine-tuning benefits are not observed relative to ImageNet pretraining in this benchmark.","section":"Abstract and Section 5"}],"minor_comments":[{"comment":"The WMSE description ends with a sentence fragment ('The loss could be represented as: uses the mutual information maximization...'), and the displayed loss only contains the whitening term; please rewrite for clarity.","section":"Section 3.3"},{"comment":"There are several typos: 'horiziontal' (horizontal), 'temperatur' (temperature), 'redundandy reduction loss' (redundancy reduction loss), and 'abilty' (ability) in Section 3.4.","section":"Sections 3.3 and 4.1"},{"comment":"Please state explicitly whether the 3-fold cross-validation folds are stratified by class and how the 9 accuracy values (3 seeds x 3 folds) are combined; the current text reports only that the mean and standard deviation are computed.","section":"Section 4.1"},{"comment":"The text says ImageNet baselines are 'significantly worse,' but no significance tests are reported; please either add appropriate statistical tests or rephrase to 'numerically worse' where the standard deviations overlap.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the MSRS dataset and the code/model release are valuable contributions, and the benchmark scope is appropriate for the journal. The main issues are experimental protocol (preprocessing mismatch, resampling order, training-budget confound) and an overstated fine-tuning claim; I believe these are addressable in a revision rather than requiring a new paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful benchmark, but the central quantitative claim is not yet supported because the ImageNet baselines are given a preprocessing disadvantage in linear evaluation, and the abstract overstates the fine-tuning results. The new MSRS dataset and the scope of the comparison are real contributions.\n\nWhat's new: six SSL methods (SimCLR, BYOL, DINO, WMSE, SwAV, All4One) pretrained on curated and uncurated radio images, evaluated on four downstream tasks including the new MSRS dataset, with code and trained models released. The use of one library (solo-learn) and a consistent evaluation protocol (3 seeds × 3 folds, mean/std) is good practice. Prior work (Slijepcevic et al., Riggi et al.) covered a subset of this space; the systematic comparison is new.\n\nWhat's solid: the linear evaluation table is clearly reported and the results are internally consistent for the radio-pretrained models. The fine-tuning table shows the expected flattening across methods, and the authors are honest that advantages are less pronounced there. Acknowledging the MSRS overlap with uncurated pretraining is also a point in their favor.\n\nThe soft spots are real and need addressing.\n\nFirst, the preprocessing mismatch. Section 4.1 normalizes every input image using per-cutout min-max scaling. ImageNet-pretrained BYOL was trained with per-channel mean/std normalization. In linear evaluation the backbone is frozen, so feeding min-max-scaled cutouts to an ImageNet backbone puts the input far outside its training distribution. That can degrade features substantially. The fine-tuning results in Table 2 are consistent with this: once the backbone can adapt, ImageNet baselines match or exceed the radio-pretrained models. So the headline gap in Table 1, including the big MiraBest gap, may be partly an artifact. A proper ImageNet-normalized baseline is needed before the claim 'radio SSL features are substantially better' is established.\n\nSecond, the abstract says fine-tuning still outperforms pretraining, but Table 2 shows ImageNet baselines winning or tying on some datasets. That's a factual error.\n\nThird, the curated vs uncurated comparison is confounded: curated is trained for 600 epochs, uncurated for 100. Different training budgets, so the claim that curation matters more than size is not supported by this design.\n\nFourth, the resampling order for the balanced downstream datasets isn't specified. If duplication happens before the 3-fold split, copies of the same source can appear in both train and test. MiraBest isn't resampled, so the largest gap isn't explained by this, but the other datasets could be affected. The paper should state when resampling occurs.\n\nThese are all addressable. The benchmark is worth a serious referee. My recommendation: send to peer review, but the authors need to rerun the linear evaluation with proper ImageNet normalization, fix the abstract, control or drop the curation claim, and clarify the resampling procedure. If those are done, the paper would be a useful reference for the SKA-era community.","headline":"Useful benchmark, but the headline result is partly an artifact of a preprocessing mismatch and the abstract overstates the fine-tuning outcome.","tokens_in":13436,"tokens_out":5362,"would_cite":false,"duration_ms":46730,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that self-supervised pretraining on unlabeled radio images beats ImageNet pretraining for radio-source classification.","keywords":["self-supervised learning","radio astronomy","source classification","benchmark","representation learning","transfer learning","data curation","interferometry"],"falsifier":"Run the linear-evaluation protocol again on the same downstream datasets, but perform the balanced resampling only on the training fold after each split (or remove all duplicate cutouts before splitting), and check whether the radio-pretrained models still beat the ImageNet baselines by the same margins; if the margins collapse, the reported advantage is an artifact of train/test leakage.","tokens_in":12409,"feed_emoji":"📡","tokens_out":5735,"duration_ms":49098,"temperature":0.7,"pith_summary":"The paper argues that self-supervised pretraining on radio-interferometry images produces features better suited to radio-source classification than the standard practice of pretraining on optical or natural images. It compares six self-supervised methods (SimCLR, BYOL, DINO, WMSE, SwAV, All4One) pretrained on either a curated set of 17,062 SMGPS source cutouts or an uncurated set of 285,585 sliding-window cutouts, then evaluated on four downstream classification datasets (MiraBest, RGZ, MSRS, VLASS). In linear evaluation, where a frozen backbone feeds a single linear classifier, the radio-pretrained models outperform ImageNet-pretrained BYOL baselines by substantial margins, e.g., 91.0% vs 67.5% on MiraBest. When the whole backbone is fine-tuned, the gap narrows and the advantage is modest. The paper concludes that domain-specific self-supervision, especially with curated data, is a valuable and efficient alternative to natural-image pretraining for the upcoming SKA data deluge.","feed_headline":"Radio-only pretraining beats ImageNet for radio-source classification","feed_subtitle":"SSL-trained on unlabeled MeerKAT and ASKAP cutouts outperforms ImageNet on four radio classification benchmarks.","key_machinery":"The benchmark is built on two pretraining datasets and a fixed evaluation protocol: the Curated dataset (17,062 cutouts centered on SMGPS extended sources, sized at 2.5 times each source bounding box), the Uncurated dataset (285,585 fixed 256x256 sliding-window cutouts from SMGPS and EMU maps), and six augmentation-based SSL objectives (SimCLR, BYOL, DINO, WMSE, SwAV, All4One). Representations are evaluated by 3-fold cross-validation on four downstream classification datasets, in two regimes: linear evaluation (train a linear classifier on frozen backbone features) and fine-tuning (update the whole backbone with AdamW). The load-bearing comparison is linear evaluation, where the frozen-feature accuracy isolates the quality of the pretrained representation.","core_discovery":"The central claim is that SSL pretraining on radio images yields representations that transfer better to radio-astronomy classification than representations pretrained on natural images, and that this advantage is most visible when the pretrained backbone is used as a fixed feature extractor. The paper supports this with a systematic benchmark: for every SSL method and both backbones, the radio-pretrained models exceed the ImageNet-pretrained baseline on almost all downstream tasks in linear evaluation, with All4One, BYOL, and SimCLR leading. Curated data outperform uncurated data despite being nearly seventeen times smaller, indicating sample quality matters more than volume. The authors interpret the results as evidence that radio-specific pretraining can reduce dependence on large labeled sets and better exploit the abundant unlabeled radio sky.","pith_inferences":["If the ranking holds under a stricter evaluation protocol, one immediate extension is to use these SSL backbones for anomaly detection and source-finding, where the same latent representations could flag rare morphologies without labels.","The success of curated pretraining suggests a data-engineering recipe for other wavelength regimes: cut out sources from catalogues rather than sampling blindly; the same recipe could apply to optical and infrared surveys.","The authors leave multimodal extensions implicit; combining radio cutouts with optical or infrared hosts in a joint SSL objective could produce representations that transfer even better to classification of multi-wavelength source populations."],"forward_implications":["Radio-pretrained SSL models can serve as ready-made feature extractors for SKA-era surveys, giving accurate classifications without per-task backbone fine-tuning.","Data curation, not just volume, is the main driver of representation quality, so survey teams should invest in catalogue-based cutout extraction.","The margins in linear evaluation suggest that future radio-domain foundation models should be pretrained on radio data rather than initialized from ImageNet.","MiraBest is nearly saturated in fine-tuning (above 96% for all methods), so future benchmarks should rely on harder tasks like RGZ or MSRS to discriminate between pretraining strategies."],"supporting_citations":[{"why":"Closest prior work; pretrains BYOL on RGZ and fine-tunes on MiraBest, providing the direct comparison baseline for the fine-tuning results.","marker":"[33]"},{"why":"Implementation library used for all six SSL methods, ensuring a consistent training setup across methods.","marker":"[9]"},{"why":"Downstream benchmark of FRI/FRII radio galaxies; where the largest SSL-over-ImageNet gap in linear evaluation appears.","marker":"[28]"},{"why":"Radio Galaxy Zoo dataset used both as a downstream classification task and as the pretraining corpus of the prior BYOL model.","marker":"[2]"},{"why":"VLASS quick-look catalogue used to build the VLASS downstream classification dataset.","marker":"[17]"},{"why":"SMGPS extended source catalogue from which the Curated pretraining cutouts are extracted.","marker":"[16]"},{"why":"SMGPS survey images used for both Curated and Uncurated pretraining cutouts.","marker":"[15]"},{"why":"BYOL method used as both a radio-pretrained SSL model and the ImageNet-pretrained baseline.","marker":"[18]"}],"fun_headline_variants":["Radio-pretrained SSL beats ImageNet in radio benchmarks","Self-supervised radio models outperform ImageNet pretraining","Curated radio data beats ImageNet for SSL pretraining","MeerKAT SSL outshines ImageNet for radio source classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies assume that the balanced resampling of the downstream datasets is applied inside each cross-validation fold, so duplicated minority-class cutouts never appear in both the training and test partitions; the paper does not state where resampling happens in the pipeline.","fun_headline_variants_meta":{"raw":{"variants":["Radio-pretrained SSL beats ImageNet in radio benchmarks","Self-supervised radio models outperform ImageNet pretraining","Curated radio data beats ImageNet for SSL pretraining","MeerKAT SSL outshines ImageNet for radio source classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000369,"raw_usage":{"total_tokens":1962,"prompt_tokens":914,"completion_tokens":1048,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":979}},"tokens_in":530,"tokens_out":1048,"duration_ms":9678,"temperature":1.0,"reasoning_tokens":979,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:33:20.844161+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the linear-evaluation protocol again on the same downstream datasets, but perform the balanced resampling only on the training fold after each split (or remove all duplicate cutouts before splitting), and check whether the radio-pretrained models still beat the ImageNet baselines by the same margins; if the margins collapse, the reported advantage is an artifact of train/test leakage.","supporting_citations":[{"cited_title":"RAS Techniques and Instruments 3(1), 19–32 (12 2023)","cited_arxiv_id":null,"evidence_quote":"Closest prior work; pretrains BYOL on RGZ and fine-tunes on MiraBest, providing the direct comparison baseline for the fine-tuning results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Implementation library used for all six SSL methods, ensuring a consistent training setup across methods."},{"cited_title":"Monthly Notices of the Royal Astronomical Society453(3), 2326–2340 (2015)","cited_arxiv_id":null,"evidence_quote":"Radio Galaxy Zoo dataset used both as a downstream classification task and as the pretraining corpus of the prior BYOL model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VLASS quick-look catalogue used to build the VLASS downstream classification dataset."},{"cited_title":"Monthly Notices of the Royal Astronomical Society531(1), 649–681 (2024)","cited_arxiv_id":null,"evidence_quote":"SMGPS extended source catalogue from which the Curated pretraining cutouts are extracted."}],"review_version":1}