{"id":"6f8f4105-479a-46fa-ae0d-7b4d1d21e6d0","arxiv_id":"2504.18049","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Applying a SparK-style masked autoencoder to a lightweight CNN improves retinal disease classification, but the AD/PD gains are weakened by participant overlap between pre-training and evaluation sets.","lead":"A computer model called nn-MobileNet was pre-trained on 178,803 unlabeled retinal images using a masked-image self-supervised method, then fine-tuned to detect Alzheimer's, Parkinson's, and retinal disease. The authors report accuracy gains, but the pre-training data includes the same patients used in the AD and PD tests, which may inflate the improvements.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pre-training overlap with downstream AD/PD participants is not tested; if it inflates results, the headline gain claim weakens.","rationale":"I agree with the reader's identification of the weakest assumption: the AD/PD evaluation is the central claimed deliverable, and the pre-training corpus overlaps with the evaluation participants. The reader's verdict of CONDITIONAL is appropriate because the paper itself provides one genuinely disjoint validation (MMAC), where pre-training improves kappa by 0.0232, so the method is not vacuous. However, the AD/PD headline numbers are the ones that support the strongest claim, and their credibility depends entirely on the overlap being harmless. The paper's justification (no labels used; 'consistent with common practice') is an assertion, not a measurement. Self-supervised pre-training can still leak identity information through image-level features, and with two eyes per participant plus repeated visits, the same participant can appear in both pre-training and test splits. The missing error bars compound the problem: the PD gain of 0.0124 kappa (0.9876 vs 0.9752) is small relative to the near-ceiling accuracy, and without per-fold variance we cannot distinguish a real improvement from noise. The proposed disjoint-participant pre-training experiment is the single check that would settle this. I would not move the verdict to REJECT because the MMAC result provides independent, if weaker, evidence that the pre-training helps, and the method itself is a reasonable SparK-style adaptation. I would not move it to ACCEPT because the headline claim is not yet supported by a clean evaluation. CONDITIONAL with the disjoint-participant check as the condition matches the reader's verdict, and I agree with that assessment.","tokens_in":7911,"tokens_out":1913,"duration_ms":16370,"concrete_test":"Re-run the pre-training on a disjoint subset of UK Biobank images that excludes all participants appearing in the AD and PD evaluation sets (remove the 623 AD/PD participants and their images from the 178,803-image corpus), then fine-tune the same nn-MobileNet backbone with the same hyperparameters and report AD/PD accuracy, AUC, and kappa with per-fold standard deviations. If the PD kappa gain over the non-pre-trained baseline falls to within one standard deviation, the headline claim of significant improvement on AD/PD is not supported; if the gain persists, the overlap concern is settled.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central empirical claim is that BERT-style pre-training on 178,803 UK Biobank fundus images 'significantly improves performance' on AD, PD, and retinal disease tasks. The load-bearing condition is that this improvement is attributable to generalizable representation learning rather than to accidental exposure to the same participants' images used in downstream evaluation. Section 3 states that AD/PD test images come from the same UK Biobank cohort used for pre-training, and the paper's only defense is that labels are not used and that this is 'consistent with common self-supervised approaches.' That defense is not evidence. The MMAC experiment is a genuinely disjoint test and does support the method, but the AD/PD tables are the headline results: a 99.38% PD accuracy and a kappa gain of 0.0124 over the non-pre-trained nn-MobileNet baseline. The absolute numbers are so high that small overlaps or near-duplicate images (same participant, both eyes, multiple visits) could plausibly account for the remaining gap between Ours (0.9938) and nn-MobileNet (0.9876). Also, the paper reports no per-fold standard deviations, so we cannot tell whether the PD gain is within run-to-run noise. These gaps are the crux of the claimed improvement, and neither a disjoint-participant pre-training run nor a variance estimate is provided. The correct test is to re-run pre-training with all AD/PD participants excluded from the pre-training corpus and compare downstream performance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a BERT-style masked-image self-supervised pre-training scheme for the lightweight CNN nn-MobileNet. The method masks 60% of each input, applies sparse convolution to process only visible pixels, and reconstructs masked regions using a hierarchical encoder-decoder with an MSE loss. The model is pre-trained on 178,803 unlabeled UK Biobank fundus images and then evaluated on Alzheimer's disease (AD), Parkinson's disease (PD), and MMAC myopic maculopathy classification. The authors report that pre-training improves PD accuracy and MMAC metrics, and that the pre-trained model is on par with or better than strong baselines including RETFound.","tokens_in":8180,"tokens_out":5608,"duration_ms":55103,"significance":"If the evaluation is valid, the paper makes a useful empirical contribution: it shows that a lightweight CNN can obtain masked-image pre-training benefits with roughly one-tenth of the data used by ViT-based foundation models, while retaining CNN locality and computational efficiency. The method itself is sensible and builds on existing SparK/ConvNeXt-v2 ideas, and the MMAC experiment provides a genuinely disjoint evaluation in which pre-training yields plausible gains (Kappa +0.0232, Weighted F1 +0.0412). The main limitation is that the headline AD/PD result is compromised by participant overlap between pre-training and downstream evaluation, and by the absence of variance estimates, so the central claim of 'significant improvement' is not yet established.","major_comments":[{"comment":"The AD/PD evaluation is contaminated by the fact that the unlabeled pre-training corpus contains the same UK Biobank participants whose images are used downstream. The paragraph beginning 'Although we use the fundus images derived from patients with a diagnosis of AD and PD from the UK Biobank' argues that pre-training does not use labels and is consistent with common self-supervised practice, but this is not evidence that participant overlap does not inflate performance. Because the PD gain over the non-pre-trained nn-MobileNet baseline is only +0.0062 accuracy (+0.0124 kappa), overlap or near-duplicate images from the same participants could plausibly account for the gap. Please re-run pre-training with all AD/PD participants excluded from the pre-training corpus and compare downstream performance, or otherwise provide a direct test that overlap does not affect the reported gains.","section":"Section 3, Fig. 4, Table 1"},{"comment":"No error bars, per-fold results, or significance tests are reported for the AD/PD experiments, so the abstract's claim of 'significantly improves performance' is not statistically supported. On AD, the pre-trained model yields exactly the same metrics as nn-MobileNet (ACC 0.9933, AUC 1.0, Kappa 0.9866), so pre-training shows no measurable gain there; on PD, the difference between 0.9938 and 0.9876 could easily be within run-to-run noise given the small dataset (415 PD images). Please provide per-fold statistics with standard deviations and a paired significance test across folds.","section":"Table 1, Section 3"},{"comment":"The MMAC experiment is the strongest evidence for the method, but it also reports only 5-fold means without variance or significance testing. The claims that Kappa, Accuracy, and F1 'significantly improved' should be accompanied by per-fold results or confidence intervals, especially with only 1143 images and four imbalanced classes. In addition, the text states that 'Kappa value improved by 0.232' and 'Accuracy increased by 0.0238', which disagree with Table 2's +0.0232 and +0.0208; these numbers should be reconciled.","section":"Section 3, Table 2"}],"minor_comments":[{"comment":"The text after Table 1 states that the AD result has 'an AUC of 0.9997', but Table 1 reports AD AUC as 1.0; please correct the inconsistency.","section":"Section 3"},{"comment":"The description 'we randomly divided the data into training and validation sets in the ratio of 8:2. We employed 5-fold stratified cross-validation' is ambiguous; 5-fold cross-validation and a single 80/20 split are different procedures, so please clarify which evaluation protocol was used.","section":"Section 3, paragraph 1"},{"comment":"The figure shows both a 'Normal Control group' with 176,392 images / 86,069 participants and an 'NC group with GOOD quality images' with 389 images / 282 participants; it is unclear how many control subsets are used for AD versus PD, and whether the same control images are used in both tasks.","section":"Fig. 4"},{"comment":"The phrase 'significantly improve performance in the downstream tasks' is stronger than the evidence in Table 1, where AD metrics are unchanged by pre-training; please soften the claim or add supporting statistics.","section":"Abstract and Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical engineering contribution rather than a methodological novelty, and the MMAC result suggests the pre-training scheme has merit. The main blocker is the evaluation design for AD/PD: the participant overlap between pre-training and downstream evaluation, together with the absence of error bars, makes the headline claim unsupported. If the authors can add a disjoint-participant pre-training run and report per-fold statistics, the paper could be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the paper with the stress-test note in hand, and I think the note lands. The paper does one useful thing and then weakens its own headline. The useful thing is the MMAC result: a SparK/ConvNeXt V2 style masked autoencoder applied to nn-MobileNet gives a genuinely disjoint benchmark gain—kappa +0.023, F1 +0.041, accuracy +0.021. That is modest but plausible, and it supports the claim that lightweight CNNs can get BERT-style pre-training benefits without ViT-scale data. The method section is honest about provenance; SparK and ConvNeXt V2 are cited plainly, and the novelty is framed as a new backbone and domain, which is fair.\n\nThe soft spot is the AD/PD evaluation, and it is load-bearing. The pre-training corpus is 178,803 UK Biobank images from 87,245 participants, and the AD/PD test sets come from the same UK Biobank cohort. The paper's defense—that labels are unused and that this is common practice—does not address the actual risk: the model has seen the same participants' eyes, both eyes, during pre-training. At 99.38% PD accuracy, even a small identity cue could explain the gap over the 98.76% baseline. The stress-test note is right that the correct fix is a disjoint-participant pre-training run. Also, there are no per-fold variances, so we cannot tell if the PD gain is noise.\n\nMinor issues: the text says kappa improved by 0.232 but the table says 0.0232; the AD AUC in the text is 0.9997 while the table says 1.0; no code release. All fixable. The MMAC experiment should be replicated and reported with error bars.\n\nWho is this for? People working on label-scarce medical imaging who want a lightweight CNN alternative to ViT foundation models. The MMAC result alone deserves a look. But the AD/PD headline should not be trusted until the overlap is resolved. I would not desk-reject; a serious referee should ask for the disjoint pre-training experiment and variance estimates. If the MMAC gains survive, the paper is a useful contribution even with the AD/PD claims trimmed.","headline":"SparK-style pre-training on a lightweight CNN shows real gains on a disjoint retinal benchmark, but the headline AD/PD results are compromised by participant overlap between pre-training and test sets.","tokens_in":8732,"tokens_out":3142,"would_cite":false,"duration_ms":26654,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A BERT-style masked-image pre-training scheme lets a lightweight CNN identify Parkinson's disease from retinal images with 99.38% accuracy, matching or beating transformer-based foundation models while using far less data.","keywords":["self-supervised learning","pre-training","convolutional neural networks","retinal fundus images","masked image modeling","sparse convolution","BERT-style pre-training","disease identification"],"falsifier":"Pre-train the same CNN on 178,803 unlabeled images drawn only from participants who never appear in the AD or PD evaluation sets, fine-tune on the same labelled splits, and compare the resulting accuracy and kappa to the reported 99.38% PD accuracy and 0.9876 kappa. A substantial drop would show that patient overlap between pre-training and evaluation, not the pre-training method alone, drives the reported improvement.","tokens_in":7723,"feed_emoji":"👁️","tokens_out":7762,"duration_ms":71823,"temperature":0.7,"pith_summary":"The paper tries to establish that a lightweight convolutional network can receive the same kind of benefit from masked-image pre-training that transformer-based foundation models receive, without the heavy compute and data appetite. Its method randomly masks 60% of a retinal fundus image, reconstructs the hidden regions through a hierarchical encoder-decoder using sparse convolution, and pre-trains on 178,803 unlabeled images. After pre-training, the same CNN reaches 99.33% accuracy for Alzheimer's disease identification and 99.38% for Parkinson's disease, and improves grading of myopic maculopathy by 0.0232 in quadratic-weighted kappa over the non-pre-trained network. A sympathetic reader would care because it suggests that label scarcity in medical imaging can be addressed with modest computing resources, using CNNs that already dominate clinical image analysis.","feed_headline":"Pre-training on masked eye images lifts a light CNN to 99.38%","feed_subtitle":"A lightweight CNN with sparse-convolution masking rivals heavy transformer foundation models at a fraction of the data.","key_machinery":"The load-bearing mechanism is sparse convolution in a hierarchical masked-image reconstruction pipeline. Sparse convolution computes the convolution operation only at unmasked, non-zero pixel positions, so masked regions cannot leak into feature computations; the paper extends this to self-supervised pre-training by randomly masking 60% of each image and feeding only visible pixels into the encoder. The encoder produces feature maps at five downsampled resolutions, a projection layer adapts their channels, and a lightweight UNet-style decoder with four upsampling blocks reconstructs the masked area, with the loss being mean squared error on the masked region only. This combination lets a MobileNetV2-derived CNN behave like a masked autoencoder while preserving the spatial hierarchy and localization that make CNNs attractive for medical images.","core_discovery":"The central claim is that BERT-style self-supervised pre-training, implemented on a CNN through sparse convolution, significantly improves downstream disease identification from retinal images. The authors argue that prior masked-image modelling worked for transformers because transformers process variable-length patch sequences, whereas ordinary convolutions sweep over masked pixels and corrupt the signal; sparse convolution removes that obstruction by computing only over unmasked pixels. During pre-training, a random 60% mask is applied, hierarchical feature maps at five resolutions are built, and a lightweight UNet decoder reconstructs the masked regions under an MSE loss. After pre-training, the network matches or beats comparisons including a foundation model pre-trained on 1.6 million retinal images, using about one tenth of that data, and produces attention maps consistent with known retinal vessel changes in Alzheimer's disease.","pith_inferences":["Editorial inference: a decisive control experiment would pre-train on a participant-disjoint set of unlabeled images and re-run the AD and PD fine-tuning; the paper does not report this, so its headline gains could overstate generalization if the pre-training set contains images from the same patients used for evaluation.","Editorial inference: the method's components are not intrinsically tied to this specific backbone, so the same sparse-convolution masking recipe could plausibly transfer to other efficient CNNs, such as ResNet or EfficientNet variants; testing that would show whether the gain is a property of the recipe or of the chosen backbone.","Editorial inference: because the reconstruction target is the original image's pixels, the pre-training also learns background and imaging artifacts; a version that reconstructs only anatomical structures, such as vessels, might learn representations that generalize better across acquisition devices."],"forward_implications":["If the central claim holds, masked-image pre-training is no longer tied to transformer architectures; lightweight CNNs can be pre-trained with the same BERT-style recipe and then fine-tuned on small labelled sets.","The reported gains on myopic maculopathy grading imply that the pre-training improves multi-class, multi-level disease classification, not just binary case-control tasks.","Because the pre-trained network reaches parity with a foundation model trained on roughly nine times more images, the method points toward a lower-data and lower-compute route to medical foundation-model-style performance.","The same architecture could be retrained and evaluated on other imaging modalities such as optical coherence tomography or magnetic resonance imaging, which the paper names as future work."],"supporting_citations":[{"why":"defines the masked-language-modelling paradigm that inspires the image-masking strategy.","marker":"[2]"},{"why":"supplies the masked autoencoder baseline that this work compares against and adapts to a CNN backbone.","marker":"[7]"},{"why":"introduces sparse convolution, the operation that lets the CNN skip masked pixels consistently.","marker":"[10]"},{"why":"provides the myopic maculopathy challenge benchmark used to measure pre-training gains in multi-class grading.","marker":"[12]"},{"why":"provides the UNet decoder architecture used for reconstructing masked feature maps.","marker":"[15]"},{"why":"supplies the 178,803 unlabeled retinal fundus images used for pre-training.","marker":"[20]"},{"why":"demonstrates hierarchical sparse masked modelling for CNNs, the template this method follows.","marker":"[21]"},{"why":"is the transformer foundation model baseline pre-trained on over 1.6M retinal images that the method matches or beats.","marker":"[27]"},{"why":"supplies the lightweight CNN backbone used throughout the experiments.","marker":"[28]"}],"fun_headline_variants":["Sparse convolution unlocks BERT-style pre-training for CNNs","Light CNN learns from masked retinal images, bests heavy models","Masked image modelling on a CNN: less data, high accuracy","Self-supervised CNN: pre-train on unlabeled eyes, beat transformers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that using unlabeled images from the same Alzheimer's and Parkinson's patients in pre-training does not inflate downstream accuracy; if the pre-training corpus overlaps with the evaluation cohort, some of the reported gains could come from having effectively seen those patients' images before, rather than from learning general retinal representations.","fun_headline_variants_meta":{"raw":{"variants":["Sparse convolution unlocks BERT-style pre-training for CNNs","Light CNN learns from masked retinal images, bests heavy models","Masked image modelling on a CNN: less data, high accuracy","Self-supervised CNN: pre-train on unlabeled eyes, beat transformers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000294,"raw_usage":{"total_tokens":1720,"prompt_tokens":962,"completion_tokens":758,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":683}},"tokens_in":578,"tokens_out":758,"duration_ms":7704,"temperature":1.0,"reasoning_tokens":683,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:25:02.083673+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pre-train the same CNN on 178,803 unlabeled images drawn only from participants who never appear in the AD or PD evaluation sets, fine-tune on the same labelled splits, and compare the resulting accuracy and kappa to the reported 99.38% PD accuracy and 0.9876 kappa. A substantial drop would show that patient overlap between pre-training and evaluation, not the pre-training method alone, drives the reported improvement.","supporting_citations":[{"cited_title":"upsaclay.fr/competitions/12441(2023) 10 Authors Suppressed Due to Excessive Length","cited_arxiv_id":null,"evidence_quote":"provides the myopic maculopathy challenge benchmark used to measure pre-training gains in multi-class grading."},{"cited_title":"PLoS Med12(3), e1001779 (Mar 2015)","cited_arxiv_id":null,"evidence_quote":"supplies the 178,803 unlabeled retinal fundus images used for pre-training."},{"cited_title":"Nature622(7981), 156–163 (2023) A BERT-Style Self-Supervised Learning CNN 11","cited_arxiv_id":null,"evidence_quote":"is the transformer foundation model baseline pre-trained on over 1.6M retinal images that the method matches or beats."},{"cited_title":"In: International Conference on Medical Image Computing and Computer-Assisted Intervention","cited_arxiv_id":null,"evidence_quote":"supplies the lightweight CNN backbone used throughout the experiments."}],"review_version":1}