{"id":"7bf46711-2761-4501-bea1-3e8a4c16c721","arxiv_id":"2608.00586","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Pretraining objective strongly affects frozen feature transfer in MIL for ultra-widefield retinal imaging; DINO-based encoders outperform MAE on five-class DR grading, a gap mostly closed by partial fine-tuning.","lead":"This paper compares how different pretrained AI vision models transfer to diagnosing diseases in ultra-widefield retinal images. It finds that the pretraining method matters strongly when the model is frozen, with DINO-style self-distillation beating reconstruction-based MAE on five-class diabetic retinopathy grading.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pooling strategy is co-varied with pretraining objective, so the attribution to pretraining objective is not fully identified; the sensitivity analysis covers only two of the controlled encoders.","rationale":"Reader's weakest_assumption aligns with the main unresolved issue: the controlled comparison does not fully isolate pretraining objective because pooling strategy is co-varied. The paper has credible controls—fixed MIL architecture, same ImageNet-1k pretraining for the three main encoders, patient-level MMRDR split, three seeds, partial fine-tuning control, and candid limitations (Section VI). However, the claim that pretraining objective 'substantially influenced' transfer is an attribution claim, and the experimental setup does not fully identify the objective because pooling is interleaved with it. The sensitivity analysis narrows but does not close the gap for the two central discriminative encoders. A matched-pooling rerun is inexpensive and decisive. The abstract's overgeneralization beyond the primary dataset is a secondary concern; the pooling confound is the more fundamental risk. Thus the verdict remains CONDITIONAL rather than ACCEPT or REJECT.","tokens_in":18847,"tokens_out":7849,"duration_ms":102622,"concrete_test":"Re-run the MMRDR five-class comparison (Section IV.A.2, three seeds) for ViT-Supervised, ViT-DINOv1, ViT-MAE, and ViT-DINOv3 under two matched pooling protocols: (a) all encoders use the CLS token; (b) all encoders use mean-pooled spatial tokens, keeping the MIL training protocol, bag construction, and early stopping unchanged. Then compare QWK rankings and attention concentration (top-3 cumulative attention, Fig. 5a). If the supervised/DINOv1 advantage over MAE persists at comparable magnitude under both protocols, the pretraining-objective claim stands; if rankings change or the gap collapses, the reported differences are partly pooling artifacts and the abstract/conclusion need qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central inference—that pretraining objective, not another factor, drives the frozen-transfer gap—requires the comparisons in Section III.C.2 to vary only the encoder. They do not: for the three controlled ImageNet ViT-B encoders, token pooling is chosen per encoder (CLS for ViT-Supervised and ViT-DINOv1; average over spatial tokens for ViT-MAE), and DINOv3 also uses average pooling. Appendix C acknowledges this, but the sensitivity analysis in Appendix Table I only covers ViT-MAE and ViT-DINOv3. It shows MAE with CLS (0.4258 QWK) remains far below DINOv3 (0.8633), so the broad 'MAE is worse' conclusion is plausibly robust; however, it does not test ViT-Supervised or ViT-DINOv1 under average pooling. Since the attribution to pretraining objective rests on the three-way controlled comparison, the missing pooling match is the weakest load-bearing point. The attention-concentration claims (Fig. 5) are also entangled with this choice, because CLS versus mean tokens feed different distributions into the attention pooling of Eq. (4).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates frozen Vision Transformer (ViT) encoders pretrained with supervised, masked autoencoding (MAE), and self-distillation (DINOv1/DINOv3) objectives inside a patch-based, attention-based multiple instance learning (MIL) framework for ultra-widefield retinal imaging. On the MMRDR five-class diabetic retinopathy grading task, supervised and DINO-based ViTs reach QWK around 0.84–0.86, while ImageNet-MAE reaches QWK 0.53; DINOv3 is best overall (QWK 0.8629). The authors also report binary DR and intraocular-tumour transfer results, attention-pooling statistics, UMAP visualisations, and partial fine-tuning experiments. The central claim is that the choice of pretraining objective substantially influences frozen representation transfer and patch-aggregation behaviour within MIL, with reconstruction-based MAE representations transferring worst.","tokens_in":18967,"tokens_out":8293,"duration_ms":107042,"significance":"The question addressed is practically important: many medical-imaging pipelines use frozen ImageNet or large-scale self-supervised ViTs as feature extractors, and the interaction between pretraining strategy and weakly supervised MIL aggregation is underexplored. The study's strengths are its use of public datasets and checkpoints, three-seed repeats, a second DR dataset, a cross-disease dataset, a pooling sensitivity analysis, and a candid limitations section. If the attribution to pretraining objective is supported, the result gives valuable guidance for encoder selection in high-resolution UWF imaging. The main weakness is that the controlled ImageNet-ViT-B comparison confounds pretraining objective with the per-encoder token pooling convention, so the causal claim is not fully identified as stated.","major_comments":[{"comment":"The controlled ImageNet ViT-B comparison confounds pretraining objective with token pooling. ViT-Supervised and ViT-DINOv1 use CLS-token features, while ViT-MAE uses average pooling over spatial tokens (Appendix C). Thus the large MMRDR gap (DINOv1 QWK 0.8602 vs MAE 0.5325, Table II) is not attributable to pretraining objective alone. Appendix Table I provides a sensitivity analysis only for ViT-MAE and ViT-DINOv3; it does not test ViT-Supervised or ViT-DINOv1 under average pooling, which are two of the three encoders in the central controlled comparison. Because the MIL attention mechanism in Eq. (4) receives different input distributions for CLS vs average-pooled features, the Fig. 5 attention-concentration claims are entangled with the same confound. The MAE disadvantage is plausibly robust (DINOv3 with CLS is also strong), but the causal attribution to pretraining objective requires","section":"III.C.2, Appendix C, Table II, Eq. (4)"},{"comment":"The abstract and conclusion state that supervised and self-distillation-based models outperform MAE, but Table II shows that on the DeepDRiD binary DR task ViT-MAE(ImageNet) achieves AUROC 0.8131, essentially tied with ViT-DINOv1 (0.8136). The ordering is task-dependent, as the Discussion itself acknowledges. The summary-level generalisation overstates the MMRDR-specific ranking and should be qualified, e.g., 'on five-class DR grading' or 'in most settings evaluated here.'","section":"Abstract and Section IV.A.2, Table II"}],"minor_comments":[{"comment":"The representation-strategy comparison uses a reduced protocol (instance dropout disabled, patience 5, single run) as stated in Section III.C.1. The conclusion that attention pooling is the 'most informative' strategy would be stronger with multiple seeds or a caveat that the comparison is preliminary.","section":"Section IV.A.1 / Table I"},{"comment":"The intraocular tumour dataset has no patient-level identifiers, so the same eye may appear across cross-validation folds. The authors disclose this in Limitations, but the potential for inflated AUROC should also be flagged where the tumour results are first reported in Table II, since the paper uses them as cross-disease validation.","section":"Section VI / Table II"},{"comment":"Consider adding a reproducibility statement with a code repository link or explicit instructions for reproducing the checkpoints, preprocessing, and MIL training loop. The current text names timm and torchvision but no project URL.","section":"Appendix B"},{"comment":"The two positional encodings (the encoder's internal ViT patch-position encoding and the added 5x5 bag-level sine-cosine grid encoding) are described only briefly. A sentence clarifying that the bag-level grid is added after encoder feature extraction would reduce potential confusion.","section":"Section III.B.1"},{"comment":"The quantitative attention statistics are computed from unnormalised softmax weights before min-max visualisation. Please specify how many test images are aggregated and whether the reported values are per-image means or medians, to make the figure reproducible.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the primary MMRDR result is likely robust. The main obstacle is the pooling confound in the 'controlled' comparison: the manuscript should either add the missing CLS-vs-average-pooling rows for ViT-Supervised and ViT-DINOv1, or substantially soften the causal language. The abstract also overstates the MAE disadvantage on DeepDRiD. I recommend major revision rather than rejection because the core empirical finding is defensible and the requested experiments are feasible with the same public checkpoints."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here are my notes on 2608.00586.\n\nThe real contribution is a controlled, fixed-framework comparison of frozen ViT encoders (supervised, MAE, DINOv1, DINOv3) inside an attention-MIL pipeline for UWF retinal images, evaluated on three public datasets. That is genuinely missing from the literature, and the main result — DINOv1 and supervised ViT-B beat MAE by about 0.33 QWK on five-class MMRDR grading — is large enough to be practically meaningful. The attention analysis, showing near-uniform MIL attention for MAE, and the partial fine-tuning results (which close much of the gap) are also informative. The limitations section is unusually candid, and the use of public checkpoints and datasets makes the core numbers reproducible.\n\nThe soft spots are real but neither is fatal. First, the controlled comparison varies pooling strategy along with pretraining objective: CLS for supervised and DINOv1, average pooling for MAE and DINOv3. The appendix sensitivity analysis covers only MAE and DINOv3. For MAE, CLS is worse than average; for DINOv3, CLS is slightly worse. So the main conclusion for the five-class task is probably robust, but the attribution to pretraining objective rather than pooling is not fully identified, and the attention-behaviour claims inherit the same entanglement. Second, the abstract claims supervised and self-distillation models 'outperform' MAE without qualification. That is false on DeepDRiD, where DINOv1 and MAE are essentially tied. The paper's own discussion acknowledges the ranking shifted, but the abstract does not. Minor concerns: the tumour dataset lacks patient-level split (acknowledged), no code is released, and the AlzEye checkpoint is not public.\n\nThe central argument holds up for the primary five-class task, which is where the effect is largest. If you are choosing an encoder for a frozen UWF MIL pipeline, this paper gives useful, honest evidence. It deserves a serious referee. My recommendation: send it to review with a request for matched-pooling experiments on all four encoders, a softened abstract, and code release.","headline":"A useful benchmark for frozen encoder choice in UWF MIL, with a large DINO/supervised vs MAE gap on five-class DR, but the pooling confound and an overreaching abstract keep it from being a clean accept.","tokens_in":19607,"tokens_out":2731,"would_cite":true,"duration_ms":31414,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that in frozen patch-based MIL for UWF retinal images, the pretraining objective matters: supervised and self-distillation ViTs transfer much better than MAE, with DINOv3 and DINOv1 reaching QWK ≈ 0.86 versus MAE's 0.53, a","keywords":["ultra-widefield retinal imaging","diabetic retinopathy grading","multiple instance learning","vision transformer","self-supervised pretraining","masked autoencoder","DINO self-distillation","representation transfer"],"falsifier":"On the same MMRDR MIL setup, repeat the comparison with identical pooling (e.g., average pooling over all patch tokens) and identical preprocessing for every encoder. If MAE's QWK rises from 0.53 toward the supervised/DINO range while the others stay near 0.86, the claimed pretraining-objective effect is largely a pooling or preprocessing artifact; if MAE stays near 0.53, the effect is real.","tokens_in":1727,"feed_emoji":"👁️","tokens_out":3308,"duration_ms":109443,"temperature":0.7,"pith_summary":"The paper asks why one foundation model might work better than another when used as a frozen feature extractor for medical images. In a controlled patch-based multiple instance learning (MIL) setup on 10,404 ultra-widefield retinal images, it compares ViT-B encoders pretrained on ImageNet with supervised, masked autoencoder (MAE), and self-distillation objectives. The result: supervised and self-distillation (DINOv1) encoders vastly outperform MAE for five-class diabetic retinopathy grading (QWK 0.84–0.86 vs 0.53), and the same ranking appears on two additional UWF tasks. A larger-scale DINOv3 model performs best overall (QWK 0.863) but statistically ties DINOv1. Partial fine-tuning of the last encoder block nearly erases MAE's disadvantage, showing the gap is specific to frozen transfer.","feed_headline":"DINO and supervised ViTs beat MAE by 0.33 kappa on frozen UWF grading","feed_subtitle":"Frozen ImageNet pretraining objectives change MIL-based diabetic retinopathy grading, and fine-tuning the last block closes most of the gap.","key_machinery":"The load-bearing setup is patch-based MIL with frozen encoders: each UWF image is tiled into a 5×5 grid of 224×224 patches, each patch is embedded by a pretrained ViT, a fixed 2D positional encoding is added, and an attention-based MIL pooling layer (Ilse et al.) aggregates patch embeddings into a bag representation for classification. The encoders are the only thing varied; the comparison isolates pretraining objective (supervised, MAE reconstruction, self-distillation) by using ViT-B ImageNet checkpoints, plus DINOv3 and a domain-specific MAE. The feature pooling per encoder (CLS vs average) is chosen according to the pretraining objective, which is the mechanism's main uncontrolled knob.","core_discovery":"Within one fixed MIL pipeline—25 overlapping 224×224 patches per image, attention-based aggregation, frozen encoder—the choice of pretraining objective changes both the quality and the behavior of transferred representations. On MMRDR, ViT-MAE(ImageNet) achieves QWK 0.5325 while ViT-Supervised reaches 0.8396, ViT-DINOv1 0.8602, and ViT-DINOv3 0.8629, a margin the paper attributes to the objective rather than architecture, capacity, or downstream module. The MIL attention also diverges: MAE features yield near-uniform attention, effectively reducing attention pooling to mean pooling, while supervised and DINO features concentrate weight on central retinal patches. Unfreezing only the final en","pith_inferences":["Because pooling strategy is not held constant across objectives, a fair re-test would use identical token pooling for all encoders; the paper's own sensitivity data for MAE and DINOv3 show pooling can move results by roughly 0.1 QWK, which is smaller than the objective gap but not negligible.","The common grayscale preprocessing and 1024×1024 downsampling may favor some pretraining objectives over others, so the ranking should be rechecked at native Optomap resolution and with each checkpoint's native preprocessing.","The DINOv1-versus-DINOv3 tie hints that for medical patch features, the marginal value of larger self-supervised pretraining data is small relative to the objective family, perhaps because the downstream task rewards fine-grained local discriminability more than global semantic richness.","The attention statistics themselves could serve as a cheap diagnostic for whether a frozen representation is usable in MIL: near-uniform attention weights may indicate the encoder's patch features are not locally discriminative enough."],"forward_implications":["For frozen-feature medical imaging pipelines, selecting a supervised or self-distillation pretrained ViT over a MAE encoder can be worth about 0.3 QWK in DR grading before any fine-tuning.","Reconstruction-based encoders should not be dismissed: with a single unfrozen block, MAE(ImageNet) reaches frozen-supervised-level performance, suggesting a fine-tuning budget can substitute for pretraining choice.","MIL attention behavior is representation-dependent: MAE collapses to uniform weights, so attention-based MIL confers little benefit with such features.","Patch-based attention aggregation beats global and max pooling for the same encoder, supporting the use of high-resolution patch evidence in UWF imaging.","DINOv1 and DINOv3 perform comparably despite DINOv3's much larger pretraining data, so scale alone did not translate into a large frozen-transfer gain."],"supporting_citations":[{"why":"Defines the MAE reconstruction-based pretraining objective that supplies the ViT-MAE baseline.","marker":"[5]"},{"why":"Defines the ViT architecture and supervised ImageNet pretraining that supplies the supervised baseline and backbone.","marker":"[4]"},{"why":"Supplies the DINOv1 self-distillation ImageNet encoder used as a controlled self-distillation baseline.","marker":"[6]"},{"why":"Supplies the DINOv3 large-scale self-distillation encoder that achieves the strongest overall performance.","marker":"[7]"},{"why":"Supplies the MMRDR dataset, the primary five-class UWF DR benchmark whose test set drives the main comparisons.","marker":"[18]"},{"why":"Supplies the attention-based MIL pooling module that aggregates patch embeddings and produces the attention maps.","marker":"[30]"},{"why":"Supplies DeepDRiD for the binary referable-DR validation task.","marker":"[27]"},{"why":"Supplies the intraocular tumour UWF dataset for cross-disease validation.","marker":"[28]"}],"fun_headline_variants":["Pretraining objective swings UWF grading kappa 0.53 to 0.86","MAE lags supervised and DINO in frozen UWF transfer","Frozen ViT choice alters MIL attention and grading","DINO and supervised ViTs beat MAE by 0.33 kappa"],"cache_read_input_tokens":21248,"weakest_assumption_plain":"The main comparison assumes that using each model's recommended patch-pooling method (CLS token for supervised and DINOv1, average pooling for MAE and DINOv3) does not itself create the performance gap; since only MAE and DINOv3 were tested with both, the objective effect could be partly a pooling effect.","fun_headline_variants_meta":{"raw":{"variants":["Pretraining objective swings UWF grading kappa 0.53 to 0.86","MAE lags supervised and DINO in frozen UWF transfer","Frozen ViT choice alters MIL attention and grading","DINO and supervised ViTs beat MAE by 0.33 kappa"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1117,"prompt_tokens":796,"completion_tokens":321,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":241}},"tokens_in":540,"tokens_out":321,"duration_ms":4776,"temperature":1.0,"reasoning_tokens":241,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T00:30:41.784001+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same MMRDR MIL setup, repeat the comparison with identical pooling (e.g., average pooling over all patch tokens) and identical preprocessing for every encoder. If MAE's QWK rises from 0.53 toward the supervised/DINO range while the others stay near 0.86, the claimed pretraining-objective effect is largely a pooling or preprocessing artifact; if MAE stays near 0.53, the effect is real.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MAE reconstruction-based pretraining objective that supplies the ViT-MAE baseline."},{"cited_title":"Dosovitskiyet al., ‘‘An image is worth 16×16 words: Transformers for image recognition at scale,’’ inProc","cited_arxiv_id":null,"evidence_quote":"Defines the ViT architecture and supervised ImageNet pretraining that supplies the supervised baseline and backbone."},{"cited_title":"Tanget al., ‘‘A multimodal retinal image dataset for diabetic retinopathy detection using foundation models,’’Sci","cited_arxiv_id":null,"evidence_quote":"Supplies the MMRDR dataset, the primary five-class UWF DR benchmark whose test set drives the main comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the attention-based MIL pooling module that aggregates patch embeddings and produces the attention maps."},{"cited_title":"Sunet al., ‘‘An ultra-wide-field fundus image dataset for intelligent diagnosis of intraocular tumors,’’Sci","cited_arxiv_id":null,"evidence_quote":"Supplies the intraocular tumour UWF dataset for cross-disease validation."}],"review_version":1}