{"id":"c55de366-1327-4e83-a59f-00cad83e1080","arxiv_id":"2509.00226","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuned Vision Transformers and MLP-Mixer models reach lens-detection accuracy comparable to convolutional baselines on the common test sample from More et al. (2024).","lead":"A team tested ten pretrained AI image models, including Vision Transformers and an MLP-Mixer, for spotting gravitational lenses in sky images. These models match older convolutional networks on a shared benchmark, making them a practical choice for future large surveys.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best-per-test-set selection and missing uncertainty quantification undermine the statistical basis for the claim that ViT/MLP-Mixer models match or exceed CNNs; the L2 recall difference is one image out of 138.","rationale":"The reader's weakest assumption about shared simulation provenance is valid and the paper itself acknowledges it in Section 6 ('broad generalization of conclusions is a challenge'). That issue matters for interpreting the near-perfect scores on L3/L4 mocks as simulator recognition rather than genuine generalization to new real lenses. However, the more load-bearing problem is that the headline comparisons are statistically ungrounded even for the real-lens subsets. Table 3 and Table 4 select the best of many configurations without error bars, so the reported 'best' AUC/recall values are inflated and the differences against More et al. (2024) are not meaningful. The most striking example is the L2 recall: 132/138 vs 131/138 is a one-image difference, well within the binomial sampling noise, and the absence of a stated decision threshold makes it unreproducible. This concern directly attacks the central claim ('comparable to or exceeding CNN values') and is actionable: adding confidence intervals, fixing thresholds, and accounting for multiple comparisons could either rescue or refute the claim. I therefore keep the reader's CONDITIONAL verdict: the work is transparent, reproducible, and useful as a benchmark, but the central comparative claim requires additional statistical support. Agreement is partial because I locate the weakest point in the selection/uncertainty methodology rather than in the mock-test provenance, although the reader's rationale already lists both issues.","tokens_in":18918,"tokens_out":7516,"duration_ms":89639,"concrete_test":"Re-run L2 inference for all ten architectures in the A2 and C3 settings using a fixed decision threshold (e.g., 0.5) and record per-architecture recall. Compute 95% Clopper-Pearson confidence intervals for each recall value, and bootstrap (10,000 resamples) the max-recall-over-architectures statistic for A2 and C3 to obtain the distribution of the difference. Apply McNemar's test to the 138 paired predictions between the best A2 model and the best C3 model. If the bootstrap 95% CI for the max-recall difference includes 0, or McNemar's p>0.05, then the claimed superiority of the A2 MLP-Mixer over the C3 ResNet-18 is not supported. Report the threshold and raw counts for full reproducibility.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim rests on Tables 3 and 4, but both tables report maxima over many configurations without any uncertainty quantification. Table 3 selects, for each test set, the best AUC-ROC among ten architectures and twelve training/fine-tuning settings (A/B/C/S × 1/2/3), i.e., ~120 candidate models. Reporting the maximum of such a grid as 'the model's performance' creates an upper envelope with substantial winner's bias; it cannot be compared against a single reference CNN as if it were a fixed model. Table 4 applies the same selection: for each experiment (A1–S3), the recall on the 138-image L2 subset is the maximum over ten architectures. The headline A2 MLP-Mixer result (132/138, 95.65%) is just one image more than the C3 ResNet-18 result (131/138, 94.93%). With n=138, the standard error of a proportion near 0.95 is about 1.9 percentage points (~2.6 images), and the max-of-10 statistic is upwardly biased, so a one-image gap is fully consistent with chance. In addition, the decision threshold used for L2 'detections' is never stated, making the recall values unreproducible. Because the broad claim that ViT/MLP-Mixer models are comparable to or better than CNN baselines is built on this unquantified best-of-many selection and on differences that are smaller than the sampling noise, the central claim is not statistically established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents GraViT, a PyTorch pipeline for transfer learning applied to strong gravitational lens detection. The authors fine-tune ten pretrained vision-transformer variants (ViT, DeiT, CaiT, DeiT III, Swin, Twins-SVT, Twins-PCPVT, PiT, CvT) and MLP-Mixer on two training datasets (C21 and J24, and their combination, plus a reduced C21 set), and evaluate on the common test sample defined by More et al. (2024). Results are reported as AUC-ROC and F1 for 12 test-set configurations, with additional inference on the L2 subset of real candidates. The paper claims performance comparable to or exceeding convolutional baselines, discusses ensemble predictions, and provides inference-time complexity analysis. The code and fine-tuned models are made publicly available.","tokens_in":19324,"tokens_out":8332,"duration_ms":100655,"significance":"If the empirical claims were statistically robust, this would be a useful contribution to the growing literature on machine-learning-based lens finding, especially for LSST: it provides a reproducible transfer-learning pipeline, public models, and a comparison across many transformer architectures on a shared benchmark. The authors also reproduce and extend the systematic CNN comparison of More et al. (2024), which is valuable for the community. However, the central comparative claim is weakened by the absence of uncertainty quantification and by a best-of-many selection procedure that makes the headline numbers difficult to interpret. The paper also includes an internal inconsistency in the list of evaluated architectures. These issues are fixable and do not invalidate the overall approach, but they currently limit the strength of the conclusions that can be drawn.","major_comments":[{"comment":"The list of evaluated architectures is inconsistent. Section 4 states that ten architectures are evaluated (ViT, DeiT, CaiT, DeiT III, Swin, Twins-SVT, Twins-PCPVT, PiT, CvT, MLP-Mixer), yet Tables 3 and 4 report results for 'ResNet-18' as a trained model in several experiments (e.g., Table 3 d and i; Table 4 C3). It is therefore unclear whether ResNet-18 is an additional evaluated architecture, a reference model, or an error. This ambiguity affects the definition of the candidate set over which the 'best' results are selected and must be clarified before the results can be reproduced.","section":"Section 4; Tables 3 and 4"},{"comment":"The 'best model' entries in Table 3 are maxima over a large grid: 10 architectures (or 11 including ResNet-18) × 12 training/fine-tuning settings, plus ensemble variants. Reporting the maximum of such a grid as the model's performance and comparing it against a single reference value introduces substantial winner's bias. Many entries are actually below the More et al. (2024) values (e.g., set a: 0.97 vs 0.98; set c: 0.96 vs 0.99; set l: 0.92 vs 0.94), so the statement in §5.1 that results are 'comparable to or exceeding the state of the art' is not strongly supported by the table. The authors should either report means and standard deviations over multiple seeds, apply a proper model-selection rule (e.g., validation-based), or explicitly frame Table 3 as an upper envelope rather than a fixed-model comparison.","section":"Section 5.1, Table 3"},{"comment":"The L2 inference results are statistically fragile. The difference between the top recall (A2 MLP-Mixer, 132/138) and the second-best (C3 ResNet-18, 131/138) is a single image. For n=138, the standard error of a proportion near 0.95 is about 2 percentage points, so this gap is fully consistent with sampling noise. Moreover, the values in Table 4 are maxima over ten architectures for each experiment, further increasing the selection bias. In addition, the decision threshold used to define a 'detection' is never stated; without this threshold the recall numbers are not reproducible. The authors should specify the threshold, provide confidence intervals (e.g., Wilson intervals), and report results from multiple training seeds for at least the key configurations.","section":"Section 5.2, Table 4"},{"comment":"The common test sample contains mock lens sets L3 and L4 that are generated by the same simulation pipelines used to build the C21 and J24 training data. Consequently, the near-perfect AUC-ROC values on test sets g-k (and the fact that 6000 of the 6181 lenses in set l come from L3/L4) largely measure a model's ability to recognize its own training simulator rather than generalization to new real lenses. The authors do acknowledge provenance dependence in the Conclusions and Appendix C, but the abstract and §5.1 frame the results more broadly. The only independent held-out evidence is the 138-image L2 subset and the L1/L2 combinations in sets a-f. The comparative claims should be explicitly scoped to these independent subsets, and the wording 'state of the art' should be tempered accordingly.","section":"Section 2.5 and Section 5.1; Table 1"}],"minor_comments":[{"comment":"The abstract states that 'the MLP-Mixer ensemble with C3 reaches 0.92 on the fully integrated test set l', but Table 3 lists the MLP-Mixer (not the ensemble) for C3. The ensemble AUC for l in Table C1 is 0.88. Please clarify whether the 0.92 refers to the individual MLP-Mixer or to an ensemble.","section":"Abstract and Section 5.1"},{"comment":"The metrics definitions (§4.1) do not specify the probability threshold used to compute F1 or to count 'detections' in Table 4. A threshold (e.g., 0.5 or the validation-optimal threshold) should be clearly stated, as different thresholds can change the recall values substantially.","section":"Section 4.1"},{"comment":"The text says that marker sizes in Figure 10 are scaled by 'mean AUC-ROC', but the figure description does not specify over which test sets or experiments the mean is computed. Please state the averaging domain (e.g., all test sets a-l for the C3 experiment).","section":"Figure 10"},{"comment":"The sentence 'any model not included (reference ResNet-18) simply does not contribute to the normalization factor N' is confusing. If ResNet-18 is a reference and not part of the ensemble, this is fine, but it again highlights the need to clarify the full set of trained models.","section":"Section 4.2"},{"comment":"The ensemble tables C1 and C2 report a single value per cell with no measure of variability. Even if the authors do not run multiple seeds, a statement to that effect and a caution that these are single-run results would aid interpretation.","section":"Table C1/C2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal as a machine-learning application in astronomy. The reported pipeline and public release of models are positive contributions. However, the statistical issues (best-of-many selection, no uncertainty quantification, threshold ambiguity) are load-bearing for the central comparative claim, and the architecture-list inconsistency must be fixed. With a revision that addresses these points and narrows the generalization claims to the independent subsets, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent and honest transfer-learning benchmark for strong lens finding, but the central comparative claim is fragile. The paper ships code, fine-tuned models, and full result tables, which is real value for practitioners. It deserves a serious referee, but the tables need reworking and the interpretation of mock-test scores needs to be reframed.\n\nWhat's new: a clean grid of ten ImageNet-pretrained transformer/mixer models, fine-tuned at three depths on two public training sets and their union, evaluated on the More et al. common test sample. The Appendix tables are a genuinely useful resource, and the complexity analysis (params vs FLOPs) is practical and clear. The training pipeline is described in enough detail to reproduce, and the authors are transparent about data provenance, explicitly noting that J24-trained models dominate on L4-based test sets.\n\nThe soft spots are real. Tables 3 and 4 pick, for each test set, the best score across ten architectures times twelve training/fine-tuning configurations—about 120 models. That is an upper envelope with winner's bias, and there are no error bars or seeds. The L2 recall difference that makes the headline (132/138 vs 131/138) is one image out of 138, well within sampling noise. More importantly, the L3 and L4 mock test sets share simulation pipelines with the C21 and J24 training sets, so near-perfect AUC on sets g–k mostly measures a model's ability to recognize its own training simulator, not generalization. The only genuinely independent lens evidence is the 138-image L2 subset, where a ResNet-18 baseline is within one image of the best MLP-Mixer. There is also a small internal inconsistency: ResNet-18 appears in Tables 3–4 but is not listed among the ten architectures in Section 4.\n\nThe paper's own closing sentence is honest—\"broad generalization of conclusions is a challenge\"—but the abstract over-promises. With error bars, a correction for multi-selection, and an explicit statement of the L2 detection threshold, the comparative claim could be made honestly. Without those, the main takeaway is not \"transformers win,\" but \"a transfer-learning grid works about as well as the existing CNN baseline, with some architecture-specific trade-offs.\"\n\nWho this is for: astro-ML practitioners who want a ready-made recipe and a benchmark sanity check for LSST-scale lens finding. It will not change physics, but as an empirical contribution it is useful. I would send it to peer review, with a clear request for uncertainty quantification and a more careful presentation of the mock-test results. If the authors address those, the paper is a solid addition to the lens-finding toolkit.","headline":"Useful transfer-learning benchmark for lens finding, but the headline claim that ViTs/MLP-Mixers match or beat CNNs is not statistically established.","tokens_in":19775,"tokens_out":4244,"would_cite":false,"duration_ms":50866,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned ViT and MLP-Mixer models identify strong gravitational lenses at levels comparable to or better than convolutional baselines on a common test sample, with one MLP-Mixer recovering 132 of 138 real lens candidates.","keywords":["strong gravitational lensing","vision transformer","MLP-Mixer","transfer learning","image classification","ensemble prediction","HSC survey","LSST"],"falsifier":"Take the A2 MLP-Mixer that recovered 132/138 L2 candidates and run it on a sample of confirmed strong lenses drawn from surveys or simulations not used in C21 or J24, matched to HSC-like imaging. If recall on that independent sample is far below 95.65 percent while mock-set recall stays near perfect, the transfer-learning gain is largely simulator recognition, not general lens detection.","tokens_in":18909,"feed_emoji":"🔭","tokens_out":8354,"duration_ms":87447,"temperature":0.7,"pith_summary":"This paper tries to establish that transfer learning from large-scale pretrained image models is enough to make Vision Transformers and MLP-Mixers competitive with—and in places better than—the convolutional networks that have dominated strong gravitational lens search. It reproduces the evaluation protocol of a previous systematic CNN comparison, fine-tuning ten architectures on two existing HSC-based training sets and testing them on the same twelve test sets. On most single-class test sets the fine-tuned transformers reach AUC-ROC and F1 scores comparable to or above the CNN references, and the best integrated-test configuration reaches 0.92 AUC-ROC against the 0.94 reference. The clearest independent evidence is the held-out set of 138 real lens candidates, of which a partially fine-tuned MLP-Mixer recovers 132. A key scope condition, which the paper itself acknowledges, is that high scores on the mock-rich test sets largely reflect each model learning the simulation pipeline it was trained on, so broad generalization claims rest on the small real-candidate sample.","feed_headline":"MLP-Mixer finds 95.65% of real lens candidates","feed_subtitle":"Fine-tuned transformers match CNN baselines on the shared test set and scale linearly to LSST data.","key_machinery":"The load-bearing mechanism is the transfer-learning pipeline: HSC gri cutouts are remapped to RGB, resized to 224×224, normalized with ImageNet statistics, and fed to models pretrained on ImageNet; fine-tuning is applied at one of three depths (head only, half the layers, or all layers). The evaluation machinery is the common test sample of twelve sets built from real lenses, real candidates, and mocks. The paper's distinctive architecture is the MLP-Mixer, which alternates token-mixing and channel-mixing MLPs, has O(n) cost in the number of image patches, and achieves the best L2 recovery and the best integrated-set score among the newly evaluated models.","core_discovery":"The central claim, on the paper's own terms, is that fine-tuning pretrained transformer-style models is a viable and scalable route to strong lens discovery. On the common test sample, fine-tuned ViT/MLP-Mixer models achieve AUC-ROC and F1 scores comparable to or exceeding the CNN values reported by the reference comparison for most single-class test sets and for the real-candidate subset L2. Only on the fully integrated test set l does the best new configuration, an MLP-Mixer trained with full fine-tuning on combined data, fall slightly below the 0.94 reference, reaching 0.92. The authors read this as evidence that global-context architectures boosted by large-scale pretraining can capture","pith_inferences":["I read the near-perfect scores on the L3 mock sets as measuring how well a model recognizes its own training simulator; the only clean evidence about real lenses in this paper is the 138-image L2 subset, so the comparative claim should be read with that scope.","A decisive next experiment would be to run the same fine-tuned models on confirmed lenses from an independent survey with comparable image quality and compare recall with the 95.65 percent L2 figure.","For LSST, a practical extension would be to increase MLP-Mixer patch size (its cost grows linearly, unlike self-attention) and train on a mix of current mocks plus real candidates, then use the soft-voting ensemble as a prescreener for human vetting.","The S experiment's finding that halving the C21 training sample barely changes performance suggests diminishing returns from adding more mocks of the same pipeline; diverse real data, not more simulation volume, is the likely bottleneck."],"forward_implications":["Fine-tuned transformer and mixer models can replace CNNs in lens-search pipelines without a loss in detection quality on matched simulated data.","Partial fine-tuning often matches or exceeds full fine-tuning, so lens classifiers can be adapted with modest training budgets.","The MLP-Mixer's linear cost scaling makes it a natural fit for the large cutout volumes expected from LSST.","Models trained on one simulation pipeline fail on lenses produced by another, so survey-ready classifiers should be trained on the simulation matching the target survey, or on combined data.","Soft-voting ensembles of diverse architectures improve robustness on the hardest non-lens sets and can be formed without retraining the individual models."],"supporting_citations":[{"why":"Builds the common test sample and supplies the CNN benchmark values that define the comparison.","marker":"More et al. (2024)"},{"why":"Provides the C21 training set of 40,000 simulated lenses and the ResNet-18 classifier that acts as a key baseline.","marker":"Cañameras et al. (2021)"},{"why":"Provides the J24 training set and the SuGOHI X pipeline whose lenses define the L4 test sets.","marker":"Jaelani et al. (2024)"},{"why":"Introduces the Vision Transformer, whose pretrained checkpoints are fine-tuned in this work.","marker":"Dosovitskiy et al. (2021)"},{"why":"Introduces MLP-Mixer, the architecture with the best L2 recovery and best integrated-set score in this study.","marker":"Tolstikhin et al. (2021)"},{"why":"A CNN baseline trained on J24; the paper reproduces its experimental setting as experiment B.","marker":"Ishida et al. (2024)"},{"why":"Supplies the N3 non-lens test set and another CNN baseline that appears in the common-sample comparison.","marker":"Shu et al. (2022)"},{"why":"The ImageNet-21k pretraining recipe for the Google ViT and MLP-Mixer checkpoints used in fine-tuning.","marker":"Ridnik et al. (2021)"}],"fun_headline_variants":["ViT and MLP-Mixer match CNNs on strong lens search","Pretrained transformers rival CNNs in lens discovery","Transfer learning lifts transformer models for lens finding","Fine-tuned ViTs find real lens candidates as well as CNNs"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the common test sample measures real generalization: because the L3 and L4 mock sets come from the same simulation pipelines as the training sets, near-perfect scores on those sets may only show each model recognizing its own simulator, leaving the 138-image L2 real-candidate set as the main evidence for genuine lens-finding ability.","fun_headline_variants_meta":{"raw":{"variants":["ViT and MLP-Mixer match CNNs on strong lens search","Pretrained transformers rival CNNs in lens discovery","Transfer learning lifts transformer models for lens finding","Fine-tuned ViTs find real lens candidates as well as CNNs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1214,"prompt_tokens":721,"completion_tokens":493,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":425}},"tokens_in":465,"tokens_out":493,"duration_ms":6416,"temperature":1.0,"reasoning_tokens":425,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:48:19.627102+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the A2 MLP-Mixer that recovered 132/138 L2 candidates and run it on a sample of confirmed strong lenses drawn from surveys or simulations not used in C21 or J24, matched to HSC-like imaging. If recall on that independent sample is far below 95.65 percent while mock-set recall stays near perfect, the transfer-learning gain is largely simulator recognition, not general lens detection.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Builds the common test sample and supplies the CNN benchmark values that define the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces MLP-Mixer, the architecture with the best L2 recovery and best integrated-set score in this study."},{"cited_title":"C., Jaelani A","cited_arxiv_id":null,"evidence_quote":"A CNN baseline trained on J24; the paper reproduces its experimental setting as experiment B."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The ImageNet-21k pretraining recipe for the Google ViT and MLP-Mixer checkpoints used in fine-tuning."}],"review_version":1}