{"id":"e661cbf5-6b9b-40f6-bda6-240a990e06e4","arxiv_id":"2412.11696","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CNN classifiers using EfficientNetV2M and a Zoobot-backed EfficientNetB0 label spiral arm number from Galaxy Zoo 2 images with F1 above 0.8 for most classes, but fail on the rare four-arm class.","lead":"The paper trains two EfficientNet-based neural networks to count the spiral arms of galaxies using 11,718 Galaxy Zoo 2 images, reporting good accuracy for one-, two-, and three-arm galaxies and poor results for four-arm galaxies. A generalist might read it because such classifiers could automate arm counting for upcoming billion-galaxy surveys and because the authors connect arm number to stellar mass.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rare-class validation sets are too small for the reported metrics to be meaningful: m=4 has ~9 validation images, so the headline 'high accuracy except m=4' is not statistically supported.","rationale":"I read the paper in good faith. The architecture, transfer-learning setup, and preprocessing are standard; the comparison between ImageNet fine-tuning and Zoobot feature extraction is reasonable, and the paper correctly identifies the m=4 bottleneck as a sample-size problem. The load-bearing weakness is not the algorithm but the evaluation protocol for rare classes. The reader's weakest assumption names label noise and the small 70:30 split; I focus on the small-split statistical uncertainty because even with perfectly clean labels, the reported metrics for m=4 and m=5+ are single draws from a tiny binomial sample, so the quantitative claims in the abstract and Section 5.1 are not reproducible without error bars or repeated splits. This does not change the CONDITIONAL verdict: it strengthens the conditions under which acceptance would be appropriate (report confidence intervals, add repeated-seed experiments, or validate on an external sample with more rare-class objects).","tokens_in":22847,"tokens_out":4563,"duration_ms":39677,"concrete_test":"Obtain the validation predictions for m=4 and m=5+ and compute Wilson 95% confidence intervals for precision, recall, and F1; then re-run the same training configuration with 100 different 70:30 random seeds and report the mean and 95% CI of F1 per class. If the m=4 F1 interval spans ±0.2 or the recall interval includes 0.5, the headline claim that performance is high except for m=4 needs revision to acknowledge that rare-class metrics are unconstrained. A simpler alternative: report the confusion matrix counts (e.g., 2/9) alongside metrics and state the effective validation n per class.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — both models achieve high accuracy with most metrics >0.8 except m=4 — rests on point estimates computed from 70:30 splits of a highly imbalanced dataset. With N_m=4=28 and N_m=5+=31, the validation sets contain only ~8–10 images for each of these two rarest classes. The reported m=4 recall of 0.222 (Table 3) corresponds to ~2 correct out of 9; flipping a single validation image changes recall by 0.11 and F1 by roughly 0.15. No error bars, repeated-seed statistics, or cross-validation are given, so these point estimates are single draws from a high-variance binomial. The issue is precisely where the paper admits weakness: the abstract's 'except m=4' clause concedes the class where the sample is smallest, yet the same statistical fragility applies to m=5+ (V2M precision 0.769, also below the 0.8 threshold) and to the class-combination result: the merged m=4+5+ class has ~18 validation images, so the V2M recall of 0.842 is 16/19 correct with a 95% Wilson interval of roughly (0.60, 0.95). The paper's own conclusion states that 'further improvements are needed for accurate categorisation of high-arm galaxies,' which is consistent with this reading. Because the headline metrics for the rare classes are dominated by the chance composition of one split, the central claim that the classifiers are 'high accuracy' for anything beyond the m=2 majority class is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper trains two CNN classifiers (EfficientNetV2M fine-tuned on ImageNet, and EfficientNetB0 with frozen Zoobot weights) to classify galaxy images from Galaxy Zoo 2 into spiral arm numbers m=1, 2, 3, 4, 5+ and 'can't tell'. The dataset is filtered with GZ2 vote-fraction cuts (Section 2.1) and down-sampled to balance classes. The authors report precision/recall/F1 in Table 3, class-combination experiments with confusion matrices (Figure 7), Grad-CAM++ and SmoothGrad visualizations, and a comparison of stellar mass distributions between GZ2 labels and model predictions (Section 6). They conclude that both models achieve high accuracy except for m=4, and that merging m=4 and m=5+ improves V2M performance.","tokens_in":23002,"tokens_out":6974,"duration_ms":58893,"significance":"The paper's strengths are its detailed and conservative dataset selection, transparent reporting of confusion matrices, and direct comparison of two transfer-learning strategies. If the performance numbers are robust, the classifiers would be a useful, scalable tool for arm-number counting in upcoming surveys. However, the central accuracy claims for the rare classes m=4 and m=5+ are statistically fragile because they rest on roughly 8-19 validation images without uncertainty quantification, and the B0 comparison is complicated by the Zoobot backbone having been pre-trained on the same GZ decision-tree question. These issues are fixable but currently limit the strength of the conclusions.","major_comments":[{"comment":"The headline metrics for the two rarest classes are point estimates from a single 70:30 split with no uncertainty quantification. With 28 m=4 and 31 m=5+ images in the full dataset, the validation sets contain roughly 8-10 images; the reported m=4 recall of 0.222 therefore corresponds to about 2 correct of about 9 images, so a single image changes recall by ~0.11 and F1 by ~0.15. The m=5+ precision of 0.769 for V2M is also below the stated 0.8 baseline. Please report per-class validation counts and either Wilson/binomial confidence intervals or repeated-split / k-fold distributions for all metrics; without this, the abstract's 'except m=4' clause rests on a statistically fragile class.","section":"Section 5.1, Table 3"},{"comment":"The class-combination result is likewise based on about 19 validation images for the merged m=4+5+ class. V2M recall 0.842 corresponds to 16/19 correct, and the 95% Wilson interval is approximately (0.60, 0.95); the '280% improvement' quoted later is therefore not a precisely determined quantity. Please report raw counts for the merged class and include confidence intervals, and state explicitly that the improvement is in relative recall rather than accuracy.","section":"Section 5.2, Figure 7"},{"comment":"The B0 model uses a frozen Zoobot backbone that was pre-trained to predict GZ decision tree responses, including the arm-count question (T11/Q11), so its high F1 on that same question may largely reflect transfer from pre-training rather than a newly learned classifier. This is a correctness risk for the B0 vs. V2M comparison, not an assertion of intentional leakage. A concrete control would be to compare B0 against an unfrozen EfficientNetB0, to report how the raw Zoobot model performs on this same validation split, or to quantify the overlap between the DECaLS training set used by Zoobot and the GZ2 main sample used here.","section":"Section 3.3, Section 5.1"},{"comment":"The stellar-mass analysis uses model predictions on 'our entire data set (including all training, validating and testing images)', so the predicted-class distributions include in-sample predictions on training data. This can bias the comparison between label and prediction distributions and weakens the claim that the mass tendencies are 'reduced in the model predictions'. Please recompute the predicted distributions using only the validation (or a nested) split, or justify why training-set contamination does not affect the conclusion.","section":"Section 6"}],"minor_comments":[{"comment":"There is a grammar issue: 'the spiral arm number, offer' should read 'the spiral arm number offers'.","section":"Abstract"},{"comment":"Typo: 'Number of hyparameters' should be 'Number of hyperparameters'.","section":"Table 2"},{"comment":"The reported coefficient of variation values 2.203 and 2.124 are inconsistent with F1-scores near 0.9 if CV is defined as sigma/mu in the standard sense; please check whether these are percentages or have a missing factor of 100, and define the convention explicitly.","section":"Section 4.2.1"},{"comment":"The text says 'A considerably high percentage of m = 3 galaxies are misclassified as m = 4 (67%)', but the confusion matrices show the opposite: 66.7% of m=4 galaxies are misclassified as m=3. The Figure 7 caption's 'Up to 22% of m = 3 galaxies are misclassified as m = 4' also does not match the shown values of 13.3% (B0) and 6.7% (V2M).","section":"Section 5.2 text and Figure 7 caption"},{"comment":"The '280% accuracy improvement' for the V2M merged class should be stated as a relative improvement in recall (from 0.222 to 0.842), not as an improvement in accuracy.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a competent machine-learning application, but its central contribution is incremental; the main novelty is the comparison between ImageNet fine-tuning and a frozen galaxy-domain backbone. The absence of uncertainty quantification for rare classes and the Zoobot transfer concern are the primary barriers to acceptance. Also, given the lack of cosmological content, suitability for JCAP is a question the editor may wish to consider; an astronomy/ML-focused venue might be a better fit. Reproducibility would be improved by releasing the exact train/validation split and model weights."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read this as a straightforward application paper: train two EfficientNet variants on a carefully filtered GZ2 sample to predict spiral arm number. The new elements are the strict purity cuts, the 224x224 crop, and the direct comparison of ImageNet fine-tuning (V2M) against frozen Zoobot features (B0). For the common classes, m=1,2,3,?, the results look credible: F1 scores above 0.8 on a down-sampled validation set, with confusion matrices and per-class numbers given. That part is honestly presented and probably reproducible enough from the text.\n\nThe soft spots are real but concentrated. The rarest classes, m=4 and m=5+, have only 28 and 31 images total, so the 70:30 split leaves about 9 validation images each. The reported recall of 0.222 for m=4 is roughly 2 correct out of 9; a single image flip changes the metric by 0.11. No error bars, repeated-seed statistics, or cross-validation are given. The abstract's \"except m=4\" clause is exactly where the sample is smallest, and m=5+ precision for V2M is also below 0.8. The merged m=4+5+ class helps, but 16/19 correct (V2M recall 0.842) still has a Wilson interval of roughly (0.60,0.95). So the rare-class headline metrics are not statistically supported.\n\nThe B0 model's strong performance is partly inherited: the frozen Zoobot backbone was trained on GZ decision tree responses, including the very arm-count question T11 used here. The paper does not develop this point. The V2M model is the cleaner test, since it starts from ImageNet, and its results are also good.\n\nTwo more concrete issues. The abstract says m=3 galaxies tend to have lower stellar masses, but Table 4 gives p=0.18 for the K-S test and p=0.72 for the t-test. The conclusion is more careful, but the abstract overstates it. And there is an internal contradiction: Section 5.2 says \"m=3 galaxies are misclassified as m=4 (67%)\", but the confusion matrices show the reverse — 67% of m=4 galaxies are misclassified as m=3. That needs fixing.\n\nNo code is released, which limits exact reproduction, though the method is described in enough detail to rebuild.\n\nOverall: a useful application note, not a breakthrough. The common-class claims hold up; the rare-class claims need error bars and hedging. With revisions (error bars, Zoobot leakage discussion, abstract correction, the misclassification fix), it would be a solid contribution.\n\nRecommendation: send to peer review. The editor should require the rare-class error analysis and the corrections above before acceptance.","headline":"A useful GZ2 arm-count classifier with credible results for common classes, but rare-class metrics are statistically thin, the B0 model inherits the target question from Zoobot, and the abstract overstates the m=3 stellar mass result.","tokens_in":23725,"tokens_out":3474,"would_cite":false,"duration_ms":31997,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two convolutional classifiers can sort spiral galaxies by arm number with most F1 scores above 0.8, except for the rare four-arm class; merging it with the five-plus class lifts the V2M model's recall to about 0.84.","keywords":["spiral arm number","galaxy morphology classification","convolutional neural network","EfficientNet","Galaxy Zoo 2","transfer learning","stellar mass","deep learning"],"falsifier":"Re-train both models on a larger, independent sample with expert-confirmed arm counts, or on many random 70:30 splits of the same data, and check whether m=4 recall stays near 0.222 and whether the merged-class improvement holds; a stable m=4 recall well above 0.5 on independent labels would refute the sample-size explanation, while a collapse across splits would show the headline metrics are artifacts of split composition.","tokens_in":22476,"feed_emoji":"🔭","tokens_out":5018,"duration_ms":42641,"temperature":0.7,"pith_summary":"The paper tries to establish that a straightforward CNN pipeline—no arm-segmentation or sophisticated structure analysis—can classify spiral galaxies by the number of spiral arms accurately enough for large surveys. Using Galaxy Zoo 2 vote-fraction labels filtered to a clean sample of 11,718 images, both an ImageNet-fine-tuned EfficientNetV2M and a Zoobot-feature EfficientNetB0 reach F1 scores above 0.8 for most arm-number classes on a balanced down-sampled set. The four-arm class is the persistent failure, with only 28 training images and recall near random (0.222). Merging four-arm and five-plus-arm galaxies repairs the V2M model's performance for that combined class, at the cost of the scientifically meaningful four-arm distinction. The authors further connect arm number to stellar mass, finding m=1 galaxies skew massive and m=3 galaxies skew light, consistent with earlier Galaxy Zoo work.","feed_headline":"AI classifiers count spiral arms, scoring above 0.8 on most classes","feed_subtitle":"Only the rare four-arm class fails; merging it with five-plus arms lifts recall to 0.84.","key_machinery":"The load-bearing machinery is transfer learning on EfficientNet backbones: a fine-tuned EfficientNetV2M initialized with ImageNet weights, and an EfficientNetB0 whose weights are frozen from the Zoobot galaxy-morphology model, each topped with a global average pooling layer, a dense ReLU layer, and a six-class softmax. The training set is the crucial second piece: Galaxy Zoo 2 images pre-filtered by weighted and debiased vote-fraction cuts (rho_features/disc>0.430, rho_not edge-on>0.715, rho_spiral,yes>0.715, N_spiral,yes>20, rho_m>0.8), down-sampled to roughly 300 per class with rotation, flipping, and zoom augmentation. Gradient heatmaps (GradCAM++) and saliency maps (SmoothGrad) carry the interpretability argument, showing that decisions track spiral structure rather than background artifacts for correct predictions.","core_discovery":"The central claim is that spiral arm number is learnable from raw survey images by generic CNN transfer learning, and that classifier labels reproduce known physical trends well enough to be usable at survey scale. On a down-sampled Galaxy Zoo 2 dataset with about 300 images per class, the V2M model (EfficientNetV2M fine-tuned on ImageNet) and the B0 model (EfficientNetB0 with frozen Zoobot weights) both deliver precision, recall, and F1 above 0.8 for m=1, 2, 3, and 'can't tell', and near or above 0.8 for m=5+, with m=4 the only class at chance-level recall. Combining m=4 with m=5+ raises the V2M model's recall for the merged class to about 0.84, which the authors argue balances scientific discrimination against the scarcity of high-arm samples. GradCAM++ and SmoothGrad indicate the networks base decisions on galaxy structure and spiral-arm extraction, with V2M the stronger of the two. On physical tests, the GZ2 labels show a significant shift of m=1 galaxies toward higher stellar mass and a weaker shift of m=3 toward lower mass; both tendencies are attenuated in model predictions.","pith_inferences":["The tiny m=4 class (28 images total, about 8–10 in validation) makes the reported 0.222 recall extremely sensitive to which images land in the validation split; refitting across multiple random splits would show whether the m=4 numbers are a stable property or split luck.","A testable extension is to run the same frozen-backbone B0 pipeline on deeper imaging (e.g., Stripe-82 coadds or DECaLS) where faint high-arm structure is more visible; if m=4 and m=5+ recall improves markedly, the current limitation is image depth, not class distinctness.","The results suggest a practical rule for other morphology tasks: when a rare class sits at the edge of a decision tree, merging it with its nearest neighbor and reporting both merged and unmerged metrics is more informative than forcing a six-way split."],"forward_implications":["If reliable, these classifiers give a simple, scalable way to assign spiral arm numbers to the millions of galaxies expected from Euclid, Roman, and Rubin/LSST without human inspection.","Merging m=4 and m=5+ is a practical class scheme that preserves the low/high-arm distinction while keeping V2M F1 above 0.8 for every class.","The attenuated stellar-mass trends in model predictions imply that classifier noise dilutes physical correlations, so any survey-scale arm-number catalog should budget for that dilution.","The m=4 failure quantifies the sample-size floor: with about 28 clean examples, even strong transfer learning cannot learn the class, suggesting dedicated collection or synthesis is needed.","The alignment of predictions with the known m=1 high-mass and m=3 low-mass tendencies, despite dilution, suggests the labels carry real physical signal worth pursuing."],"supporting_citations":[{"why":"Supplies the Galaxy Zoo 2 main sample, vote-fraction definitions, and the clean/debiased selection framework from which the dataset of 11,718 images is drawn.","marker":"[8]"},{"why":"The rotation-invariant CNN classification baseline that this paper's models are compared against and reported to surpass by over 100% in F1 for several classes.","marker":"[46]"},{"why":"Provides the earlier GZ2 stellar-mass demographics for spiral arm number that the paper reproduces, and the redshift-debiasing method that motivates their rho_m threshold.","marker":"[28]"},{"why":"Introduces the EfficientNet architecture and scaling method that forms the base of both models used here.","marker":"[65]"},{"why":"Introduces EfficientNetV2, the architecture of the V2M model fine-tuned on ImageNet.","marker":"[68]"},{"why":"Describes Zoobot, the domain-specific galaxy-morphology model whose pre-trained EfficientNetB0 weights are frozen in the B0 model.","marker":"[70]"},{"why":"Provides the MPA-JHU stellar mass estimates used for the spiral-arm-number versus stellar-mass correlation.","marker":"[95]"},{"why":"Grad-CAM++ heatmaps are used to show that network decisions rest on galaxy structure rather than background artifacts.","marker":"[82]"}],"fun_headline_variants":["CNN spiral-arm counter: >0.8 F1, but 4-arm galaxies stump it","AI sorts galaxies by arm count; 4-arm class remains a challenge","Merging spiral-arm classes boosts CNN recall for 4-arm galaxies","EfficientNet counts spiral arms; rare 4-arm galaxies trip it up"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation rests on the Galaxy Zoo 2 debiased vote-fraction labels being correct for every class, especially the 28 four-arm and 31 five-plus-arm galaxies; if those rare labels are noisy or the selection cuts exclude certain galaxy types, the reported F1 scores and the stellar-mass trends collapse.","fun_headline_variants_meta":{"raw":{"variants":["CNN spiral-arm counter: >0.8 F1, but 4-arm galaxies stump it","AI sorts galaxies by arm count; 4-arm class remains a challenge","Merging spiral-arm classes boosts CNN recall for 4-arm galaxies","EfficientNet counts spiral arms; rare 4-arm galaxies trip it up"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001028,"raw_usage":{"total_tokens":4435,"prompt_tokens":1148,"completion_tokens":3287,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":764,"completion_tokens_details":{"reasoning_tokens":3211}},"tokens_in":764,"tokens_out":3287,"duration_ms":22217,"temperature":1.0,"reasoning_tokens":3211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:39:58.407803+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-train both models on a larger, independent sample with expert-confirmed arm counts, or on many random 70:30 splits of the same data, and check whether m=4 recall stays near 0.222 and whether the merged-class improvement holds; a stable m=4 recall well above 0.5 on independent labels would refute the sample-size explanation, while a collapse across splits would show the headline metrics are artifacts of split composition.","supporting_citations":[{"cited_title":"Dieleman, K.W","cited_arxiv_id":null,"evidence_quote":"The rotation-invariant CNN classification baseline that this paper's models are compared against and reported to surpass by over 100% in F1 for several classes."},{"cited_title":"Hart, S.P","cited_arxiv_id":null,"evidence_quote":"Provides the earlier GZ2 stellar-mass demographics for spiral arm number that the paper reproduces, and the redshift-debiasing method that motivates their rho_m threshold."},{"cited_title":"Tan and Q","cited_arxiv_id":null,"evidence_quote":"Introduces the EfficientNet architecture and scaling method that forms the base of both models used here."},{"cited_title":"Tan and Q","cited_arxiv_id":null,"evidence_quote":"Introduces EfficientNetV2, the architecture of the V2M model fine-tuned on ImageNet."},{"cited_title":"Walmsley, C","cited_arxiv_id":null,"evidence_quote":"Describes Zoobot, the domain-specific galaxy-morphology model whose pre-trained EfficientNetB0 weights are frozen in the B0 model."},{"cited_title":"Kauffmann, T.M","cited_arxiv_id":null,"evidence_quote":"Provides the MPA-JHU stellar mass estimates used for the spiral-arm-number versus stellar-mass correlation."},{"cited_title":"Chattopadhay, A","cited_arxiv_id":null,"evidence_quote":"Grad-CAM++ heatmaps are used to show that network decisions rest on galaxy structure rather than background artifacts."}],"review_version":1}