{"id":"c94be04a-4ac3-42e4-afe8-1c4c481d2f4e","arxiv_id":"2501.09112","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-survey CNN estimates photometric redshifts from GALEX, PanSTARRS, and UnWISE cutouts, with early and late image fusion performing comparably.","lead":"The paper trains Mantis Shrimp, a deep learning model that estimates galaxy redshifts from ultraviolet, optical, and infrared images across three sky surveys. It finds that combining the images early or late in the network gives nearly equal accuracy, and it releases the dataset, code, and a web app.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The equivalence of early and late fusion is asserted from one training run per architecture; without seed-variance estimates, Table 4's small differences cannot support 'perform very similarly,' and Figure 9's W1 sign flip undercuts the behavioral half of the claim.","rationale":"The reader's weakest assumption is the same one I would flag: the early/late fusion equivalence is inferred from a single training run per architecture. I agree that this is the load-bearing vulnerability. The paper deserves credit for releasing data, code, and a web-app, for performing ablations, and for explicitly citing seed-variance work and conceding the design limitation in Section 9.2. But the conclusion in Sections 7.1 and 10 converts that conceded null result into a positive claim ('perform and behave very similarly'; 'choice should be left to the user'). The bootstrap uncertainties in Table 4 are often misread as total uncertainty; they only cover test-set sampling. The proposed multi-seed check would settle whether the observed small differences are within training noise. If seed variance is large, the equivalence claim is unsupported; if small, the claim holds. The Shapley W1 sign flip is a secondary, independent reason to query the 'behave very similarly' wording, but the performance-equivalence claim is the central one. I would keep the reader's CONDITIONAL verdict: the recommended fix is either repeated-seed training or a tempered statement that no performance difference was detected in a single-run comparison. Thus verdict_should_be is UNCHANGED relative to the reader.","tokens_in":38455,"tokens_out":6899,"duration_ms":72483,"concrete_test":"Retrain the early and late fusion models with K=5 random seeds (fresh initialization, data order, and augmentation order) using the Table 3 best hyperparameters, and evaluate on the same held-out test set. Report mean and standard deviation across seeds for NMAD, bias, eta, and CDE loss, plus the distribution of early-minus-late differences; if the 95% confidence interval for each difference excludes a pre-registered meaningful threshold (e.g., NMAD = 0.01), the equivalence claim is supported, otherwise it must be softened to 'no significant difference detected.' For the behavior claim, recompute the per-band Shapley/MM-SHAP curves for each seed and check whether the W1 sign flip between architectures at 0<z<0.8 persists; if it is robust, the conclusion that the two architectures 'behave very similarly' should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Sections 7.1, 8.1, 10) is that early and late fusion 'perform and behave very similarly,' so the architecture choice can be left to the user. The evidence is one model per architecture, trained once. Table 4 shows small metric gaps (NMAD 2.438 vs 2.461; CDE loss -7.45 vs -8.81), but the quoted uncertainties are bootstrap resamples of test residuals only (Section 6.1), which do not include randomness from initialization, data order, or PyTorch nondeterminism. The authors cite Banerjee et al. (2024) for substantial seed-to-seed variance and concede in Section 9.2 that 'our experiment design prevents us from definitively saying either architecture is more performant in this task.' Treating that null result as positive evidence of equivalence is an absence-of-evidence inference. The behavioral claim is additionally strained by the paper's own Figure 9: the caption reports that the W1 Shapley value is positive for late fusion and negative for early fusion over 0<z<0.8 and that UV/IR behavior is 'very different,' which contradicts 'behave very similarly' even within the single trained runs. The design-choice recommendation therefore leans on an underpowered comparison and an overstated behavioral match.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Mantis Shrimp, a convolutional-network photometric redshift pipeline that fuses GALEX, PanSTARRS, and UnWISE cutouts into conditional density estimates of redshift. The authors assemble and release a 4.4-million-object, nine-band training sample from eight spectroscopic surveys, train ConvNeXt-based early- and late-fusion models, evaluate point and probabilistic metrics, compare against catalog-based photo-z methods on matched subpopulations, and use Shapley/MM-SHAP values with a source-extractor-based baseline to study band usage. The main claim is that early and late fusion perform and behave very similarly, so the fusion strategy can be left to user preference.","tokens_in":38718,"tokens_out":6168,"duration_ms":57491,"significance":"The public dataset, training code, and web-app make this a useful community resource, and the domain-aware Shapley baseline is a thoughtful contribution to photo-z interpretability. The ablation results in Table 4 support the secondary claim that all nine bands improve on optical-only inputs, and the external comparisons with B22, Tarrío+20, and Zhou+23 provide a valuable sanity check. The supervised training on spectroscopic labels and validation against external catalogs is methodologically sound, and I see no equation-level circularity. The central equivalence claim, however, is currently under-supported: it rests on one training run per architecture, and the paper's own Section 9.2 concedes that the design cannot determine which architecture is more performant. As a pathfinder study these limitations are understandable, but the abstract and conclusions state the equivalence more strongly than the evidence allows.","major_comments":[{"comment":"The central claim that early and late fusion 'perform very similarly' rests on one trained model per architecture, and the quoted uncertainties in Table 4 are bootstrap resamples of test residuals only (Section 6.1 explicitly does not include randomness from initialization, data order, or PyTorch nondeterminism). Since the authors cite Banerjee et al. (2024) for substantial seed-to-seed variance and concede in Section 9.2 that 'our experiment design prevents us from definitively saying either architecture is more performant in this task,' the small metric gaps (e.g., NMAD 2.438 vs 2.461 in Table 4) cannot distinguish equivalence from an underpowered comparison. To support the design-choice recommendation, the paper should either add multi-seed variance estimates (at least for the full-data early/late pair) or explicitly reframe the claim as 'no large difference detected in single runs' throughout the abstract, Section 8.1, and Section 10.","section":"§7.1, Table 4, §6.1"},{"comment":"The statement in Section 7.1 that 'the early and late fusion architectures show very similar performance improvements in each group of ablation experiment' is not uniformly supported by Table 4. For the Optical+UV ablation, the CDE loss is -6.00(4) for late fusion versus -3.90(5) for early fusion, a difference of about 2.1 that is far larger than the reported bootstrap uncertainties and that is in the opposite direction from the full-model comparison, where early fusion has the lower CDE loss. The CDE loss is one of the paper's primary probabilistic metrics, so the 'very similar' wording should be qualified by metric and by ablation group, or this discrepancy should be explained.","section":"§7.1, Table 4"},{"comment":"The behavioral half of the equivalence claim is contradicted by the paper's own Shapley analysis. The Figure 9 caption states that the average W1 Shapley value is positive for late fusion and negative for early fusion over 0<z<0.8 and that UV and IR behavior is 'very different'; Section 7.4 also reports higher overall Shapley sensitivity in the late-fusion model. Section 8.4 nonetheless concludes that 'the shapes of the Shapley curves are overall similar' and uses this to support leaving the choice to the user. This apparent contradiction needs to be resolved, either by quantifying the similarity of the Shapley curves or by restricting the claim to performance metrics and explicitly acknowledging the behavioral differences shown in Figure 9.","section":"§7.4, Figure 9, §8.4"}],"minor_comments":[{"comment":"The grid search values for the learning rate are listed as [5e-4, 3e-4, 1e-4], while the best values are reported as 3e-3 and 1e-3, which are outside that grid; this should be corrected or explained.","section":"Table 3"},{"comment":"Appendix E reports 4.2e6 samples after quality cuts and losses, while Section 3 states N = 4.4e6; the numbers should be reconciled in the text.","section":"Appendix E vs Section 3"},{"comment":"The abstract quotes the catastrophic outlier rate as η>0.15 = 4.51%, whereas Table 4 and Section 6.1 define η as the fraction with scaled residual greater than 0.05; the abstract should state which threshold is being used.","section":"Abstract vs Table 4"},{"comment":"The text contains a typo in 'Shapely value' (twice); the intended term is 'Shapley value'.","section":"Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper header says it is accepted at ApJ; my review assesses the content as submitted. If revisions are possible after acceptance, the major comments should be addressed by either adding seed-variance runs or softening the equivalence claims in the abstract and conclusions. The public data release is a genuine strength and should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing upfront. First, this paper gives the community a real resource: a 4.4-million-galaxy, nine-band cutout dataset (GALEX + PanSTARRS + UnWISE) with trained models, code, and a web app. That alone makes it worth a look. Second, the headline conclusion — early and late fusion \"perform and behave very similarly\" — is weaker than it reads. It rests on a single trained model per architecture, with no seed-to-seed variance, which the authors themselves concede in Section 9.2. There is also an internal tension: Figure 9 shows the W1 Shapley value flips sign between early and late fusion over 0<z<0.8, which does not look like \"behave very similarly.\"\n\nWhat is genuinely new: combining image cutouts from those three surveys into one CNN, comparing early vs late fusion on the same data, and using a domain-informed Shapley baseline (masking the target galaxy with sep rather than zero-filling) to ask whether the models use bands the way an astronomer would. The ablation study is clean and the sub-population comparisons against B22, Tarrío+20, and Zhou+23 are done fairly, with the limitations of each comparison stated. The authors are open about the spectroscopic-selection bias and about the fact that their model does not beat the DESI LS tabular photo-z on the matched sample; that is honest reporting.\n\nSoft spots, in proportion. The main one is the equivalence claim. Table 4's differences (NMAD 2.438 vs 2.461) are small, but the quoted uncertainties only cover test-residual resampling, not training stochasticity. Given that the paper cites Banerjee et al. (2024) for large seed variance, the appropriate conclusion is \"we do not detect a difference in this pathfinder,\" not \"the design choice is left to the user.\" That is a modest downgrade, not a fatal flaw. The behavioral half of the claim is more strained, because Figure 9 does show W1 sign disagreement and the caption itself says the UV/IR behavior is \"very different.\" This should be reconciled or the wording softened.\n\nTwo smaller issues. The abstract says \"well calibrated,\" but Figure 7 shows overconfidence in the PIT; \"reasonably calibrated\" would be closer. And Table 3 lists a learning-rate grid with no value above 5e-4, yet the \"best\" values are 3e-3 and 1e-3 — likely a typo, but it needs fixing.\n\nBottom line: this is a solid, honest empirical contribution with public artifacts. It deserves a serious referee, but I would send it back demanding either a multi-seed comparison (even two or three runs per architecture) or a clearly softened equivalence conclusion, plus the small fixes. Who should read it: anyone working on photometric redshifts with imaging data, and people interested in multi-survey fusion in astronomy. I would bring it to our reading group.","headline":"Solid public dataset and honest comparison, but the early-vs-late fusion equivalence claim is underpowered and partly contradicted by the paper's own Shapley figure.","tokens_in":39286,"tokens_out":3130,"would_cite":true,"duration_ms":31372,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A nine-band convolutional network for photometric redshifts shows that early and late fusion of multi-survey images perform nearly identically.","keywords":["photometric redshift estimation","convolutional neural networks","multi-survey image fusion","early fusion","late fusion","conditional density estimation","Shapley values","forced photo-z"],"falsifier":"Train both architectures from scratch with the same data and hyperparameters across, say, eight random seeds and compare the spread in NMAD and catastrophic-outlier rate against the early-vs-late gap in Table 4; if the seed-to-seed spread is comparable to or larger than the roughly 0.02-point NMAD gap, the claimed equivalence is not established.","tokens_in":38237,"feed_emoji":"🌌","tokens_out":12414,"duration_ms":101039,"temperature":0.7,"pith_summary":"The paper presents Mantis Shrimp, a deep convolutional network that estimates the full probability distribution of a galaxy's redshift from nine image cutouts spanning three surveys: GALEX ultraviolet, PanSTARRS optical, and UnWISE infrared. On the test set of spectroscopically confirmed galaxies it reaches a bias near $10^{-2}$, scatter (NMAD) of $2.44\\times10^{-2}$, and a catastrophic-outlier rate near 18 percent, with well-calibrated density estimates. Its central claim is that the two standard ways to combine multi-survey images — stacking them into a single input (early fusion) or feeding each survey through separate encoders and merging later (late fusion) — produce nearly identical performance and behavior, so the choice can be left to user convenience. The authors support this with ablation studies showing both architectures incorporate information from all surveys, and with Shapley-value analysis showing learned band usage tracks astrophysical expectations such as the 4000 Å break moving through the optical filters.","feed_headline":"For galaxy photo-z, early and late fusion strategies tie","feed_subtitle":"Merging UV, optical, and IR galaxy cutouts, a new CNN shows the simple stacked design loses nothing.","key_machinery":"The load-bearing comparison is between two fusion architectures built on the ConvNext family. Early fusion resamples all cutouts to a common pixel scale and stacks the nine bands into a single input to a ConvNext-Large network. Late fusion keeps each survey at native pixel scale, runs the five-band PanSTARRS cutout through a ConvNext-Large encoder and the two-band GALEX and UnWISE cutouts through ConvNext-Small encoders, and concatenates the latent vectors for a dense classification head. Both output a 400-bin probability distribution over redshift $z\\in(0,1.6)$, trained with cross entropy on one-hot binned spectroscopic redshifts, from which point estimates are taken as the expectation value and calibration is evaluated via the probability integral transform and CDE loss. The other carrying mechanism is the Shapley-value analysis, which ablates the target galaxy's flux using source detection to build a sky-noise baseline image rather than zeroing pixels, then measures each band's contribution to the redshift point estimate; MM-SHAP normalizes these to relative importances.","core_discovery":"On its own terms, the paper's discovery is that a computer-vision model trained on cutouts from GALEX, PanSTARRS, and UnWISE can produce accurate and well-calibrated photometric redshift distributions for the northern sky, and that the two far-apart design options for merging those surveys are effectively interchangeable. With all nine bands, the early-fusion model achieves bias $=1.15\\times10^{-2}$, NMAD $=2.44\\times10^{-2}$, and catastrophic outlier fraction $\\eta = 17.6\\%$ on the full test set, and both fusion architectures meet the community benchmark of NMAD below $10^{-2}$ on the SDSS main galaxy sample. Ablations show the infrared bands drive most of the gain over optical-only, with ultraviolet adding a marginal improvement. The Shapley-value analysis finds that both models use the bands in ways consistent with the physics of redshifted galaxy spectra: the $g$-band importance falls near $z\\approx0.4$ as the 4000 Å break leaves the filter, and infrared bands carry positive importance, reflecting the $r-W1$ color's known redshift sensitivity.","pith_inferences":["If the equivalence of early and late fusion survives repeated-seed tests, it would indicate that weight sharing across nine photometric bands is not a capacity bottleneck for this task — a useful rule of thumb for other multi-survey image problems.","The paper's finding that a tabular model on deeper Legacy Survey photometry wins at every PanSTARRS signal-to-noise level suggests photometric depth, not model class, is the limiting factor; retraining this architecture on deeper images is a direct test.","The match between learned band importances and the physics of the 4000 Å break and $r-W1$ color suggests a general validation protocol: check that a photo-z network's per-band usage varies with redshift the way galaxy SEDs do, which could expose models that rely on survey selection artifacts."],"forward_implications":["Multi-survey image models can be built with the simpler early-fusion design without sacrificing accuracy, since late fusion's separate encoders buy no measurable performance gain.","The model's calibrated conditional densities can serve directly as prior distributions over redshift in stellar population synthesis and other downstream analyses.","Because the model runs on cutouts centered at arbitrary coordinates, it enables forced photo-z estimates for sources not detected in one or more of the input surveys.","Infrared photometry drives most of the multi-survey gain over optical alone, with ultraviolet contributing only marginally — a guide to which bands future surveys should prioritize.","On the SDSS main galaxy sample the model reaches NMAD below $10^{-2}$, the level earlier work treated as a practical accuracy floor."],"supporting_citations":[{"why":"The CNN photo-z work this study extends; supplies the SDSS MGS benchmark and the dust-extinction concatenation approach.","marker":"J. Pasquet et al. 2019"},{"why":"B22, the tabular PanSTARRS+WISE model that is the paper's main multi-survey comparison and whose survey compilation the dataset follows.","marker":"R. Beck et al. 2022"},{"why":"The DESI Legacy Survey photometric redshift catalog whose matched-sample performance the paper must be explained against.","marker":"R. Zhou et al. 2023a"},{"why":"The official SDSS tabular photo-z algorithm used as the baseline that computer vision models beat.","marker":"R. Beck et al. 2016"},{"why":"Supplies the SHAP/Shapley value machinery used for the per-band importance analysis.","marker":"S. Lundberg & S.-I. Lee 2017"},{"why":"Provides the sep source-detection library used to identify and mask the target galaxy when constructing Shapley baseline images.","marker":"K. Barbary 2016"},{"why":"Provides the ZooBotV2 self-supervised weights that initialize the ConvNext encoders.","marker":"M. Walmsley et al. 2024"},{"why":"Defines the ConvNext architecture family on which both fusion models are built.","marker":"Z. Liu et al. 2022"}],"fun_headline_variants":["Nine-band CNN photo-z: fusion style doesn't matter","Mantis Shrimp photo-z: early and late fusion tie","Stacked or late: CNN photo-z choice is a wash","UV-optical-IR cutouts rival photo-z benchmarks","For photo-z, early fusion matches late fusion in CNN"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that early and late fusion perform alike rests on a single training run per architecture, and the paper itself notes that seed-to-seed variability could be large enough to mask real differences.","fun_headline_variants_meta":{"raw":{"variants":["Nine-band CNN photo-z: fusion style doesn't matter","Mantis Shrimp photo-z: early and late fusion tie","Stacked or late: CNN photo-z choice is a wash","UV-optical-IR cutouts rival photo-z benchmarks","For photo-z, early fusion matches late fusion in CNN"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1339,"prompt_tokens":1096,"completion_tokens":243,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":712,"completion_tokens_details":{"reasoning_tokens":161}},"tokens_in":712,"tokens_out":243,"duration_ms":3677,"temperature":1.0,"reasoning_tokens":161,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:10:23.019717+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train both architectures from scratch with the same data and hyperparameters across, say, eight random seeds and compare the spread in NMAD and catastrophic-outlier rate against the early-vs-late gap in Table 4; if the seed-to-seed spread is comparable to or larger than the roughly 0.02-point NMAD gap, the claimed equivalence is not established.","supporting_citations":[],"review_version":1}