{"id":"0c622c9c-71fc-49f8-8b19-b3ea8a9fe335","arxiv_id":"2608.08398","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A post-hoc framework that categorizes galaxy morphology classification uncertainty into four types and predicts them with lightweight estimators trained on frozen foundation model features.","lead":"This paper introduces UEGMC, a framework that estimates four types of uncertainty for AI galaxy morphology classifiers using lightweight post-hoc models. It matters because astronomical surveys increasingly rely on foundation models that currently provide no confidence measures.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (2) is not a valid expansion of Eq. (1), and the morphology-boundary score inherits this defect: without a justified perturbation set R and posterior weights, UEGMC-P measures sensitivity to hand-chosen label changes, not established physical ambiguity.","rationale":"The reader's weakest_assumption identified the hand-constructed R and uniform weighting; I agree this is the right place to look, and my formal objection to Eq. (2) is the root of that assumption. The four-way taxonomy is the paper's distinctive contribution, so the validity of the expansion and the completeness of R are load-bearing: if either fails, the claimed physical interpretation of the four scores is unsupported, even if the MLP estimators are well trained. I am not alleging misconduct: the evaluation may be honest, but the theoretical linchpin is presented as a derivation when it is actually an assumption. The proposed sensitivity test on R would settle whether the morphology-boundary score is a stable physical quantity or a label-perturbation artifact. The paper also has experimental weaknesses (missing error bars, no explicit train/test split, no code), but those are already captured by the reader's conditional verdict. Because the required checks are feasible and the verdict already asks for clarification of Eq. (2) and fair comparison, I would keep the verdict at conditional rather than moving to reject or unverified. Agreement with the reader is partial: their weakest_assumption is the same general area, but my emphasis is on the invalid integral identity underlying it.","tokens_in":13140,"tokens_out":10172,"duration_ms":111554,"concrete_test":"Re-run the Table 4 morphology-boundary evaluation with R expanded to all pairwise class transitions (not just the evolution-driven list), and also with R replaced by an equal number of randomly paired classes, keeping the estimator architecture and training procedure fixed. If the AUC or the per-sample ranking changes materially under either substitution, the score is an artifact of the chosen perturbation set rather than a robust measure of intrinsic physical ambiguity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty is the claim that UEGMC-P quantifies four physically distinct uncertainty types. The load-bearing step is Eq. (2), presented as an expansion of the Bayesian predictive distribution Eq. (1). As written, it is not an identity: x is defined as {x_g, x_M}, yet x_M is also the integration variable, so the integrand p(y|m, θ, x) is conditioned on the very variable being marginalized; the left side would have to be p(y|x_g, D) for the right side to be a hierarchical marginalization, and even then p(m|θ, x) is an unspecified latent distribution, not a physically calibrated quantity. The UEGMC-B approximations in Eqs. (4)-(7) then replace the posterior p(θ|D) by 1/|Θ| over a few model sizes and replace p(m|x, θ) by 1/|R| over a short hand-picked list of evolutionary transitions. Consequently, the morphology boundary uncertainty computed in Eq. (7) and via UEGMC-P's υ_m is a sensitivity-to-label-perturbations score under the chosen R, not an established measure of intrinsic physical ambiguity. If R is incomplete or the uniform weights do not approximate the true integrals, the central claim that UEGMC-P provides physically meaningful uncertainty types does not follow from the presented derivation or experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes UEGMC, a post-hoc uncertainty estimation framework for galaxy morphology classification. UEGMC defines four uncertainty types—model epistemic, data aleatoric, modality aleatoric, and morphology boundary—and offers two instantiations: UEGMC-B, a Bayesian entropy approximation over sets of models, pretraining datasets, modality configurations, and label perturbations; and UEGMC-P, lightweight MLP estimators that predict Brier-score-derived uncertainty values from frozen foundation-model features. The authors evaluate on Galaxy10 DECaLS and GalaxyMNIST with ViT and ConvNeXt backbones, reporting relative AUC gains when the most uncertain samples are refined or their predictions are replaced by better ones.","tokens_in":13444,"tokens_out":7053,"duration_ms":65356,"significance":"The motivation is sound and the application is timely: static foundation-model classifiers in astrophysics currently do not expose uncertainty, and a lightweight, sampling-free, post-hoc method would be practically valuable. The paper also makes a useful proposal in distinguishing multiple uncertainty sources, and it reports experiments across two public datasets and several UQ baselines, including an ablation over feature layers and a computational-efficiency measurement. However, as detailed below, the central validation is compromised by a formally incorrect derivation, a partly circular evaluation protocol, and missing statistics, so the claim of 'competitive performance' is not yet established.","major_comments":[{"comment":"Equation (2) is presented as an expansion of Eq. (1), but it is not a valid marginalization. Since x is defined as {x_g, x_M}, the left-hand side conditions on x_M while the right-hand side integrates over x_M; the left-hand side would at least have to be p(y|x_g,D) for the right-hand side to be a hierarchical marginalization. Moreover, p(m|θ,x) is introduced without a generative story or a likelihood, and the uniform substitutions in Eqs. (4)-(7) (p(θ|D)=1/|Θ|, etc.) are asserted rather than derived. Consequently, the morphology-boundary score computed in Eq. (7) and predicted by UEGMC-P is a sensitivity-to-label-perturbations measure under the hand-selected set R, not an established measure of intrinsic physical ambiguity. This point is load-bearing because the paper's central novelty is the physical interpretation of the four uncertainty types.","section":"Methodology, Eq. (2)"},{"comment":"The evaluation of UEGMC-P is circular for at least the first three uncertainty types. The estimators ρ are trained to regress Brier scores or Brier-score differences on the very model, data, and modality configurations used to define the AUC targets. For example, model epistemic uncertainty is defined as υ_Θ = ρ_T(ξ) − ρ_L(ξ), with ρ_T and ρ_L trained on Brier scores of tiny and large models, and Table 1 then measures how well υ_Θ ranks samples whose predictions improve under a large model. This measures the fit of the regressor to the target, not an independent uncertainty-estimation capability. The comparison is also unfair to baselines that do not use Brier-score training targets. The authors should evaluate UEGMC-P on uncertainty targets held out from training, or compare against baselines trained on the same target construction.","section":"Experiments, Tables 1-4 and UEGMC-P training"},{"comment":"Table 4 omits the UEGMC-B row, although the text states that 'UEGMC-P and its Bayesian baseline, UEGMC-B, consistently perform competitively' across the four uncertainty types. Without these numbers, the claim that UEGMC-B provides a valid baseline for the morphology-boundary type is unsubstantiated. The authors should report UEGMC-B in Table 4 or explain why it cannot be computed.","section":"Table 4"},{"comment":"No variance information is reported for any experiment. All AUC values appear to be single runs, and no seeds, number of repeats, or significance tests are given. Some differences are small (e.g., Table 1, mAP rows differ by less than 0.5 AUC), so without error bars the claimed improvements over the next-best method are not statistically supported. The authors should provide repeated-run statistics or at least bootstrap confidence intervals for the headline comparisons.","section":"Experimental Setup and Tables 1-4"},{"comment":"The manuscript does not specify optimizer, learning rate, number of epochs, MLP width and depth, feature-layer fusion details, loss function, or the train/validation split used for the estimators. The reported inference speed and parameter count also lack hardware and batch-size context. Since UEGMC-P is the main algorithmic contribution, these details are needed for reproducibility.","section":"Methodology, UEGMC-P, and Experimental Setup"}],"minor_comments":[{"comment":"Equation (7) is labeled H_XM but should be labeled H_R (or H_m) to distinguish it from the modality aleatoric entropy in Eq. (6); the current notation makes Eqs. (5)-(7) hard to follow.","section":"Eq. (7)"},{"comment":"Figure 5 reports Pearson correlations between UEGMC-P estimates and 'Oracle' measurements, but the text never defines the Oracle construction for each uncertainty type; without this definition the figure is difficult to interpret.","section":"Figure 5"},{"comment":"The MaxLogit baseline is used in Tables 1-3 but not in Table 4; if max logit cannot be defined for the morphology-boundary task, the authors should say so explicitly.","section":"Experimental Setup, UQ methods under comparison"},{"comment":"The statement that the Brier score 'simultaneously measures reliability and uncertainty' is informal; the Brier score is a proper scoring rule for predictive distributions and is not itself a decomposition into reliability and uncertainty without further assumptions. The text should distinguish these concepts.","section":"Methodology, Eq. (8)"},{"comment":"The evaluation protocol description says AUC is computed by substituting original estimates with a better one, but the exact substitution rule differs across Tables 1-4; a precise formal definition of the AUC protocol and its normalization would improve reproducibility.","section":"Experimental Setup, Evaluation Metrics"},{"comment":"The accuracy curves in Figure 6 have no error bars, and the y-axis ranges differ per panel, making cross-panel gains hard to compare; consider common scales or shaded confidence intervals.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The core idea is worth pursuing, but the derivation and evaluation need substantial rework. In particular, the circularity between UEGMC-P's training targets and the AUC evaluation is the most serious concern and should be addressed before the manuscript is reconsidered. I recommend major revision rather than rejection because the framework's practical motivation and modular design could be salvaged with a cleaner evaluation protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the paper has a genuinely useful taxonomy and a lightweight post-hoc estimator, but the evaluation is stacked in its favor and the headline numbers should not be trusted as-is.\n\nThe thing that's actually new is the four-way decomposition of uncertainty into model epistemic, data aleatoric, modality aleatoric, and morphology boundary. That matches real challenges in GMC—noisy images, multimodal reference standards, and galaxies in transitional stages—and the idea of predicting each from frozen foundation-model features is practical. If it worked, astronomers could flag ambiguous systems for human follow-up and decide when scaling or adding modalities is worthwhile. The qualitative examples in Figure 4 look reasonable, and the inference speed claim is nice.\n\nThe problems are in the evaluation. The estimators in UEGMC-P are trained by regressing Brier scores (or differences of Brier scores), and then the AUC is computed by ranking samples with those same predicted Brier scores and measuring improvement from correcting the top fraction. The regressor is literally trained on the target it is then evaluated to rank well on. That gives it an unfair advantage over generic baselines like MaxLogit or Deep Ensembles, which are not trained against the evaluation metric. I don't think this is fraud—it's a subtle experiment-design flaw—but the large AUC gains in Tables 1-3 are likely optimistic. Also, Table 4 omits UEGMC-B, there are no error bars, and the hand-picked label perturbation set R with uniform weights is not justified.\n\nEq (2) is also not a rigorous expansion. The left side conditions on x, but x_M is both conditioning variable and integration variable, and p(m|θ,x) is never defined as a proper distribution. That matters because the morphology boundary uncertainty is the most novel claim. The stress-test note has this right.\n\nThe central idea does not collapse, though. A fair comparison on an independent calibration set, error bars, and a cleaner derivation would make this a solid contribution. The taxonomy alone is worth discussing. I would send it to review, with a request for major revision.\n\nWho is this for? Astronomers applying foundation models to morphology, and UQ researchers interested in structured uncertainty types. The paper deserves a serious referee. My own verdict: conditional, leaning skeptical on the present numbers, but the concept is worth another round.","headline":"Useful taxonomy, stacked evaluation: the uncertainty gains are likely inflated and Eq (2) doesn't hold, but the idea merits a serious revision.","tokens_in":13979,"tokens_out":5118,"would_cite":false,"duration_ms":50313,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that uncertainty in galaxy morphology classification splits into four source types, and that a lightweight post-hoc estimator (UEGMC-P) can predict all four from frozen foundation-model representations without sampling.","keywords":["uncertainty quantification","galaxy morphology classification","foundation models","post-hoc estimation","epistemic uncertainty","aleatoric uncertainty","Brier score","Bayesian approximation"],"falsifier":"One direct check: replace the physically motivated label transitions in $\\mathcal{R}$ with random alternative labels and rerun the morphology-boundary uncertainty evaluation; if the relative AUC does not drop, the physical-evolution content of $\\mathcal{R}$ is not doing the work. A complementary check: on a set of galaxies with human disagreement (e.g., from a citizen-science decision tree), the boundary uncertainty scores should correlate with disagreement more strongly than plain model confidence does; if not, the fourth uncertainty type is just recalibrated confidence.","tokens_in":12904,"feed_emoji":"🔭","tokens_out":10729,"duration_ms":100276,"temperature":0.7,"pith_summary":"This paper claims that uncertainty in galaxy-morphology classification is not one number but four: uncertainty from limited model capacity, from imperfect or scarce image data, from mismatches introduced by supplementary reference materials such as schematic symbols and text, and from the intrinsic ambiguity of galaxies caught between evolutionary states. The authors build a Bayesian approximation baseline (UEGMC-B) that averages predictions over model sizes, pretraining datasets, multimodal input configurations, and a hand-chosen set of galaxy-evolution label transitions, then train lightweight per-type estimators (UEGMC-P) to reproduce those uncertainty scores directly from the hidden layers of a frozen foundation model, avoiding sampling. On two public galaxy-morphology benchmark datasets and two backbone architectures, UEGMC-P ranks the most uncertain samples better than prior post-hoc uncertainty methods on all four uncertainty types, and it adds fewer than six million parameters. If correct, this would let astronomers see why a classifier is unsure — whether to scale the model, gather cleaner data, fix reference standards, or defer an intrinsically ambiguous galaxy to a human expert — without retraining large models.","feed_headline":"New method splits galaxy-AI uncertainty into four sources","feed_subtitle":"A sampling-free post-hoc module reads frozen backbone features and flags where galaxy labels need human checks.","key_machinery":"The load-bearing object is the four-component decomposition of the predictive integral in Eq. (2), which the paper writes as $p(y|x,D)=\\int_m\\int_{x_M}\\int_\\theta p(y|m,\\theta,x)\\,p(m|\\theta,x)\\,p(x_M|x_g)\\,p(\\theta|D)\\,d\\theta\\,dx_M\\,dm$. Each uncertainty type is approximated by the Shannon entropy of a uniformly weighted average over a perturbation set (Eqs. (4)-(7)), and UEGMC-P replaces those averages with lightweight estimators. The key mechanism is feature fusion $\\xi = \\operatorname{Concat}(z^{(j)})$: concatenating hierarchical embeddings from the frozen backbone so each estimator sees both the image content and the model's predictive inclination, then regressing to Brier scores or Brier-score differences. The four resulting scores $\\upsilon_\\Theta$, $\\upsilon_\\mathcal{A}$, $\\upsilon_{\\mathcal{X}_M}$, and $\\upsilon_\\mathcal{R}$ are the outputs that let a user attribute misclassification risk to its source.","core_discovery":"The central claim is that the Bayesian predictive distribution for galaxy morphology can be decomposed into four physically motivated components — data aleatoric, model epistemic, modality aleatoric, and morphology boundary — and that each component can be estimated post-hoc. UEGMC-B realizes the decomposition as discrete entropy estimates by averaging over four perturbation sets: different model sizes for epistemic uncertainty, different pretraining corpora for data aleatoric uncertainty, different modality configurations (none, text, schematic symbols, or both) for modality aleatoric uncertainty, and label shifts along galaxy-evolution transitions for morphology boundary uncertainty. UEGMC-P then replaces these expensive averages with four small MLP estimators that map a concatenation of hierarchical embeddings from the frozen backbone to Brier-score-based targets. The paper reports that this sampling-free design yields the best or second-best relative AUC in nearly all of the compared settings, with the largest gains on morphology boundary uncertainty, which it interprets as evidence that the decomposition captures physical ambiguity rather than mere model confidence.","pith_inferences":["Beyond the paper: the same four-way decomposition could be transplanted to other noisy astronomical classification tasks (transients, spectra, radio sources), but the categories would need re-grounding; the real test is whether the uncertainty types predict distinct intervention outcomes, not just higher ranking AUC.","Beyond the paper: the uniform weights over the perturbation sets are the paper's pragmatic choice; a probabilistic version that derives transition probabilities from galaxy-evolution models or inter-annotator agreement could either strengthen the boundary-uncertainty claim or reveal how much of it is an artifact of the chosen label list.","Beyond the paper: the evaluation protocol replaces uncertain samples with better predictions, which measures ranking quality; an alternative protocol comparing each uncertainty score to human annotation confidence would test whether the four numbers correspond to genuinely different physical and cognitive sources, as the paper argues."],"forward_implications":["Model epistemic scores identify samples that a larger foundation model would correct, so a user can decide whether scaling up is worthwhile before paying for it.","Data aleatoric scores flag degraded or rare galaxy images that would benefit from more or better pretraining data, providing a metric for whether expensive additional pretraining will help.","Modality aleatoric scores measure when adding text or schematic reference standards hurts rather than helps, allowing a multimodal classifier to be corrected or re-aligned.","Morphology boundary scores can be used to defer intrinsically ambiguous galaxies to human astronomers, preserving effort for cases where machine labels are not trustworthy.","Because UEGMC-P is post-hoc, sampling-free, and adds under six million parameters, it can attach uncertainty estimates to an already-deployed frozen classifier at negligible cost."],"supporting_citations":[{"why":"It supplies the aleatoric/epistemic distinction and the uncertainty-evaluation framing that UEGMC extends into four categories.","marker":"(Kendall and Gal 2017)"},{"why":"It provides the decomposition of predictive uncertainty into epistemic and aleatoric components that Eq. (2) re-decomposes.","marker":"(Depeweg et al. 2018)"},{"why":"It formulates the Bayesian predictive distribution integral that is the starting point of the framework.","marker":"(Gal and Ghahramani 2015)"},{"why":"It supplies Deep Ensembles, the principal sampling-based uncertainty baseline that UEGMC must match or beat.","marker":"(Lakshminarayanan, Pritzel, and Blundell 2017)"},{"why":"It supplies I-EDL, one of the evidential post-hoc baselines compared across all four uncertainty types.","marker":"(Deng et al. 2023)"},{"why":"It supplies R-EDL, another evidential baseline whose reported AUCs establish the comparison level for UEGMC.","marker":"(Chen, Gao, and Xu 2024)"},{"why":"It supplies ETN, the post-hoc evidential-transformation baseline that UEGMC-P is compared against.","marker":"(Chun et al. 2026)"},{"why":"It supplies GEM-FI, a gated evidential-mixture baseline used in the comparison tables.","marker":"(Mohammed, Daneshfar, and Liò 2026)"},{"why":"It supplies Galaxy10 DECaLS, one of the two benchmark datasets used for the experimental evaluation.","marker":"(Henry 2024)"},{"why":"It supplies GalaxyMNIST, the other benchmark dataset used to evaluate the uncertainty estimators.","marker":"(Walmsley 2022)"}],"fun_headline_variants":["Galaxy AI uncertainty split into four sources","Sampling-free module reads frozen features to flag uncertain galaxy labels","Four-way uncertainty breakdown for galaxy morphology AI","New method pinpoints where galaxy classification AI may be wrong","Post-hoc uncertainty estimation for galaxy morphology without sampling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that the small hand-chosen list of plausible galaxy-evolution label transitions, together with uniform weighting over model sizes, pretraining datasets, modality settings, and label perturbations, faithfully represents the true Bayesian uncertainty integrals.","fun_headline_variants_meta":{"raw":{"variants":["Galaxy AI uncertainty split into four sources","Sampling-free module reads frozen features to flag uncertain galaxy labels","Four-way uncertainty breakdown for galaxy morphology AI","New method pinpoints where galaxy classification AI may be wrong","Post-hoc uncertainty estimation for galaxy morphology without sampling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1311,"prompt_tokens":894,"completion_tokens":417,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":343}},"tokens_in":510,"tokens_out":417,"duration_ms":4827,"temperature":1.0,"reasoning_tokens":343,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:36:18.404752+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One direct check: replace the physically motivated label transitions in $\\mathcal{R}$ with random alternative labels and rerun the morphology-boundary uncertainty evaluation; if the relative AUC does not drop, the physical-evolution content of $\\mathcal{R}$ is not doing the work. A complementary check: on a set of galaxies with human disagreement (e.g., from a citizen-science decision tree), the boundary uncertainty scores should correlate with disagreement more strongly than plain model confidence does; if not, the fourth uncertainty type is just recalibrated confidence.","supporting_citations":[{"cited_title":"Github Repository , howpublished =","cited_arxiv_id":null,"evidence_quote":"It supplies Galaxy10 DECaLS, one of the two benchmark datasets used for the experimental evaluation."},{"cited_title":"Github Repository , howpublished =","cited_arxiv_id":null,"evidence_quote":"It supplies GalaxyMNIST, the other benchmark dataset used to evaluate the uncertainty estimators."}],"review_version":1}