Pith. sign in

REVIEW 3 major objections 1 minor 24 references

Characterizing Stellar Streams with Error-Aware Machine Learning

T0 review · 3 major / 1 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Incorporating observational uncertainties into neural network training lets SCREAM identify more stellar stream members than prior ML methods.

desk verdict SCREAM adapts CATHODE to stellar streams and reports an F1 of 0.745 on GD-1 with extra members recovered, but the abstract leaves the role of the uncertainty term untested. read the letter →

arxiv 2606.09576 v1 pith:UIWDYRLU submitted 2026-06-08 astro-ph.GA astro-ph.IMastro-ph.SR

Alexandros Pratsos , Biprateep Dey , Ting S. Li This is my paper
classification astro-ph.GAastro-ph.IMastro-ph.SR
keywords stellarstreamsmachinelearningobservationaluncertaintiesGaiaGD-1weakly-supervisedMilkyWayDESI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents SCREAM, a weakly-supervised machine learning framework that finds member stars of stellar streams by treating them as localized over-densities in feature space. It is the first such method to fold observational uncertainties directly into the neural network training objective, building on techniques from particle physics without assuming specific gravitational potentials or isochrones. Applied to Gaia and DESI data for the GD-1 stream, SCREAM reaches an F1 score of 0.745 against independent labels, outperforming existing ML approaches in both precision and recall while recovering the expected diffuse cocoon and faint main-sequence stars missed by classical algorithms like STREAMFINDER. A sympathetic reader would care because stellar streams are sensitive tracers of the Milky Way's dark matter and assembly history, so better membership lists enable tighter constraints on those processes.

What carries the argument

SCREAM, the weakly-supervised neural network framework that builds on CATHODE to locate over-densities while adding observational uncertainties to the training loss.

What would settle it

Retraining the identical network architecture on the same GD-1 data but with the uncertainty term removed from the objective, then checking whether the F1 score against the independent labels drops substantially.

Watch

Extended reading notes

Core claim

SCREAM identifies stream members as localized feature-space over-densities while directly incorporating observational uncertainties into the neural network training objective. Validated against independent labels on the GD-1 stream, the method achieves an F1 score of 0.745, substantially outperforming existing ML methods in precision and recall, and recovers the physically expected diffuse cocoon plus faint main-sequence members that classical physics-based algorithms miss.

Load-bearing premise

That the reported performance gain and recovery of additional members are driven by the explicit inclusion of observational uncertainties rather than by other modeling choices or by biases in the independent validation labels.

Editorial extensions

If this is right

  • Uncertainty-aware training improves both precision and recall compared with standard ML methods for stream selection.
  • The approach recovers diffuse structures and faint members without relying on rigid gravitational potentials or strict isochrone filters.
  • Weakly-supervised over-density detection can be applied to other streams using Gaia and DESI-like catalogs.
  • Direct inclusion of measurement errors in the objective reduces the impact of noisy data on membership classification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the uncertainty term drives the gains, the same loss modification could improve other astronomical tasks that classify objects from catalogs with varying error bars.
  • The framework might extend to identifying other low-surface-brightness galactic substructures where classical methods struggle with selection biases.
  • Larger future surveys with higher typical uncertainties could see even larger relative gains from this style of training.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 1 minor

Summary. The manuscript introduces SCREAM, a weakly-supervised machine learning framework building on CATHODE for identifying stellar stream members as localized over-densities in feature space without rigid physical priors. It claims to be the first such method to directly incorporate observational uncertainties into the neural network training objective, reports an F1 score of 0.745 on GD-1 validated against independent labels, substantially outperforms existing ML methods, and recovers the expected diffuse cocoon plus faint main-sequence members missed by classical algorithms such as STREAMFINDER.

Significance. If the performance gains and additional member recovery are demonstrably attributable to the explicit uncertainty term rather than other modeling choices, the work would represent a useful advance in applying uncertainty-aware ML to galactic archaeology, enabling more robust identification of stream structures as probes of dark matter and Milky Way history. The emphasis on validation against independent labels is a positive element.

major comments (3)
  1. [Abstract] Abstract: the central claim that SCREAM is the first framework to directly incorporate observational uncertainties into the NN training objective, and that this drives the reported F1=0.745 and recovery of cocoon/faint members, cannot be evaluated because the abstract supplies no equation, loss-function definition, or description of how uncertainties enter the objective (e.g., as an additional term, modified likelihood, or input feature).
  2. [Abstract] Abstract: no ablation or controlled comparison is described that removes or replaces the uncertainty component while holding other elements (architecture, features, weak-supervision details) fixed; without this, the attribution of performance gains specifically to uncertainty incorporation remains untested and is load-bearing for the novelty claim.
  3. [Abstract] Abstract: the validation protocol against independent labels is not specified (how labels were constructed, selection criteria, potential biases, or cross-validation scheme), preventing assessment of whether the F1 score and member-recovery results are robust or could arise from label artifacts.
minor comments (1)
  1. The expansion of the SCREAM acronym uses non-standard bolding; consider conventional formatting for readability.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their careful review and constructive feedback on the abstract. We address each major comment point-by-point below. We agree that additional detail in the abstract would improve clarity and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that SCREAM is the first framework to directly incorporate observational uncertainties into the NN training objective, and that this drives the reported F1=0.745 and recovery of cocoon/faint members, cannot be evaluated because the abstract supplies no equation, loss-function definition, or description of how uncertainties enter the objective (e.g., as an additional term, modified likelihood, or input feature).

    Authors: We agree that the abstract is too concise to convey the precise mechanism. Section 3.2 of the manuscript defines the modified training objective, in which observational uncertainties enter as an explicit additive term in the loss that penalizes predictions inconsistent with error bars. We will revise the abstract to include a one-sentence description of this term so that the novelty claim can be evaluated from the abstract alone. revision: yes

  2. Referee: [Abstract] Abstract: no ablation or controlled comparison is described that removes or replaces the uncertainty component while holding other elements (architecture, features, weak-supervision details) fixed; without this, the attribution of performance gains specifically to uncertainty incorporation remains untested and is load-bearing for the novelty claim.

    Authors: The referee is correct that no such controlled ablation is described. While the manuscript reports overall gains relative to prior ML methods, it does not isolate the uncertainty term by retraining with the same architecture and supervision but without the uncertainty component. We will add this ablation study to the revised manuscript to strengthen the attribution. revision: yes

  3. Referee: [Abstract] Abstract: the validation protocol against independent labels is not specified (how labels were constructed, selection criteria, potential biases, or cross-validation scheme), preventing assessment of whether the F1 score and member-recovery results are robust or could arise from label artifacts.

    Authors: The validation details (construction of independent spectroscopic labels, selection cuts, and cross-validation scheme) appear in Section 4.3. We acknowledge that the abstract does not summarize these steps. We will add a brief clause to the abstract describing the independent-label validation protocol. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity in derivation or validation chain

full rationale

The paper introduces SCREAM as an extension of the external CATHODE method, with performance (F1=0.745 on GD-1) reported against independent validation labels rather than any self-fitted quantity. No equations, procedures, or self-citations are described that reduce the claimed gains or recovered members to quantities defined by the model's own parameters or prior author work. The central result is therefore benchmarked externally and does not reduce to its inputs by construction.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

Abstract-only information; the central claim rests on the domain assumption that streams manifest as localized over-densities in astrometric-photometric space and that the CATHODE adaptation transfers without additional physical priors. No free parameters or invented entities are enumerated.

assumptions (1)
  • domain assumption Stellar streams appear as localized over-densities in feature space without requiring assumed gravitational potentials or strict isochrone cuts
    The method is described as avoiding rigid physical priors and identifying streams purely via over-densities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Characterizing Stellar Streams with Error-Aware Machine Learning." pith.science (2026). https://pith.science/paper/UIWDYRLU

@misc{pith2026260609576,
  author       = {Pith},
  title        = {Pith review of: Characterizing Stellar Streams with Error-Aware Machine Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UIWDYRLU}},
  note         = {Machine review of arXiv:2606.09576}
}
abstract

Stellar streams are thin, elongated collections of stars formed by gravitational disruption of orbiting star clusters or dwarf galaxies and are highly sensitive probes of the Milky Way's dark matter distribution and formation history. We present $\texttt{SCREAM}$ ($\textbf{S}$tream $\textbf{C}$ha$\textbf{R}$acterization with $\textbf{E}$rror $\textbf{A}$ware $\textbf{M}$achine Learning), a weakly-supervised framework to identify member stars of stellar streams. Building on the $\texttt{CATHODE}$ method originally developed for particle physics, $\texttt{SCREAM}$ identifies streams as localized feature-space over-densities, avoiding rigid physical priors like assumed gravitational potentials or strict isochrone filtering. Crucially, $\texttt{SCREAM}$ is the first machine learning (ML) framework in this domain to directly incorporate observational uncertainties into the neural network training objective. Using astrometric and photometric data from Gaia Data Release 3 and the Dark Energy Spectroscopic Instrument (DESI) Legacy imaging survey, we demonstrate our algorithm's performance on the prominent GD-1 stream. Validated against independent labels, $\texttt{SCREAM}$ achieves an F1 score of 0.745, substantially outperforming existing ML methods in both precision and recall. Furthermore, $\texttt{SCREAM}$ recovers the physically expected diffuse "cocoon" of GD-1 and faint main-sequence members that classical physics-based algorithms (e.g., $\texttt{STREAMFINDER}$) miss. Our results highlight the transformative potential of uncertainty-aware, weakly-supervised ML to uncover complex galactic structures.

Figures

Figures reproduced from arXiv: 2606.09576 by the authors.

Figure 1
Figure 1. Schematic representation of the SCREAM methodology. Left (Data Cross-match and Partition￾ing): Kinematic and photometric features (x) and associated measurement uncertainties (σx), are compiled from Gaia and the DESI Legacy Survey. Data is partitioned into a Signal Region (SR) and a background-dominated Sideband Region (SB) based on along-stream proper motion (µϕ1 ). Middle (Generative Modeling of Back￾ground): A co… view at source ↗
Figure 2
Figure 2. Visualization of SCREAM performance relative to the STREAMFINDER (SF) base￾line. The background distribution is shown in gray. Top Left (ϕ2 vs. ϕ1): SCREAM recovers stars along the full spatial extent of GD-1. Many apparent False Positives (orange crosses) located off the narrow central track are confirmed members via independent radial velocity labels. Right (r vs. g − r): Predictions trace the dense main sequence.… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages

  1. [1]

    , keywords =

    doi: 10.1016/j.newar.2024.101713. John Franklin Crenshaw, J. Bryce Kalmbach, Alexander Gagliano, Ziang Yan, Andrew J. Connolly, Alex I. Malz, Samuel J. Schmidt, and The LSST Dark Energy Science Collaboration. Probabilistic Forward Modeling of Galaxy Catalogs with Normalizing Flows. AJ, 168(2):80, August

    work page doi:10.1016/j.newar.2024.101713 2024

  2. [2]

    Arjun Dey, David J

    doi: 10.3847/1538-3881/ad54bf. Arjun Dey, David J. Schlegel, Dustin Lang, Robert Blum, Kaylan Burleigh, Xiaohui Fan, Joseph R. Findlay, Doug Finkbeiner, David Herrera, Stéphanie Juneau, Martin Landriau, Michael Levi, Ian McGreer, Aaron Meisner, Adam D. Myers, John Moustakas, Peter Nugent, Anna Patej, Edward F. Schlafly, Alistair R. Walker, Francisco Valde...

    work page doi:10.3847/1538-3881/ad54bf

  3. [3]

    NICE: Non-linear Independent Components Estimation

    doi: 10.3847/1538-3881/ab089d. Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear Independent Components Estimation.arXiv e-prints, art. arXiv:1410.8516, October

    work page Pith review arXiv doi:10.3847/1538-3881/ab089d

  4. [4]

    NICE: Non-linear Independent Components Estimation

    doi: 10.48550/arXiv.1410.8516. Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural Spline Flows.arXiv e-prints, art. arXiv:1906.04032, June

    work page Pith review arXiv doi:10.48550/arxiv.1410.8516 1906

  5. [5]

    Neural Spline Flows

    doi: 10.48550/arXiv.1906.04032. Gaia Collaboration, A. Vallenari, A. G. A. Brown, T. Prusti, J. H. J. de Bruijne, F. Arenou, C. Babu- siaux, M. Biermann, O. L. Creevey, C. Ducourant, D. W. Evans, L. Eyer, R. Guerra, A. Hutton, C. Jordi, S. A. Klioner, U. L. Lammers, L. Lindegren, X. Luri, F. Mignard, C. Panem, D. Pourbaix, S. Randich, P. Sartoretti, C. So...

    work page Pith review doi:10.48550/arxiv.1906.04032 1906

  6. [6]

    doi: 10.1051/0004-6361/202243940. C. J. Grillmair and O. Dionatos. Detection of a 63° Cold Stellar Stream in the Sloan Digital Sky Survey. ApJ, 643(1):L17–L20, May

    work page doi:10.1051/0004-6361/202243940

  7. [7]

    J., & Dionatos , O

    doi: 10.1086/505111. Anna Hallin, Joshua Isaacson, Gregor Kasieczka, Claudius Krause, Benjamin Nachman, Tobias Quadfasel, Matthias Schlaffer, David Shih, and Manuel Sommerhalder. Classifying anomalies through outer density estimation. Phys. Rev. D, 106(5):055006, September

    work page doi:10.1086/505111

  8. [8]

    Via Machinae 3.0: A search for stellar streams in Gaia with the CATHODE algorithm

    doi: 10.1103/ PhysRevD.106.055006. Anna Hallin, David Shih, Claudius Krause, and Matthew R. Buckley. Via Machinae 3.0: A search for stellar streams in Gaia with the CATHODE algorithm.arXiv e-prints, art. arXiv:2509.08064, September

    work page Pith review arXiv

Show all 24 references
  1. [9]

    Dan Hendrycks and Kevin Gimpel

    doi: 10.48550/arXiv.2509.08064. Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus).arXiv preprint arXiv:1606.08415,

    doi

  2. [10]

    Li, Sergey E

    7 Emma Jarvis, Ting S. Li, Sergey E. Koposov, Raymond G. Carlberg, Monica Valluri, Nasser Mohammed, J. Aguilar, S. Ahlen, Carlos Allende Prieto, Leandro Beraldo e Silva, D. Bianchi, D. Brooks, Amanda Byström, T. Claybaugh, A. P. Cooper, A. Cuceu, A. de la Macorra, Arjun Dey, B...

    arXiv

  3. [11]

    Kathryn V

    doi: 10.48550/arXiv.2604.20958. Kathryn V . Johnston. A Prescription for Building the Milky Way’s Halo from Disrupted Satellites. ApJ, 495(1):297–308, March

    doi arXiv

  4. [12]

    Diederik P

    doi: 10.1086/305273. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InInternational Conference on Learning Representations (ICLR),

    doi

  5. [13]

    Khyati Malhan and Rodrigo A

    doi: 10.1093/mnras/275.2.429. Khyati Malhan and Rodrigo A. Ibata. STREAMFINDER - I. A new algorithm for detecting stellar streams. MNRAS, 477(3):4063–4076, July

    doi

  6. [14]

    Cecilia Mateu

    doi: 10.1093/mnras/sty912. Cecilia Mateu. galstreams: A library of Milky Way stellar stream footprints and tracks. MNRAS, 520(4):5225–5258, April

    doi

  7. [15]

    doi: 10.1093/mnras/stad321. Eric M. Metodiev, Benjamin Nachman, and Jesse Thaler. Classification without labels: learning from mixed samples in high energy physics.Journal of High Energy Physics, 2017(10):174, October

    2017 doi

  8. [16]

    Benjamin Nachman and David Shih

    doi: 10.1007/JHEP10(2017)174. Benjamin Nachman and David Shih. Anomaly detection with density estimation. Phys. Rev. D, 101 (7):075042, April

    2017 doi

  9. [17]

    Mariel Pettee, Sowmya Thanvantri, Benjamin Nachman, David Shih, Matthew R

    doi: 10.1103/PhysRevD.101.075042. Mariel Pettee, Sowmya Thanvantri, Benjamin Nachman, David Shih, Matthew R. Buckley, and Jack H. Collins. Weakly supervised anomaly detection in the Milky Way. MNRAS, 527(3): 8459–8474, January

    doi

  10. [18]

    Frank Rosenblatt

    doi: 10.1093/mnras/stad3663. Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain.Psychological review, 65(6):386,

    doi

  11. [19]

    Debajyoti Sengupta, Stephen Mulligan, David Shih, John Andrew Raine, and Tobias Golling

    doi: 10.1086/305772. Debajyoti Sengupta, Stephen Mulligan, David Shih, John Andrew Raine, and Tobias Golling. SKY- CURTAINS: model-agnostic search for stellar streams with Gaia data. MNRAS, 536(2):1104–1114, January

    doi

  12. [20]

    David Shih, Matthew R

    doi: 10.1093/mnras/stae2570. David Shih, Matthew R. Buckley, Lina Necib, and John Tamanas. VIA MACHINAE: Searching for stellar streams using unsupervised machine learning. MNRAS, 509(4):5992–6007, February

    doi

  13. [21]

    David Shih, Matthew R

    doi: 10.1093/mnras/stab3372. David Shih, Matthew R. Buckley, and Lina Necib. VIA MACHINAE 2.0: Full-sky, model-agnostic search for stellar streams in Gaia DR2. MNRAS, 529(4):4745–4767, April

    doi

  14. [22]

    Zechang Sun, Joshua S

    doi: 10.1093/ mnras/stae446. Zechang Sun, Joshua S. Speagle, Song Huang, Yuan-Sen Ting, and Zheng Cai. Zephyr : Stitching Heterogeneous Training Data with Normalizing Flows for Photometric Redshift Inference.arXiv e-prints, art. arXiv:2310.20125, October

  15. [23]

    Bingjie Wang, Joel Leja, V

    doi: 10.48550/arXiv.2310.20125. Bingjie Wang, Joel Leja, V . Ashley Villar, and Joshua S. Speagle. SBI ++: Flexible, Ultra-fast Likelihood-free Inference Customized for Astronomical Applications. ApJ, 952(1):L10, July

    doi

  16. [24]

    doi: 10.3847/2041-8213/ace361. 8

    doi

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.