REVIEW 3 major objections 5 minor 1 cited by
AMPEL workflows for LSST: Modular and reproducible real-time photometric classification
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper presents three modular AMPEL channels that classify simulated LSST alerts in real time, with SNGuess tagging about 99% of young supernovae and FinalBet correctly classifying more than 80% of extragalactic transients.
desk verdict Solid, reproducible engineering; the abstract's headline numbers overstate what the paper actually measures, so the claims need rescaling before the paper is accepted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the AMPEL tiered unit system, through which alerts flow from T0 filtering, T1 state assembly, T2 per-transient analysis, and T3 sample-level reaction, with every state, configuration, and output tied to a provenance record. The classification itself is carried by T2RiseDeclineStat, which extracts up to 32 early-detection features for SNGuess's XGBoost trees, and by T2RunParsnip, which fits ParSNIP, a hybrid generative model combining explicit physical variables such as redshift and peak time with latent variables learned by a variational autoencoder, at discrete sampled redshifts. FinalBet's T2ElasticcReport then multiplies the ParSNIP probabilities by priors built from training-sample redshift distributions, relative rates from a bright-transient survey, and host-galaxy color fits. The jobfile encodes the whole chain, so any user with an AMPEL environment can rerun it.
What would settle it
Run the public jobfiles on the blinded alert stream used for the challenge, or later on real LSST alerts, and compare per-class true-classification fractions with the paper's roughly 99% young-supernova retention and more than 80% extragalactic success; the paper's own blinded appendix already indicates the fractions fall to roughly 50 to 70% for FinalBet, which would falsify the headline figures as a statement about deployed performance. Alternatively, feed identical simulated lightcurves through the two alert-generation tools and check whether the resulting alert streams differ systematically.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a code-to-data framework can host complete real-time classification programs for the LSST alert stream, and that the three supplied channels perform well on the ELAsTiCC v1 simulation. SNGuess, based on features designed for ZTF alerts and trained with XGBoost, correctly tags roughly 99% of young supernovae among alerts with few detections. FollowMe demonstrates that the ambiguous middle of a ParSNIP-based SNIa/SNIbc classification can be sampled to build a labeled follow-up set whose host-galaxy colors match the parent population. FinalBet adds explicit priors, including redshift distribution, relative rates, and host color, and achieves AUC scores above 0.9 for most classes, with trustworthy classifications correct well above 80% for most transient types.
Load-bearing premise
The headline performance numbers are measured on a held-out test sample explicitly built from the same relative class, redshift, and rate distributions as the training data, and through an alert-generation path that the paper says was never checked against the official blind-test tools; if either of those differences matters, the reported percentages will not transfer to the real LSST stream.
Editorial extensions
If this is right
- If the reported performance holds, groups can take the public jobfiles and adapt them to their own VRO science programs without rebuilding an alert-processing infrastructure.
- The SNGuess channel, combined with external multi-messenger constraints, can provide a pure stream of young-transient and counterpart candidates in real time.
- FinalBet demonstrates that adding explicit population priors improves classification accuracy over the raw model output, at the cost of a biased selection that must be tracked.
- Because AMPEL records provenance for each classification, follow-up decisions made in real time can later be linked to final, reprocessed classifications for sample correction.
- The same workflows can be rerun at scale at a computer center or on the live AMPEL broker channel, making simulation-era development translate directly to the LSST stream.
Reading between the lines
- The paper's own blinded test results show much lower true-classification fractions, roughly 50 to 70% for FinalBet; if that gap persists on real LSST data, the headline 99% and >80% figures should be read as matched-distribution upper bounds rather than deployed expectations.
- A direct side-by-side test of the two alert-generation paths used here would isolate whether the performance drop comes from the priors or from unrecognized format differences between the internal loader and the official LSST tools.
- The placeholder machinery described, including classifier combination, host information, and non-Gaussian redshift sampling, suggests the architecture itself may be the lasting contribution, allowing newer classifiers to be plugged in and compared fairly within the same jobfile.
- One testable extension is to rerun SNGuess with the updated unit that handles negative-flux detections, which should reduce the AGN and microlensing contamination visible in the blinded confusion matrix.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes three AMPEL-based processing channels designed for real-time classification of LSST alert streams, using the ELAsTiCC v1 simulated dataset as a testbed. SNGuess is a gradient-boosted decision tree filter for identifying young extragalactic transients; FollowMe uses ParSNIP to select an unbiased, observationally accessible subset of Ia/Ibc supernovae for spectroscopic follow-up; FinalBet adds ad hoc priors based on redshift, host-galaxy color, and expected rates to produce final classifications. The authors report SNGuess retaining 99% of young supernovae, FollowMe producing a representative uncertain-class subset, and FinalBet achieving AUC > 0.9 for most classes with correct-class rates above 80% on the classified subset. They also present results from the blind DESC ELAsTiCC evaluation, which show somewhat lower performance. The paper emphasizes reproducibility through public jobfiles, provenance tracking, and a local execution workflow.
Significance. If the reported performance is correctly stated, the paper provides a valuable, fully public, modular framework for real-time transient classification, with concrete implementations that other groups can adapt. Strengths include the explicit provenance system, the release of jobfiles and code, the use of a realistic simulated alert stream, and the careful description of the three distinct science-use cases. The internal held-out evaluation is clearly presented with confusion matrices and ROC curves. However, the headline quantitative claims in the abstract are not supported by the metrics actually defined in Section 5.3 and Appendix B, and the main-text evaluation does not adequately separate the distribution-matched internal test from the blind stream results. The paper's central value is the workflow infrastructure and the reproducible demonstration, not the specific performance numbers; those numbers need to be qualified and corrected before the paper can be accepted.
major comments (3)
- [Abstract; Section 5.3, Figs. 10-11] The abstract claims FinalBet 'achieves successful classifications for ≳ 80% of all extragalactic transients', but the evaluation in Section 5.3 defines a different quantity: with a probability threshold of 0.68, only 67% of all transients receive a classification at all, and the correct-class rate for that subset is 'typically above 80%'. The implied overall fraction of correctly classified transients is therefore at most about 0.67 × 0.80 ≈ 54%, not >80%, and the actual fraction depends on the per-class distribution. The DESC blind test (Fig. B.3) gives true classification fractions of roughly 50-70%, consistent with this lower bound. The abstract and Section 5.3 must be rephrased to distinguish 'precision on the 67% classified subset' from 'fraction of all extragalactic transients that are correctly classified', and the headline percentages should be revised accordingly.
- [Section 5; Section 5.4] The internal evaluation is explicitly distribution-matched: the test data are described as 'based on the same relative distributions' as the training sample, and Section 5.4 states that internal alert generation used ElasticcTrainingSetLoader while the blind test used LSST tools, adding 'No tests were made to verify whether the output is similar'. Consequently, the claims 'SNGuess correctly tags 99% of all young supernovae' and 'AUC > 0.9 for most FinalBet classes' are established only for the matched internal test set, not for the actual ELAsTiCC alert stream or for future LSST data. The authors should either clearly qualify these claims throughout the abstract and conclusions as applying to the distribution-matched internal test, or they should perform and report a comparison of the two alert-generation paths.
- [Section 5.4; Appendix B] Section 5.4 acknowledges that the FinalBet priors 'certainly do not match the underlying class distribution that was simulated for ELAsTiCC', and Appendix B shows that the priors improve common classes (e.g., SNIa) at the expense of rare classes (e.g., SNIax, CART, with true classification fractions near 30%). The paper should quantify the effect of the priors more directly, for example by comparing the FollowMe and FinalBet confusion matrices in the blind test (Figs. B.2 and B.3), and should soften the conclusion that FinalBet 'includes priors to achieve successful classifications' without stating that the priors are semi-arbitrary and can reduce performance for rare classes. The current presentation overstates the benefit of the priors.
minor comments (5)
- [Section 2.1] There are several typos in this section: 'unqiue' should be 'unique', 'algorthmic' should be 'algorithmic', and 'specfiying' should be 'specifying'.
- [Section 4.3; Appendix A] The text contains typos such as 'lableed' (should be 'labeled') and 'chocen' (should be 'chosen'); in Appendix A, 'smallestsnsep/sqradius' should read 'smallest snsep/sqradius'.
- [Section 5.2] The KS test p-value of 0.22 is quoted without stating the sample sizes or the exact test used; adding this information would make the claim more verifiable.
- [Listing 1] The code block contains a line break artifact ('Ampel-HU-astro.git,→') that should be cleaned up for the published version.
- [Appendix B, Fig. B.3 caption] The blind-test caption states that true classification fractions 'generally vary between 50 and 70%'; since this is the most realistic LSST-like evaluation available, this figure should also be cited in the main text when discussing FinalBet performance.
Circularity Check
No significant circularity: the paper's performance claims rest on independent holdout and blind-test evaluations, and the self-citations are not load-bearing.
full rationale
The paper's central quantitative claims are based on standard held-out evaluation from the ELAsTiCC training batches (Section 5) and on the externally administered DESC blind test (Appendix B). The FinalBet priors are fit to the training sample (redshift prior derived from ELAsTiCC training batches, host u-g color prior from a normal fit to the training sample), and the internal test set is 'distinct from the training sample, but based on the same relative distributions' (Section 5). This is a conventional train/test split, not a construction in which the evaluation metric is the fitted quantity: the ParSNIP lightcurve fits and XGBoost features still determine the classifications, and the priors only reweight them. The paper explicitly acknowledges the priors are suboptimal for the blind test ('these were semi-arbitrarily defined, and certainly do not match the underlying class distribution'), and the blind test indeed shows lower performance (about 50-70% true classification fractions), which would not happen if the reported performance were forced by construction. The abstract's phrasing that FinalBet 'achieves successful classifications for >~80% of all extragalactic transients' overstates what is measured (Figure 10 shows a probability threshold of 0.68 is met for only 67% of transients, with correct-class rates typically above 80% within that subset), and the SNGuess '99% of all young supernovae' likewise omits the <=5-detection restriction used in the evaluation. These are reporting/completeness concerns, not circularity: the reported percentage is not defined in terms of the input in a way that makes the outcome tautological. Self-citations to Nordin et al. (2019) for the AMPEL framework and Miranda et al. (2022) for the SNGuess feature set are normal methodological references; the new content here is the workflow implementations, the ELAsTiCC evaluation, and the public jobfiles, which are independently assessable. No load-bearing argument reduces to a self-citation, and no 'uniqueness' or ansatz is imported from the authors' prior work to force the conclusion. The paper is self-contained against an external benchmark (the DESC blind test), so the circularity score is 0.
Assumptions & free parameters
free parameters (7)
- FinalBet redshift prior =
per-class redshift histograms with 0.1-width bins, normalized and summed
- FinalBet rate prior =
ZTF BTS class rates with a 0.01 floor for all other classes
- FinalBet host u-g color prior =
per-class Gaussian mean and std of (u-g), Table 2
- Negative-flux classification thresholds =
z split at 0.001; 20% negative-fraction split
- Redshift sampling scheme =
3 samples at chosen quantile means with weights 0.2/0.6/0.2; no-host grid {0.01,0.2,0.5,0.8} with weights…
- SNGuess evaluation detection cut =
alerts with <= 5 detections
- Trained ML model weights =
XGBoost and ParSNIP+lightgbm models trained on batches 1-20 with 10x augmentation
assumptions (6)
- domain assumption ELAsTiCC v1 simulation faithfully represents future LSST alerts (noise profile, cadence, alert content)
- domain assumption Held-out test batches are representative because they share the training relative distributions
- domain assumption Host galaxy associations and photometric redshift quantiles in alerts are usable for redshift sampling
- domain assumption Transient class correlates with host (u-g) color as in the training sample
- standard math Standard ML library implementations (XGBoost, lightgbm, torch/ParSNIP) perform as documented
- ad hoc to paper Rule-based labeling of negative-flux alerts is valid for the real alert stream
Cite this review
Pith. "Pith review of AMPEL workflows for LSST: Modular and reproducible real-time photometric classification." pith.science (2026). https://pith.science/paper/RXJD6FLL
@misc{pith2026250116511,
author = {Pith},
title = {Pith review of: AMPEL workflows for LSST: Modular and reproducible real-time photometric classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/RXJD6FLL}},
note = {Machine review of arXiv:2501.16511}
}
read the original abstract
Modern time-domain astronomical surveys produce high throughput data streams which require tools for processing and analysis. This will be critical for programs making full use of the alert stream from the Vera Rubin Observatory (VRO), where spectroscopic labels will only be available for a small subset of all transients. In this context, the AMPEL toolset can work as a code-to-data platform for the development of efficient, reproducible and flexible workflows for real-time astronomical application. We here introduce three different AMPEL channels constructed to highlight different uses of alert streams: to rapidly find infant transients (SNGuess), to provide unbiased transient samples for follow-up (FollowMe) and to deliver final transient classifications (FinalBet). These pipelines already contain placeholders for mechanisms which will be essential for the optimal usage of VRO alerts: combining different classifiers, including host galaxy information, population priors and sampling non-gaussian photometric redshift distributions. Based on the ELAsTiCC simulation, all three channels are already working at a high level: SNGuess correctly tags 99% of all young supernovae, FollowMe illustrates how an unbiased subset of alerts can be selected for spectroscopic follow-up in the context of cosmological probes and FinalBet includes priors to achieve successful classifications for >~80% of all extragalactic transients. The fully functional workflows presented here are all public and can be used as starting points for any group wishing to optimize pipelines for their specific VRO science programs. AMPEL is designed to allow this to be done in accordance with FAIR principles: both software and results can be easily shared and results reproduced. The code-to-data environment ensures that models developed this way can be directly applied to the real-time LSST stream parsed by AMPEL.
Figures
Figures from the paper (7 more)
Forward citations
Cited by 1 Pith paper
-
Text embedding models can be great data engineers
Text embeddings of raw, text-serialized time series, compressed by a supervised variational information bottleneck, can match or beat hand-engineered pipelines on some classification benchmarks.
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archiveprefix author booktitle chapter edition editor howpublished institution eprint journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 ...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in " " * FUNCTION format....
-
[3]
1966, in Stellar Evolution, ed.\ R
Baker, N. 1966, in Stellar Evolution, ed.\ R. F. Stein,& A. G. W. Cameron (Plenum, New York) 333
1966
-
[4]
1988, A&A, 200, 58
Balluch, M. 1988, A&A, 200, 58
1988
-
[5]
Cox, J. P. 1980, Theory of Stellar Pulsation (Princeton University Press, Princeton) 165
1980
-
[6]
N.,& Stewart, J
Cox, A. N.,& Stewart, J. N. 1969, Academia Nauk, Scientific Information 15, 1
1969
-
[7]
1980, Prog
Mizuno H. 1980, Prog. Theor. Phys., 64, 544
1980
-
[8]
Tscharnuter W. M. 1987, A&A, 188, 55
1987
Show all 36 references
-
[9]
1992, in ASP Conf
Terlevich, R. 1992, in ASP Conf. Ser. 31, Relationships between Active Galactic Nuclei and Starburst Galaxies, ed. A. V. Filippenko, 13
1992
-
[10]
Yorke, H. W. 1980a, A&A, 86, 286
-
[11]
F., Tytler, D
Zheng, W., Davidsen, A. F., Tytler, D. & Kriss, G. A. 1997, preprint
1997
-
[12]
2021, , 162, 275
Boone , K. 2021, , 162, 275
2021
-
[13]
B., Nichol , R
Campbell , H., D'Andrea , C. B., Nichol , R. C., et al. 2013, , 763, 88
2013
- [14]
-
[15]
A., Sharma , Y., et al
Fremling , C., Miller , A. A., Sharma , Y., et al. 2020, , 895, 32
2020
-
[16]
L., Connolly , A
Graham , M. L., Connolly , A. J., Ivezi \'c , Z ., et al. 2018, , 155, 1
2018
-
[17]
I., Ponder , K
Hlo z ek , R., Malz , A. I., Ponder , K. A., et al. 2023, , 267, 25
2023
-
[18]
A., et al
Hosseinzadeh , G., Dauphin , F., Villar , V. A., et al. 2020, , 905, 93
2020
-
[19]
Ishida , E. E. O., Beck , R., Gonz \'a lez-Gait \'a n , S., et al. 2019, , 483, 2
2019
-
[20]
M., Tyson , J
Ivezi \'c , Z ., Kahn , S. M., Tyson , J. A., et al. 2019, , 873, 111
2019
-
[21]
O., Scolnic , D
Jones , D. O., Scolnic , D. M., Riess , A. G., et al. 2017, , 843, 6
2017
-
[22]
P., Cinabro , D., et al
Kessler , R., Bernstein , J. P., Cinabro , D., et al. 2009, , 121, 1028
2009
-
[23]
D., Peiris , H
Lochner , M., McEwen , J. D., Peiris , H. V., Lahav , O., & Winter , M. K. 2016, , 225, 31
2016
-
[24]
A., Allison , J., et al
LSST Science Collaboration , Abell , P. A., Allison , J., et al. 2009, arXiv e-prints, arXiv:0912.0201
2009 arXiv
-
[25]
C., Nordin , J., et al
Miranda , N., Freytag , J. C., Nordin , J., et al. 2022, , 665, A99
2022
-
[26]
& de Boissi \`e re , T
M \"o ller , A. & de Boissi \`e re , T. 2020, , 491, 4277
2020
-
[27]
2022, , 514, 5159
M \"o ller , A., Smith , M., Sako , M., et al. 2022, , 514, 5159
2022
-
[28]
S., Biswas , R., & Hlo z ek , R
Muthukrishna , D., Narayan , G., Mandel , K. S., Biswas , R., & Hlo z ek , R. 2019, , 131, 118002
2019
-
[29]
2019, , 631, A147
Nordin , J., Brinnel , V., van Santen , J., et al. 2019, , 631, A147
2019
-
[30]
2019, , 627, A21
Pasquet , J., Pasquet , J., Chaumont , M., & Fouchez , D. 2019, , 627, A21
2019
-
[31]
2021, , 162, 67
Qu , H., Sako , M., M \"o ller , A., & Doux , C. 2021, , 162, 67
2021
-
[32]
A., Trotta , R., & van Dyk , D
Revsbech , E. A., Trotta , R., & van Dyk , D. A. 2018, , 473, 3969
2018
-
[33]
2021, , 161, 141
S \'a nchez-S \'a ez , P., Reyes , I., Valenzuela , C., et al. 2021, , 161, 141
2021
-
[34]
2011, Sky Event Reporting Metadata Version 2.0 , IVOA Recommendation 11 July 2011
Seaman , R., Williams , R., Allan , A., et al. 2011, Sky Event Reporting Metadata Version 2.0 , IVOA Recommendation 11 July 2011
2011
-
[35]
J., Crellin-Quick, A., & Bloom, J
van der Walt, S. J., Crellin-Quick, A., & Bloom, J. S. 2019, Journal of Open Source Software, 4
2019
-
[36]
A., Hosseinzadeh , G., Berger , E., et al
Villar , V. A., Hosseinzadeh , G., Berger , E., et al. 2020, , 905, 94
2020
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.