{"id":"cb705090-1483-4a8a-91d9-c4c0b91913bf","arxiv_id":"2501.16511","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Three public, reproducible AMPEL channels classify simulated LSST transients, retaining 99% of young supernovae and achieving >80% correct final classifications on confident predictions in the internally matched ELAsTiCC test set.","lead":"This paper presents three ready-to-run software pipelines that will sort the millions of nightly alerts from the future Vera Rubin Observatory, tagging young supernovae and classifying other cosmic explosions in real time. The authors report strong performance on a large simulated alert stream and release all code and job files so any team can rerun or adapt the workflows.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's '>80% of all extragalactic transients' measures precision on a 67% confident subset, not overall recall; the DESC blind test shows 50-70%, so the central performance claim overstates the evidence.","rationale":"The paper is a solid methods contribution: the AMPEL provenance architecture, the public job files, and the honest Appendix B blind-test disclosure are genuine evidence that the infrastructure works and that the authors are not hiding the main limitation. The load-bearing problem is at the level of claim calibration rather than code correctness. Section 5.3's evaluation does not measure what the abstract says it measures: only 67% of transients pass the 0.68 confidence gate, and correctness on that subset does not translate to >80% success over all extragalactic transients. The DESC blind test in Appendix B is the closest available proxy for the actual ELAsTiCC stream, and its 50-70% true-class fractions contradict the headline. Section 5.4's admission that the internal loader and LSST-tools stream were never cross-checked means the transfer assumption is explicitly unverified. This does not invalidate the workflows or the engineering contribution; it requires the quantitative claims to be relabeled and the blind-test numbers reported in the abstract. The reader's CONDITIONAL verdict already captures this, so no further verdict change is needed.","tokens_in":20567,"tokens_out":6920,"duration_ms":62486,"concrete_test":"Using the already-published DESC blind-test outputs (or rerunning the public jobfile elasticc_alerttar.yml on the ELAsTiCC blind alerts), compute the abstract's exact metrics: (1) SNGuess recall for young extragalactic transients with <=5 detections; (2) FinalBet fraction of all extragalactic transients whose highest-probability class is correct at a probability threshold of >=0.68. If the second value is below 80% or the first below 99%, the abstract should be revised to report precision on the confident subset of the internally matched test, with the blind-test recall stated alongside.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the abstract's quantitative claims describe a metric the paper never actually measures, and the closest measured quantity is materially lower. The abstract states FinalBet 'achieves successful classifications for >~80% of all extragalactic transients'. Section 5.3 defines the evaluation differently: with a probability threshold of 0.68, only 67% of all transients receive a classification at all (Fig. 10), and for those the correct-class rate is 'typically above 80%'. Thus at most roughly 0.67 x 0.80 = 54% of all extragalactic transients are correctly classified, not >80%. The DESC blind test (Fig. B.3) independently gives true classification fractions of about 50-70%, consistent with this lower bound and with the paper's own admission in Section 5.4 that the priors do not match the ELAsTiCC class distribution. The SNGuess 99% claim is likewise measured on a holdout 'based on the same relative distributions' as the training sample (Section 5), not on the blind stream, and Section 5.4 states that the internal ElasticcTrainingSetLoader output was never checked against the LSST-tools alert stream used for the blind test. The load-bearing transfer assumption is therefore unverified, and the abstract's headline percentages are not established for the actual ELAsTiCC alert stream.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes three AMPEL-based processing channels designed for real-time classification of LSST alert streams, using the ELAsTiCC v1 simulated dataset as a testbed. SNGuess is a gradient-boosted decision tree filter for identifying young extragalactic transients; FollowMe uses ParSNIP to select an unbiased, observationally accessible subset of Ia/Ibc supernovae for spectroscopic follow-up; FinalBet adds ad hoc priors based on redshift, host-galaxy color, and expected rates to produce final classifications. The authors report SNGuess retaining 99% of young supernovae, FollowMe producing a representative uncertain-class subset, and FinalBet achieving AUC > 0.9 for most classes with correct-class rates above 80% on the classified subset. They also present results from the blind DESC ELAsTiCC evaluation, which show somewhat lower performance. The paper emphasizes reproducibility through public jobfiles, provenance tracking, and a local execution workflow.","tokens_in":20762,"tokens_out":3121,"duration_ms":36858,"significance":"If the reported performance is correctly stated, the paper provides a valuable, fully public, modular framework for real-time transient classification, with concrete implementations that other groups can adapt. Strengths include the explicit provenance system, the release of jobfiles and code, the use of a realistic simulated alert stream, and the careful description of the three distinct science-use cases. The internal held-out evaluation is clearly presented with confusion matrices and ROC curves. However, the headline quantitative claims in the abstract are not supported by the metrics actually defined in Section 5.3 and Appendix B, and the main-text evaluation does not adequately separate the distribution-matched internal test from the blind stream results. The paper's central value is the workflow infrastructure and the reproducible demonstration, not the specific performance numbers; those numbers need to be qualified and corrected before the paper can be accepted.","major_comments":[{"comment":"The abstract claims FinalBet 'achieves successful classifications for ≳ 80% of all extragalactic transients', but the evaluation in Section 5.3 defines a different quantity: with a probability threshold of 0.68, only 67% of all transients receive a classification at all, and the correct-class rate for that subset is 'typically above 80%'. The implied overall fraction of correctly classified transients is therefore at most about 0.67 × 0.80 ≈ 54%, not >80%, and the actual fraction depends on the per-class distribution. The DESC blind test (Fig. B.3) gives true classification fractions of roughly 50-70%, consistent with this lower bound. The abstract and Section 5.3 must be rephrased to distinguish 'precision on the 67% classified subset' from 'fraction of all extragalactic transients that are correctly classified', and the headline percentages should be revised accordingly.","section":"Abstract; Section 5.3, Figs. 10-11"},{"comment":"The internal evaluation is explicitly distribution-matched: the test data are described as 'based on the same relative distributions' as the training sample, and Section 5.4 states that internal alert generation used ElasticcTrainingSetLoader while the blind test used LSST tools, adding 'No tests were made to verify whether the output is similar'. Consequently, the claims 'SNGuess correctly tags 99% of all young supernovae' and 'AUC > 0.9 for most FinalBet classes' are established only for the matched internal test set, not for the actual ELAsTiCC alert stream or for future LSST data. The authors should either clearly qualify these claims throughout the abstract and conclusions as applying to the distribution-matched internal test, or they should perform and report a comparison of the two alert-generation paths.","section":"Section 5; Section 5.4"},{"comment":"Section 5.4 acknowledges that the FinalBet priors 'certainly do not match the underlying class distribution that was simulated for ELAsTiCC', and Appendix B shows that the priors improve common classes (e.g., SNIa) at the expense of rare classes (e.g., SNIax, CART, with true classification fractions near 30%). The paper should quantify the effect of the priors more directly, for example by comparing the FollowMe and FinalBet confusion matrices in the blind test (Figs. B.2 and B.3), and should soften the conclusion that FinalBet 'includes priors to achieve successful classifications' without stating that the priors are semi-arbitrary and can reduce performance for rare classes. The current presentation overstates the benefit of the priors.","section":"Section 5.4; Appendix B"}],"minor_comments":[{"comment":"There are several typos in this section: 'unqiue' should be 'unique', 'algorthmic' should be 'algorithmic', and 'specfiying' should be 'specifying'.","section":"Section 2.1"},{"comment":"The text contains typos such as 'lableed' (should be 'labeled') and 'chocen' (should be 'chosen'); in Appendix A, 'smallestsnsep/sqradius' should read 'smallest snsep/sqradius'.","section":"Section 4.3; Appendix A"},{"comment":"The KS test p-value of 0.22 is quoted without stating the sample sizes or the exact test used; adding this information would make the claim more verifiable.","section":"Section 5.2"},{"comment":"The code block contains a line break artifact ('Ampel-HU-astro.git,→') that should be cleaned up for the published version.","section":"Listing 1"},{"comment":"The blind-test caption states that true classification fractions 'generally vary between 50 and 70%'; since this is the most realistic LSST-like evaluation available, this figure should also be cited in the main text when discussing FinalBet performance.","section":"Appendix B, Fig. B.3 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid software/methods contribution with fully public code and reproducible workflows, and the internal evaluation is clearly described. The main barrier to acceptance is the abstract's overstatement of the FinalBet performance metric and the insufficient separation between the distribution-matched internal test and the blind stream results. These are correctable with a revised abstract and a more cautious framing of the quantitative claims. I would also encourage the authors to consider moving the blind-test results into the main body, as they are the most realistic LSST-like evaluation currently available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the engineering is real and genuinely useful: three working AMPEL channels for LSST-class alert streams, with public job files, provenance tracking, and a local reproduction path. Second, the abstract's headline percentages are not supported by the evidence in the paper. The FinalBet '>80% of all extragalactic transients' claim is actually precision on a confident subset: with a probability threshold of 0.68 only 67% of transients receive any classification, and the correct-class rate for those is 'typically above 80%'. That works out to roughly half of all transients correctly classified, not 80%. The DESC blind test in Appendix B shows true classification fractions around 50–70%, consistent with that lower bound. The SNGuess 99% figure is measured on a held-out test that shares the training distribution, and the internal alert loader was never checked against the LSST-tools stream used for the blind test. The priors in FinalBet are calibrated on the same training sample, so their advertised benefit is partly an artifact of the evaluation setup.\n\nThat said, the paper earns credit where it matters. The authors are unusually honest: the body text explicitly says the priors are suboptimal, that the loader outputs were not compared, and that blind-test performance is worse. The appendices include the full blind-test confusion matrices. The workflows themselves are modular, documented, and reproducible—an actual contribution to the time-domain community as LSST comes online. The redshift-sampling unit and negative-flux handling are small but real pieces of new engineering. Citation practice is fair: the ML engines and AMPEL itself are properly credited to prior work.\n\nThe soft spots are, in proportion: the abstract overstates performance (load-bearing for the paper's claims, but fixable by relabeling what is measured); the internal test is distribution-matched so headline numbers should carry that caveat; code versions are pinned by branch rather than commit hash; and no error bars or sample sizes accompany the headline percentages. None of this undermines the infrastructure, but it does mean the advertised performance should not be taken at face value.\n\nWho benefits: anyone building broker pipelines for LSST, or groups wanting a reference implementation for real-time photometric classification. It deserves a serious referee. My recommendation: send it to peer review, but require the authors to either rescale the abstract to the confident-subset definition or report the blind-test numbers as the headline, and to state explicitly that the internal test is distribution-matched.","headline":"Solid, reproducible engineering; the abstract's headline numbers overstate what the paper actually measures, so the claims need rescaling before the paper is accepted.","tokens_in":21452,"tokens_out":1652,"would_cite":true,"duration_ms":17356,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents three modular AMPEL channels that classify simulated LSST alerts in real time, with SNGuess tagging about 99% of young supernovae and FinalBet correctly classifying more than 80% of extragalactic transients.","keywords":["AMPEL","real-time classification","LSST alert stream","supernova classification","photometric classification","ELAsTiCC","XGBoost","ParSNIP"],"falsifier":"Run the public jobfiles on the blinded alert stream used for the challenge, or later on real LSST alerts, and compare per-class true-classification fractions with the paper's roughly 99% young-supernova retention and more than 80% extragalactic success; the paper's own blinded appendix already indicates the fractions fall to roughly 50 to 70% for FinalBet, which would falsify the headline figures as a statement about deployed performance. Alternatively, feed identical simulated lightcurves through the two alert-generation tools and check whether the resulting alert streams differ systematically.","tokens_in":20233,"feed_emoji":"🔭","tokens_out":6752,"duration_ms":56990,"temperature":0.7,"pith_summary":"The paper argues that the practical bottleneck for real-time astronomy in the LSST era is not any single machine-learning classifier but the surrounding workflow: how alerts are filtered, models combined, priors applied, and results traced. It presents three public AMPEL channels built to fill that role. SNGuess uses gradient-boosted decision trees on early-detection features to flag young extragalactic transients, FollowMe selects an uncertain, label-worthy subset for spectroscopic follow-up, and FinalBet reweights FollowMe's deep-learning output with redshift, rate, and host-color priors. On the ELAsTiCC v1 simulated alert stream, the paper reports that SNGuess retains about 99% of young supernovae and FinalBet reaches successful classifications for more than 80% of extragalactic transients, with the full jobfiles released for reproduction.","feed_headline":"AMPEL tags 99% of young supernovae in simulated LSST alerts","feed_subtitle":"SNGuess, FollowMe and FinalBet show how modular real-time workflows can handle the LSST alert flood.","key_machinery":"The central machinery is the AMPEL tiered unit system, through which alerts flow from T0 filtering, T1 state assembly, T2 per-transient analysis, and T3 sample-level reaction, with every state, configuration, and output tied to a provenance record. The classification itself is carried by T2RiseDeclineStat, which extracts up to 32 early-detection features for SNGuess's XGBoost trees, and by T2RunParsnip, which fits ParSNIP, a hybrid generative model combining explicit physical variables such as redshift and peak time with latent variables learned by a variational autoencoder, at discrete sampled redshifts. FinalBet's T2ElasticcReport then multiplies the ParSNIP probabilities by priors built from training-sample redshift distributions, relative rates from a bright-transient survey, and host-galaxy color fits. The jobfile encodes the whole chain, so any user with an AMPEL environment can rerun it.","core_discovery":"On the paper's own terms, the discovery is that a code-to-data framework can host complete real-time classification programs for the LSST alert stream, and that the three supplied channels perform well on the ELAsTiCC v1 simulation. SNGuess, based on features designed for ZTF alerts and trained with XGBoost, correctly tags roughly 99% of young supernovae among alerts with few detections. FollowMe demonstrates that the ambiguous middle of a ParSNIP-based SNIa/SNIbc classification can be sampled to build a labeled follow-up set whose host-galaxy colors match the parent population. FinalBet adds explicit priors, including redshift distribution, relative rates, and host color, and achieves AUC scores above 0.9 for most classes, with trustworthy classifications correct well above 80% for most transient types.","pith_inferences":["The paper's own blinded test results show much lower true-classification fractions, roughly 50 to 70% for FinalBet; if that gap persists on real LSST data, the headline 99% and >80% figures should be read as matched-distribution upper bounds rather than deployed expectations.","A direct side-by-side test of the two alert-generation paths used here would isolate whether the performance drop comes from the priors or from unrecognized format differences between the internal loader and the official LSST tools.","The placeholder machinery described, including classifier combination, host information, and non-Gaussian redshift sampling, suggests the architecture itself may be the lasting contribution, allowing newer classifiers to be plugged in and compared fairly within the same jobfile.","One testable extension is to rerun SNGuess with the updated unit that handles negative-flux detections, which should reduce the AGN and microlensing contamination visible in the blinded confusion matrix."],"forward_implications":["If the reported performance holds, groups can take the public jobfiles and adapt them to their own VRO science programs without rebuilding an alert-processing infrastructure.","The SNGuess channel, combined with external multi-messenger constraints, can provide a pure stream of young-transient and counterpart candidates in real time.","FinalBet demonstrates that adding explicit population priors improves classification accuracy over the raw model output, at the cost of a biased selection that must be tracked.","Because AMPEL records provenance for each classification, follow-up decisions made in real time can later be linked to final, reprocessed classifications for sample correction.","The same workflows can be rerun at scale at a computer center or on the live AMPEL broker channel, making simulation-era development translate directly to the LSST stream."],"supporting_citations":[{"why":"Supplies the AMPEL architecture and provenance model that all three channels run on.","marker":"Nordin et al. 2019"},{"why":"Provides SNANA, the simulation tool used to generate the ELAsTiCC lightcurves and alerts.","marker":"Kessler et al. 2009"},{"why":"Introduces ParSNIP, the hybrid generative deep-learning model used by FollowMe and FinalBet.","marker":"Boone 2021"},{"why":"Source of the SNGuess feature set and the XGBoost training recipe based on real ZTF alerts.","marker":"Miranda et al. 2022"},{"why":"Implements the XGBoost gradient-boosted tree algorithm used by SNGuess.","marker":"Chen & Guestrin 2016"},{"why":"Provides the ZTF Bright Transient Survey relative rates used as the FinalBet rate prior.","marker":"Fremling et al. 2020"}],"fun_headline_variants":["AMPEL tags 99% of young supernovae in LSST alert simulations","Modular real-time classification for LSST achieves 99% young SN recall","Three AMPEL pipelines ready for LSST: 99% young SNe, 80%+ accurate","Open code-to-data toolkit for LSST: 99% young supernova detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline performance numbers are measured on a held-out test sample explicitly built from the same relative class, redshift, and rate distributions as the training data, and through an alert-generation path that the paper says was never checked against the official blind-test tools; if either of those differences matters, the reported percentages will not transfer to the real LSST stream.","fun_headline_variants_meta":{"raw":{"variants":["AMPEL tags 99% of young supernovae in LSST alert simulations","Modular real-time classification for LSST achieves 99% young SN recall","Three AMPEL pipelines ready for LSST: 99% young SNe, 80%+ accurate","Open code-to-data toolkit for LSST: 99% young supernova detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000319,"raw_usage":{"total_tokens":1846,"prompt_tokens":1037,"completion_tokens":809,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":718}},"tokens_in":653,"tokens_out":809,"duration_ms":7598,"temperature":1.0,"reasoning_tokens":718,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T12:46:22.567714+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the public jobfiles on the blinded alert stream used for the challenge, or later on real LSST alerts, and compare per-class true-classification fractions with the paper's roughly 99% young-supernova retention and more than 80% extragalactic success; the paper's own blinded appendix already indicates the fractions fall to roughly 50 to 70% for FinalBet, which would falsify the headline figures as a statement about deployed performance. Alternatively, feed identical simulated lightcurves through the two alert-generation tools and check whether the resulting alert streams differ systematically.","supporting_citations":[{"cited_title":"2019, , 631, A147","cited_arxiv_id":null,"evidence_quote":"Supplies the AMPEL architecture and provenance model that all three channels run on."},{"cited_title":"P., Cinabro , D., et al","cited_arxiv_id":null,"evidence_quote":"Provides SNANA, the simulation tool used to generate the ELAsTiCC lightcurves and alerts."},{"cited_title":"2021, , 162, 275","cited_arxiv_id":null,"evidence_quote":"Introduces ParSNIP, the hybrid generative deep-learning model used by FollowMe and FinalBet."},{"cited_title":"C., Nordin , J., et al","cited_arxiv_id":null,"evidence_quote":"Source of the SNGuess feature set and the XGBoost training recipe based on real ZTF alerts."},{"cited_title":"A., Sharma , Y., et al","cited_arxiv_id":null,"evidence_quote":"Provides the ZTF Bright Transient Survey relative rates used as the FinalBet rate prior."}],"review_version":1}