REVIEW 4 major objections 6 minor 5 cited by
Using Active Learning to Improve Quasar Identification for the DESI Spectra Processing Pipeline
T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Active learning lets DESI train its quasar classifier on its own spectra and match the old BOSS-trained model with a tenth of the data, while uncovering a redshift bias.
desk verdict A solid, honest active-learning pipeline paper with a real redshift-discovery side effect, but the headline 'meet or exceed eBOSS with 10% of the data' needs an independent validation set before it can be believed at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is an active-learning loop built around QuasarNET's line-finder outputs: for each of 13 wavelength boxes and each emission line, the network predicts a coarse confidence and a fine position estimate. A bootstrap ensemble of 200 QuasarNET copies, each trained on a resampled version of the current labeled set, classifies every unlabeled spectrum into three classes (NOT QSO, LOW-Z, HIGH-Z); disagreement defines an entropy $H = -\sum_i C_i \log_2 C_i$ that ranks spectra by confusion. A self-organizing map trained on the same rebinned spectra then rejects spectra in sparse cells (below the 15th percentile of cell counts), so the 1,000 highest-entropy surviving spectra sent to visual inspection are representative of the bulk quasar population rather than rare anomalies. The network itself is modified with zero padding, which permits two additional convolution layers, and dropout at rate 0.8 to improve generalization from the small training set.
What would settle it
Train the same modified QuasarNET on 6,700 spectra drawn uniformly at random from the same DESI visual-inspection pool, keeping the identical architecture, epochs, and validation split; if the random sample attains the same completeness and purity as the active-learned sample, then the gain attributed to active learning is not driven by which spectra were selected.
Extended reading notes
Core claim
The central claim is that careful selection of training spectra, not raw data volume, is what lets a deep line-finder network adapt to a new spectrograph. After two active-learning iterations, the paper obtains a weights file trained on DESI visual-inspection labels (about 5,600 spectra selected by the algorithm, later grown to 6,700 by including inspected outliers) that matches or exceeds the previous eBOSS-trained weights at DESI's nominal confidence threshold, achieving comparable validation completeness and purity with less than one tenth of the training data. On the unlabeled DESI DR1 'Guadalupe' sample, the new weights reduce classification confusion between repeated exposures and lower the scatter of redshift estimates. In building this comparison the paper identifies a systematic redshift defect: QuasarNET's coarse and fine line-position estimates clump toward the centers of its 13 wavelength boxes, producing oscillations in redshift estimates at box edges that are more visible with the smaller DESI training sample; the paper reports that adding cross-matched training data for Year 3 mitigates the oscillations without changing the underlying architecture.
Load-bearing premise
The load-bearing premise is that the 30 percent of the active-learning-selected, visually inspected spectra held out for validation represents the full population of DESI spectra the classifier will encounter; the paper's own redshift analysis shows the training sample does not cover the entire redshift range, so completeness and purity measured on that holdout may not transfer to the 3.6 million unlabeled spectra.
Editorial extensions
If this is right
- A DESI-only training set of roughly 6,700 labeled spectra is sufficient to replace the BOSS-era weights at DESI's nominal confidence threshold, matching or exceeding old completeness and purity.
- Repeated-exposure tests show the new weights classify the same object more consistently and with lower redshift scatter, which should make the DESI pipeline's quasar flags more stable across epochs.
- The bootstrap experiment indicates that the active-learning selection, not just increased dataset size, expands the quasar space covered by training, so future retraining runs can expect the same procedure to beat naive data collection.
- The box-edge redshift oscillations are a property of QuasarNET's 13-box line-position representation, so any future weights file for DESI will need either denser redshift coverage in the training sample or a change to the network output.
- The completeness gain is specific to the 0.95 confidence threshold used during active learning; at lower thresholds the eBOSS weights retain higher completeness, so threshold choice is part of the method.
Reading between the lines
- The same three-class entropy plus self-organizing-map outlier-rejection loop should transfer to any line-detection network in a large survey; the ingredients are an ensemble, a confusion score, and a density map over inputs, none of which are specific to quasars.
- A natural extension the paper does not pursue is to replace the 13-box coarse/fine output with a continuous position regression or add a loss term that penalizes box-center clustering; that would attack the oscillation at its source rather than mitigating it with more training data.
- If the validation numbers generalize, only a few thousand human labels were needed to retrain the classifier for a new instrument, suggesting visual-inspection budgets for future surveys can be reduced by orders of magnitude for similar retraining tasks.
- The reported completeness and purity are measured on a holdout drawn from the same active-learning-selected pool; an independent test set drawn from the full survey, not from the selected pool, is the direct check on whether the gains carry to the 3.6 million unlabeled spectra.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes modifications to QuasarNET, a convolutional network used for quasar classification and redshift estimation in the DESI spectroscopic pipeline. The authors add padding and dropout to the architecture, then run an active learning loop that uses a 200-network bootstrap ensemble and a self-organizing-map outlier rejection step to select 2000 additional DESI spectra for visual inspection. The resulting 6698-spectrum DESI AL2 dataset is used to train a candidate weights file, which is compared with an eBOSS-trained weights file on a 30% validation split of DESI AL2 in completeness and purity, on repeat-exposure consistency over the unlabeled Guadalupe sample, and in redshift estimation. The paper reports comparable or better classification at the DESI threshold of 0.95 with less than a tenth of the eBOSS training data, improved consistency on repeated observations, and identifies box-edge oscillations in QuasarNET redshift estimates caused by incomplete redshift coverage in the active-learning-selected training sample.
Significance. The active learning pipeline is well motivated and the paper provides a useful practical demonstration: all plots are reproducible from the released code and data, the bootstrap control for dataset size is a valuable check, and the repeat-exposure consistency test is an informative indirect metric. The SOM-based outlier rejection is a sensible guard against labeling rare pathological spectra, and the discovery and characterization of the box-edge redshift oscillations is a concrete contribution to understanding QuasarNET. However, the central quantitative claim—that the DESI-trained weights match or beat the eBOSS weights on DESI data with 10% of the training data—is not yet established for the DESI population because the validation set is a split of the active-learning-selected sample rather than an independent random sample. The paper's own Section 4.3.2 shows that the selected sample does not cover the full redshift range. With an independent validation sample or a clearly restricted claim, the result would be a solid methods paper.
major comments (4)
- [Section 3, Eqs. (3.1)-(3.2); Section 4.2, Figs. 5-7] The headline comparison is measured on a 30% random split of DESI AL2, but DESI AL2 is built by two active-learning rounds that deliberately select the 1000 highest-entropy spectra after SOM outlier rejection, so it is not a random sample of the DESI spectral population. The validation split therefore overrepresents spectra near QuasarNET's decision boundaries and underrepresents outliers, and training on those boundary objects while validating on the same enriched distribution can inflate the measured completeness/purity gain relative to what would be observed on the full unlabeled pool. The bootstrapped size control in Fig. 6 shows the added spectra help within that distribution, but it does not establish representativeness. The authors should either validate on an independent set of DESI spectra not selected by active learning (for example, random or survey-targeted spectra with visual-inspection labels) or clearly restrict the central claim to the DESI AL2 validation distribution.
- [Section 4.3.2; Conclusions] The paper's own Section 4.3.2 and Conclusions concede that the final training sample does not cover the entire redshift range, producing redshift oscillations at the edges of QuasarNET's boxes, and that for DESI Year 3 processing the oscillations were returned to the eBOSS level by adding additional DESI training data cross-matched from eBOSS. This means the weights file evaluated in Section 4.2 is a candidate rather than the production weights, and the redshift pathology is not an irrelevant detail because QuasarNET redshifts drive the redrock rerun in the DESI pipeline. The article should state clearly which weights are being claimed as an improvement and should report the redshift-coverage limitation alongside the classification claim.
- [Section 4.2, Fig. 7] The claim of meeting or exceeding eBOSS is threshold-specific: at lower confidence thresholds the eBOSS DR12 weights exceed the DESI AL2 weights in completeness with similar or better purity, as the authors state. Since the paper's title and abstract advertise improved quasar identification generally, the conclusions should carry an explicit caveat that the improvement is demonstrated only at the DESI nominal 0.95 threshold and on the AL2 validation split.
- [Section 4.3.1, Figs. 8-9] The repeat-exposure consistency results support the narrower claim that the DESI weights classify repeated observations more consistently, but consistency is not accuracy; the paper itself cautions that more consistent results need not be more correct. The abstract's wording 'more consistently classify objects in the same way' is appropriate, but it should not be summarized as improved classification accuracy on unlabeled data.
minor comments (6)
- [Abstract and Section 1] The abstract in the paper text says 'achieve similar performance' while the arXiv abstract says 'meet or exceed'; these should be aligned, and the phrase 'systemic error' should be 'systematic error'.
- [Section 4.2, after Fig. 7] The sentence 'DESI VI 2 outperforms the eBOSS weights file' appears to refer to DESI AL 2, not DESI VI 2; please correct the label.
- [Section 4.2] After removing the two corrupted spectra the total is 6698, but the text states that the validation set is 2010 spectra and 'the remaining 6790 are available for use as training data'; these numbers are inconsistent and should be reconciled.
- [Section 1] There is a duplicated word in 'Classification therefore only only uses the first 13 outputs'; remove the second 'only'.
- [Section 2] The dropout rate is reported as tuned empirically to 0.8, but no sensitivity analysis is shown; a brief statement of the range tested would help readers assess the robustness of the architecture choice.
- [General] The figure and table numbering is clear, but the caption of Figure 5 refers to the eBOSS run as 'yellow' while the plot legend appears to use orange; please make the color references consistent.
Circularity Check
No significant circularity: the central results are held-out empirical comparisons, not predictions that reduce to their own inputs.
full rationale
None of the enumerated circularity patterns is present in this paper. The central claims are empirical comparisons rather than derivations: a new QuasarNET weights file is trained on DESI-selected spectra and evaluated (i) on a 30% held-out split of the DESI AL2 truth table and (ii) on repeat exposures of unlabeled Guadalupe spectra. The held-out split is disjoint from the training split, and the same validation split is used for the eBOSS-trained comparison, so the headline 'meet or exceed the previously trained weights file in completeness and purity calculated on the validation dataset with less than one tenth of the amount of training data' is a measured outcome rather than a definitional identity. Active learning does select the 1000 highest-entropy spectra (Eq. 3.2) after SOM outlier rejection, and the confidence threshold 0.95 is used both for ensemble classification and for the reported evaluation, but this is an iterative training-data-selection loop, not a target quantity being inserted into its own definition; no fitted parameter is renamed as a prediction. The dropout rate of 0.8 is reported as an empirical tuning choice, not as a prediction, and the bootstrapped controls in Figure 6 explicitly test against the alternative that the gains are only dataset-size effects. The paper itself flags the generalization limits of its training-sample redshift coverage in Section 4.3.2 ('our resulting training sample does not cover the entire redshift range'), but that is a stated coverage limitation, not a circular step. No load-bearing self-citation chain is used: QuasarNET and the eBOSS training data are cited as external inputs, and the claimed improvements are benchmarked against those external weights on data not used for their training. The representativeness concern about validating on a split of the active-learning-selected sample is a legitimate external-validity caveat, but it does not make the derivation circular.
Assumptions & free parameters
free parameters (6)
- dropout rate =
0.8
- number of bootstrap networks =
200
- SOM outlier threshold =
15th percentile of cell counts
- HIGHZ/LOWZ redshift split =
z = 2.1
- spectra selected per active learning round =
1000
- number of active learning iterations =
2
assumptions (4)
- domain assumption Visual inspection labels from DESI Survey Validation are accurate ground truth for quasar classification and redshift.
- domain assumption Ensemble entropy from bootstrapped QuasarNET classifications is a valid uncertainty measure for active learning.
- domain assumption Sparse cells in the self-organizing map contain outlier spectra whose exclusion improves representativeness.
- domain assumption Consistency across repeat exposures is a meaningful proxy for classification and redshift quality when truth labels are absent.
Cite this review
Pith. "Pith review of Using Active Learning to Improve Quasar Identification for the DESI Spectra Processing Pipeline." pith.science (2026). https://pith.science/paper/QXJRXDYU
@misc{pith2026250501596,
author = {Pith},
title = {Pith review of: Using Active Learning to Improve Quasar Identification for the DESI Spectra Processing Pipeline},
year = {2026},
howpublished = {\url{https://pith.science/paper/QXJRXDYU}},
note = {Machine review of arXiv:2505.01596}
}
read the original abstract
The Dark Energy Spectroscopic Instrument (DESI) survey uses an automatic spectral classification pipeline to classify spectra. QuasarNET is a convolutional neural network used as part of this pipeline originally trained using data from the Baryon Oscillation Spectroscopic Survey (BOSS). In this paper we implement an active learning algorithm to optimally select spectra to use for training a new version of the QuasarNET weights file using only DESI data, specifically to improve classification accuracy. This active learning algorithm includes a novel outlier rejection step using a Self-Organizing Map to ensure we label spectra representative of the larger quasar sample observed in DESI. We perform two iterations of the active learning pipeline, assembling a final dataset of 5600 labeled spectra, a small subset of the approx 1.3 million quasar targets in DESI's Data Release 1. When splitting the spectra into training and validation subsets we meet or exceed the previously trained weights file in completeness and purity calculated on the validation dataset with less than one tenth of the amount of training data. The new weights also more consistently classify objects in the same way when used on unlabeled data compared to the old weights file. In the process of improving QuasarNET's classification accuracy we discovered a systemic error in QuasarNET's redshift estimation and used our findings to improve our understanding of QuasarNET's redshifts.
Forward citations
Cited by 5 Pith papers
-
Lyman Break Galaxy selection and redshift measurement with supervised contrastive learning
A supervised weighted contrastive learning pipeline, zlbg, improves contaminant rejection in DESI Lyman Break Galaxy spectra while matching the redshift accuracy of the existing lbgNET pipeline.
-
DESI DR2 Results IV: Alcock-Paczy\'nski Measurements from the Lyman Alpha Forest and Cosmological Constraints
The full shape of DESI DR2 Lyman-alpha forest correlations constrains the distance ratio DM/DH at z=2.33 to 1.0%, twice as precise as BAO alone.
-
Early results in the search for extreme coronal line emitters with the Dark Energy Spectroscopic Instrument
A DESI early-data search found three tidal-disruption-event-linked extreme coronal line emitters, giving a galaxy-normalized rate of 5 (+5/-3) × 10^-6 galaxy^-1 yr^-1 at z ≈ 0.2.
-
Luminosity function of quasars at $1.0<z<3.5$ from SDSS and DESI
A uniform color-selected sample of 62,426 SDSS/DESI quasars yields a new quasar luminosity function at z=1.0-3.5, with higher faint-end density and a higher bright end at z>2.5 than earlier measurements.
-
Validation of the DESI DR2 Ly$\alpha$ forest full-shape analysis
The DESI DR2 Lyman-alpha full-shape analysis passes validation for BAO and Alcock-Paczynski parameters on 400 mocks and blinded data, while f-sigma-8 is rejected due to a roughly 10% mock bias.
Reference graph
Works this paper leans on
-
[1]
M. Levi, C. Bebek, T. Beers, R. Blum, R. Cahn, D. Eisenstein et al.,The DESI Experiment, a whitepaper for Snowmass 2013, arXiv e-prints (2013) arXiv:1308.0847 [1308.0847]
arXiv 2013
-
[2]
DESI Collaboration, A. Aghamousa, J. Aguilar, S. Ahlen, S. Alam, L.E. Allen et al.,The DESI Experiment Part I: Science,Targeting, and Survey Design, arXiv e-prints (2016) arXiv:1611.00036 [1611.00036]
arXiv 2016
-
[3]
DESI Collaboration, A. Aghamousa, J. Aguilar, S. Ahlen, S. Alam, L.E. Allen et al.,The DESI Experiment Part II: Instrument Design, arXiv e-prints (2016) arXiv:1611.00037 [1611.00037]
arXiv 2016
-
[4]
DESI Collaboration, B. Abareshi, J. Aguilar, S. Ahlen, S. Alam, D.M. Alexander et al., Overview of the Instrumentation for the Dark Energy Spectroscopic Instrument, The Astronomical Journal 164 (2022) 207 [2205.10939]
arXiv 2022
- [5]
- [6]
-
[7]
C. Poppett, L. Tyas, J. Aguilar, C. Bebek, D. Bramall, T. Claybaugh et al.,Overview of the Fiber System for the Dark Energy Spectroscopic Instrument, The Astronomical Journal168 (2024) 245
work page 2024
- [8]
Show all 35 references
-
[9]
Adame, J
DESI Collaboration, A.G. Adame, J. Aguilar, S. Ahlen, S. Alam, G. Aldering et al.,Validation of the Scientific Program for the Dark Energy Spectroscopic Instrument, The Astronomical Journal 167 (2024) 62 [2306.06307]
2024 arXiv
-
[10]
Adame, J
DESI Collaboration, A.G. Adame, J. Aguilar, S. Ahlen, S. Alam, G. Aldering et al.,The Early Data Release of the Dark Energy Spectroscopic Instrument, The Astronomical Journal168 (2024) 58 [2306.06308]
2024 arXiv
-
[11]
Abdul-Karim, A.G
DESI Collaboration, M. Abdul-Karim, A.G. Adame, D. Aguado, J. Aguilar, S. Ahlen et al., Data Release 1 of the Dark Energy Spectroscopic Instrument, arXiv e-prints (2025) arXiv:2503.14745 [2503.14745]
2025 arXiv
-
[12]
Adame, J
DESI Collaboration, A.G. Adame, J. Aguilar, S. Ahlen, S. Alam, D.M. Alexander et al.,DESI 2024 III: Baryon Acoustic Oscillations from Galaxies and Quasars, arXiv e-prints (2024) arXiv:2404.03000 [2404.03000]
2024 arXiv
-
[13]
Adame, J
DESI Collaboration, A.G. Adame, J. Aguilar, S. Ahlen, S. Alam, D.M. Alexander et al.,DESI 2024 V: Full-Shape Galaxy Clustering from Galaxies and Quasars, arXiv e-prints (2024) arXiv:2411.12021 [2411.12021]
2024 arXiv
-
[14]
Adame, J
DESI Collaboration, A.G. Adame, J. Aguilar, S. Ahlen, S. Alam, D.M. Alexander et al.,DESI 2024 IV: Baryon Acoustic Oscillations from the Lyman Alpha Forest, arXiv e-prints (2024) arXiv:2404.03001 [2404.03001]
2024 arXiv
-
[15]
Adame, J
DESI Collaboration, A.G. Adame, J. Aguilar, S. Ahlen, S. Alam, D.M. Alexander et al.,DESI 2024 VI: Cosmological Constraints from the Measurements of Baryon Acoustic Oscillations, arXiv e-prints (2024) arXiv:2404.03002 [2404.03002]
2024 arXiv
-
[16]
Adame, J
DESI Collaboration, A.G. Adame, J. Aguilar, S. Ahlen, S. Alam, D.M. Alexander et al.,DESI 2024 VII: Cosmological Constraints from the Full-Shape Modeling of Clustering Measurements, arXiv e-prints (2024) arXiv:2411.12022 [2411.12022]
2024 arXiv
-
[17]
J. Guy, S. Bailey, A. Kremin, S. Alam, D.M. Alexander, C. Allende Prieto et al.,The Spectroscopic Data Processing Pipeline for the Dark Energy Spectroscopic Instrument, The Astronomical Journal 165 (2023) 144 [2209.14482]
2023 arXiv
-
[18]
Schlafly, D
E.F. Schlafly, D. Kirkby, D.J. Schlegel, A.D. Myers, A. Raichoor, K. Dawson et al.,Survey Operations for the Dark Energy Spectroscopic Instrument, The Astronomical Journal166 (2023) 259 [2306.06309]
2023 arXiv
-
[19]
Bailey,Redrock: Spectroscopic classification and redshift fitting for the dark energy spectroscopic instrument, In Prep.(2025)
S. Bailey,Redrock: Spectroscopic classification and redshift fitting for the dark energy spectroscopic instrument, In Prep.(2025)
2025
-
[20]
Chaussidon, C
E. Chaussidon, C. Yèche, N. Palanque-Delabrouille, D.M. Alexander, J. Yang, S. Ahlen et al., Target Selection and Validation of DESI Quasars, The Astrophysical Journal944 (2023) 107
2023
-
[21]
Busca and C
N. Busca and C. Balland,QuasarNET: Human-level spectral classification and redshifting with Deep Neural Networks, arXiv:1808.09955 [astro-ph](2018)
2018 arXiv
-
[22]
J. Farr, A. Font-Ribera and A. Pontzen,Optimal strategies for identifying quasars in DESI, Journal of Cosmology and Astroparticle Physics2020 (2020) 015. – 22 –
2020
-
[23]
Pâris, P
I. Pâris, P. Petitjean, N.P. Ross, A.D. Myers, É. Aubourg, A. Streblyanska et al.,The Sloan Digital Sky Survey Quasar Catalog: Twelfth data release, Astronomy & Astrophysics597 (2017) A79 [1608.06483]
2017 arXiv
-
[24]
Smee, J.E
S.A. Smee, J.E. Gunn, A. Uomoto, N. Roe, D. Schlegel, C.M. Rockosi et al.,The Multi-object, Fiber-fed Spectrographs for the Sloan Digital Sky Survey and the Baryon Oscillation Spectroscopic Survey, The Astronomical Journal146 (2013) 32
2013
- [25]
-
[26]
Srivastava, G
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov,Dropout: A Simple Way to Prevent Neural Networks from Overfitting, Journal of Machine Learning Research 15 (2014) 1929
2014
-
[27]
Alexander, T.M
D.M. Alexander, T.M. Davis, E. Chaussidon, V.A. Fawcett, A. X. Gonzalez-Morales, T.-W. Lan et al.,The DESI Survey Validation: Results from Visual Inspection of the Quasar Survey Spectra, The Astronomical Journal165 (2023) 124
2023
-
[28]
T.-W. Lan, R. Tojeiro, E. Armengaud, J.X. Prochaska, T.M. Davis, D.M. Alexander et al.,The DESI Survey Validation: Results from Visual Inspection of Bright Galaxies, Luminous Red Galaxies, and Emission-line Galaxies, The Astrophysical Journal943 (2023) 68
2023
-
[29]
Settles,Active Learning, Synthesis Lectures on Artificial Intelligence and Machine Learning, Springer International Publishing, Cham (2012)
B. Settles,Active Learning, Synthesis Lectures on Artificial Intelligence and Machine Learning, Springer International Publishing, Cham (2012)
2012
-
[30]
Lochner and B.A
M. Lochner and B.A. Bassett,Astronomaly: Personalised active anomaly detection in astronomical data, Astronomy and Computing36 (2021) 100481
2021
-
[31]
Walmsley, L
M. Walmsley, L. Smith, C. Lintott, Y. Gal, S. Bamford, H. Dickinson et al.,Galaxy Zoo: probabilistic morphology through Bayesian CNNs and active learning, Monthly Notices of the Royal Astronomical Society491 (2020) 1554
2020
-
[32]
Kohonen,Self-organized formation of topologically correct feature maps, Biological Cybernetics 43 (1982) 59
T. Kohonen,Self-organized formation of topologically correct feature maps, Biological Cybernetics 43 (1982) 59
1982
-
[33]
J.E. Geach,Unsupervised self-organized mapping: a versatile empirical tool for object selection, classification and redshift estimation in large surveys, Monthly Notices of the Royal Astronomical Society 419 (2012) 2633
2012
-
[34]
Wright, H
A.H. Wright, H. Hildebrandt, J.L.v.d. Busch and C. Heymans,Photometric redshift calibration with self-organising maps, Astronomy & Astrophysics637 (2020) A100
2020
-
[35]
Abolfathi,SOMVIZ, 2022
B. Abolfathi,SOMVIZ, 2022. – 23 –
2022
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.