REVIEW 2 major objections 5 minor 2 cited by
Selection of optically variable active galactic nuclei via a random forest algorithm
T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that reducing a random forest from 162 features to eight—chosen to separate obscured AGN from inactive galaxies—raises the recall of known obscured AGN to about 68%, nearly doubling the earlier result.
desk verdict Useful g-band extension of the VST-COSMOS RF pipeline, but the headline 68% obscured-AGN recall is likely inflated by feature selection on the full labeled set before LOOCV. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine of the result is a two-stage pruning procedure. Stage one is a Kolmogorov-Smirnov screen: for each of the 162 features, compare the distribution over obscured AGN with the distribution over inactive galaxies, and keep the 25 with distance $D>0.25$. Stage two is iterative elimination: train a random forest with class weights balanced, rank features by impurity-based importance, delete the least important, retrain, and repeat down to seven features. The random forest itself is standard, with leave-one-out cross-validation used to score each setting; the load-bearing element is the choice of the final eight features, which the paper argues separate obscured AGN from inactive galaxies better than the original 162-feature set. The K-S screen also shows which features change most when synthetic visits replace real ones, linking the feature selection to the cadence tests.
What would settle it
Take the ks8 feature set to an independent sample of spectroscopically classified AGN from a different survey and measure the obscured-AGN recall; if it falls to roughly 50 percent, the level of the single-band classifiers here, the 68 percent gain is mostly selection on the same data used for scoring rather than a portable advantage.
Extended reading notes
Core claim
The paper's central claim is that the difficulty of finding obscured AGN through optical variability is primarily a feature-selection problem, not a data problem. By measuring, for every feature, the Kolmogorov-Smirnov distance between the feature distributions of obscured AGN and inactive galaxies, the authors isolate 25 features worth keeping and then iteratively strip the least important ones until eight remain: the mid-infrared color ch21, three optical colors ($u-B$, $r-i$, $i-z$), the HST stellarity index, and three $r$-band variability features (ExcessVar, GP_DRW_sigma, GP_DRW_tau). On the same labeled set, the eight-feature random forest returns ($68.1\pm1.2$)% recall for obscured AGN and 99.1% recall for unobscured AGN, at the cost of lower precision than feature-rich classifiers. The paper also reports that bivariate features combining $r$ and $g$ light curves add no benefit, that $g$-band features vanish from the final selection, and that replacing up to half of a light curve's visits with linearly interpolated synthetic points does not significantly change the results. These findings are presented as evidence that variability selection can be made competitive for obscured AGN and ready for application to LSST-era data.
Load-bearing premise
The performance numbers assume that choosing the best 25 and then eight features on the full labeled set and evaluating those same sources with leave-one-out cross-validation yields an unbiased estimate of classifier performance on new data.
Editorial extensions
If this is right
- Obscured AGN completeness is not locked to survey depth: better feature selection on the same light curves can raise recall from roughly 50% to about 68%.
- The final eight-feature set—two variability timescales, an excess-variance measure, four colors, and morphology—is a concrete recipe to test on other surveys.
- For LSST-like monitoring, the result implies that a modest number of well-chosen features should be used when completeness for obscured AGN is the goal, rather than the full feature zoo.
- Because synthetic imputed visits did not degrade the classifiers, observing strategies that do not sample two bands simultaneously can still feed the same selection method.
- The higher obscured-AGN recall is traded against higher contamination, so samples built with the eight-feature classifier will need follow-up to keep purity.
Reading between the lines
- Editor's inference: because the 25-then-8 feature choice was made on the same labeled set later used for leave-one-out evaluation, the 68.1% recall figure is likely an optimistic in-sample estimate; an independent transfer test could settle the true performance.
- Editor's inference: the dominance of $r$-band variability features in the final set, despite the $g$-band being bluer and generally expected to show larger AGN variability, hints that the advantage comes more from the $r$-band's denser real sampling than from physics; a redder-band version of the experiment would test this.
- Editor's inference: if the eight-feature set transfers, a practical extension would be to use it as a prior for anomaly detection in streaming alert streams from wide-field surveys, flagging obscured-AGN candidates before spectra are taken.
- Editor's inference: the K-S distance table provides a feature ranking that could be repurposed as a cheap pre-filter to remove inactive galaxies, reducing the labeled-set size needed for training on new surveys.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a random forest classification pipeline for selecting AGN from VST-COSMOS optical light curves, using 162 variability, color, and morphology features measured in the r and g bands, with imputation to 33 visits per band. The labeled set contains 380 AGN (217 Type I, 104 Type II, 211 MIR-selected) and 2163 non-AGN. Using leave-one-out cross-validation (LOOCV), the authors compare several feature sets and then, in Sect. 4.2, use K-S feature selection and iterative elimination to build a compact eight-feature classifier (ks8) that reaches 68.1% recall for spectroscopically confirmed obscured AGN, described as almost doubling the previous result from De Cicco et al. (2021). The paper also investigates the effect of replacing real visits with synthetic imputed visits and finds no large impact on the main metrics.
Significance. If the 68.1% obscured-AGN recall were an unbiased estimate, the result would be genuinely useful for time-domain AGN surveys, since completeness for obscured AGN is a known weakness of optical variability selection. The study also contributes a broad feature set, careful class-imbalance handling, and a transparent comparison across classifiers. However, the headline performance is not yet supported because feature selection and model selection are performed on the full labeled set before LOOCV; the reported recall is an in-sample selection metric. With a nested cross-validation or external test, the method could be valuable for LSST-era applications.
major comments (2)
- [Sect. 4.2 and Table 4] The reported 68.1% obscured-AGN recall is likely an optimistic in-sample figure because feature selection and model selection use the full labeled set before LOOCV. Specifically, the K-S test between obscured AGN and inactive galaxies selects 25 of 162 features, the feature-importance ranking is computed from an RF trained on the same 2543 sources, and the iterative elimination from 25 to 8 features is driven by the recall values shown in Table 4. LOOCV is then applied only to the already-determined ks8 feature set, so every held-out source has contributed to the choice of features and to the number of features retained. Because the K-S pre-filter was explicitly optimized to separate obscured AGN from inactive galaxies, the bias is likely largest for the obscured-AGN recall. The authors should re-estimate performance with nested cross-validation, with feature selection inside each training loop, or on an external validation sample.
- [Sect. 2.2 and Fig. 4] The most important feature in ks8, and in all classifiers tested, is the MIR color ch21, which is the same color entering the Donley et al. (2012) criterion used to build part of the AGN labeled set (211 MIR-selected sources). This creates a risk that the classifier is partly reproducing its own label definition rather than learning a new selection rule. The obscured-AGN recall itself is less directly affected because the 104 Type II AGN are spectroscopically classified, but the general AGN-selection numbers and the feature-importance interpretation are not. I ask for a concrete test: recompute the classification and feature importances either after removing MIR-selected AGN from the positive class or after dropping ch21 from the feature set, and quantify how much of the reported performance depends on this potentially circular feature.
minor comments (5)
- [Sect. 4.2] The claim that the 68.1% recall 'almost doubles' the De Cicco et al. (2021) value is not numerically correct: the Introduction quotes 21% recall for Type II AGN, and 68.1/21 ≈ 3.2; please correct the comparison (e.g., 'more than triples') or use an appropriate baseline.
- [Sects. 3.1 and 4.1] The synthetic-visit impact test is carried out on r-band data only; because the g-band light curves contain up to 16 synthetic points out of 33, a direct check of the g-band feature distributions would be useful, especially for the g-band features that appear in the K-S preselection (Autocor_length_g, Q31_g, MedianAbsDev_g).
- [Table 4 and Appendix A] The reported uncertainties are standard deviations across ten simulations with different random seeds; they do not include the sampling uncertainty of the LOOCV estimate or the variability introduced by feature selection. The text should state this limitation explicitly.
- [Sect. 2.2] The degree of overlap between the MIR-selected AGN and the spectroscopic Type I/II samples is not quantified; explicit counts or a Venn diagram would help the reader assess the ch21 circularity concern and interpret the per-subclass recalls.
- [Sect. 4.2] The text should be careful that ks8 is chosen for maximum obscured-AGN recall, not because it is uniformly the best classifier; ks9 has higher precision, accuracy, and F1, so the choice is a scientifically motivated trade-off rather than a global optimum.
Circularity Check
The headline 68.1% obscured-AGN recall is a fitted figure of merit: feature selection and model choice use the full labeled set before LOOCV, making the reported performance optimistically biased rather than a prediction for new sources.
-
fitted input called prediction
[Section 4.2, "Selection of obscured AGN"; validation protocol in Section 3.3; Table 4, bottom section.]
"Based on the results of the K-S test, we selected the features where the distance between the two distributions in a pair is large and the corresponding probability to get by chance a larger distance is small. We identified a possible threshold for distances D>0.25, which equals keeping 25 of the initial set of 162 features, and we built a classifier using only the selected features, which are reported in Table 6."
LOOCV (Sect. 3.3) leaves each source out only from tree fitting, but the features used by those trees were chosen earlier from the same 2,543-source LS: the K-S filter with D>0.25 separates obscured AGN from inactive galaxies using labels, and backward elimination drops the least important feature based on importance rankings computed on those same data; ks8 is then selected because it returns the highest obscured-AGN recall in Table 4. Thus each left-out source's label has already influenced which features the model uses, so (68.1±1.2)% is not an unbiased estimate for unseen sources; it is a fit-selected, optimistically biased figure of merit presented as predictive performance.
full rationale
The headline value (68.1±1.2)% obscured-AGN recall from classifier ks8 (Sect. 4.2, Table 4) is the paper's central quantitative claim. The chain that produces it is: (i) a K-S test compares obscured AGN vs inactive galaxies on the full 2,543-source LS and keeps 25 of 162 features with D>0.25; (ii) a feature-importance ranking computed from the same labeled data drives backward elimination from 25 down to seven features; (iii) ks8 is singled out because it gives the highest obscured-AGN recall; (iv) LOOCV is then run only on tree fitting for the already-chosen feature set. Because every left-out source's label participated in steps (i)-(iii), the LOOCV estimate is not independent: the 68.1% value is a fit-selected, optimistically biased in-sample figure, not an unbiased prediction for new sources. This matches the pattern of a fitted input called a prediction. The ch21/MIR-label overlap is not separately definitional: the obscured-AGN labels come from optical spectroscopy (Marchesi et al. 2016), while the Donley et al. (2012) MIR criterion uses F(8.0)/F(4.5) and F(5.8)/F(3.6), not the ch21 = m(4.5)-m(3.6) color; at most this is a feature-label correlation. Self-citations to De Cicco et al. (2021) establish the LOOCV protocol and feature definitions but are not load-bearing for the new ks8 result. A separate numerical inconsistency is that 68.1% versus the quoted 21% Type II recall is more than triple, not "almost doubles"; that is an accuracy issue rather than circularity. The qualitative band and cadence comparisons in Table 4 are less affected, but the central obscured-AGN improvement claim rests on the biased estimate, so partial circularity is present.
Assumptions & free parameters
free parameters (4)
- 15-day imputation window =
15 days
- K-S feature selection threshold =
D > 0.25
- Minimum feature set size =
7 features
- RF hyperparameters =
n_estimators 100-500, max_depth 10-20, min_samples_split 2-10, min_samples_leaf 4, max_features sqrt/log2
assumptions (4)
- domain assumption Linear interpolation between visits separated by up to 15 days approximates the true light curve well enough for variability features.
- domain assumption The literature-defined labels (Marchesi 2016 spectroscopic, Donley 2012 MIR, COSMOS2015 inactive galaxies) are correct and complete.
- ad hoc to paper Leave-one-out cross-validation after feature selection on the same data yields unbiased performance estimates.
- domain assumption Bivariate similarity measures computed on magnitude vectors are meaningful for AGN variability.
Cite this review
Pith. "Pith review of Selection of optically variable active galactic nuclei via a random forest algorithm." pith.science (2026). https://pith.science/paper/MUBOVK6B
@misc{pith2026250515819,
author = {Pith},
title = {Pith review of: Selection of optically variable active galactic nuclei via a random forest algorithm},
year = {2026},
howpublished = {\url{https://pith.science/paper/MUBOVK6B}},
note = {Machine review of arXiv:2505.15819}
}
read the original abstract
Context. A defining characteristic of active galactic nuclei (AGN) that distinguishes them from other astronomical sources is their stochastic variability, which is observable across the entire electromagnetic spectrum. Upcoming optical wide-field surveys, such as the Vera C. Rubin Observatory's Legacy Survey of Space and Time, are set to transform astronomy by delivering unprecedented volumes of data for time domain studies. This data influx will require the development of the expertise and methodologies necessary to manage and analyze it effectively. Aims. This project focuses on optimizing AGN selection through optical variability in wide-field surveys and aims to reduce the bias against obscured AGN. We tested a random forest (RF) algorithm trained on various feature sets to select AGN. The initial dataset consisted of 54 observations in the r-band and 25 in the g-band of the COSMOS field, captured with the VLT Survey Telescope over a 3.3-year baseline. Methods. Our analysis relies on feature sets derived separately from either band plus a set of features combining data from both bands, mostly characterizing AGN on the basis of their variability properties and obtained from their light curves. We trained multiple RF classifiers using different subsets of selected features and assessed their performance via targeted metrics. Results. Our tests provide valuable insights into the use of multiband and multivisit data for AGN identification. We compared our findings with previous studies and dedicated part of the analysis to potential enhancements in selecting obscured AGN. The expertise gained and the methodologies developed here are readily applicable to datasets from other ground- and space-based missions.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 2 Pith papers
-
VAR-PZnn: A machine-learning framework for AGN photometric redshifts using color and variability-based features
VAR-PZnn combines ZTF g-band variability features with optical, MIR and NIR photometry in a mixture-density network to estimate AGN photometric redshifts with σ_NMAD=0.058 and an 8.2% outlier fraction on a 72,728-sour...
-
Continuum optical-UV and X-ray variability of AGN: current results and future challenges
A comprehensive review of AGN continuum variability from optical/UV to X-rays, with no new data.
Reference graph
Works this paper leans on
-
[1]
Allevato, V ., Paolillo, M., Papadakis, I., & Pinto, C. 2013, ApJ, 771, 9
work page 2013
-
[2]
Amin, A., Anwar, S., Adnan, A., et al. 2016, IEEE Access, 4, 7940
work page 2016
-
[3]
1993, ARA&A, 31, 473
Antonucci, R. 1993, ARA&A, 31, 473
1993
-
[4]
T., Cappellaro, E., Greggio, L., et al
Botticella, M. T., Cappellaro, E., Greggio, L., et al. 2017, A&A, 598, A50
work page 2017
-
[5]
2001, Machine Learning, 45, 5, cited By 34434
Breiman, L. 2001, Machine Learning, 45, 5, cited By 34434
work page 2001
- [6]
-
[7]
& Charlot, S
Bruzual, G. & Charlot, S. 2003, Monthly Notices of the Royal Astronomical Society, 344, 1000
2003
-
[8]
Burbidge, G. R., Burbidge, E. M., & Sandage, A. R. 1963, Reviews of Modern Physics, 35, 947
work page 1963
Show all 50 references
-
[9]
& Schipani, P
Capaccioli, M. & Schipani, P. 2011, The Messenger, 146, 2
2011
-
[10]
T., Pignata, G., et al
Cappellaro, E., Botticella, M. T., Pignata, G., et al. 2015, A&A, 584, A62
2015
-
[11]
2024, A&A, 687, A246
Cavuoti, S., De Cicco, D., Doorenbos, L., et al. 2024, A&A, 687, A246
2024
-
[12]
2007, Int
Cha, S.-H. 2007, Int. J. Math. Model. Meth. Appl. Sci., 1 De Cicco, D., Bauer, F. E., Paolillo, M., et al. 2021, A&A, 645, A103 De Cicco, D., Bauer, F. E., Paolillo, M., et al. 2022, A&A, 664, A117 De Cicco, D., Paolillo, M., Covone, G., et al. 2015, A&A, 574, A112 De Cicco, D...
2007
-
[13]
L., Koekemoer, A
Donley, J. L., Koekemoer, A. M., Brusa, M., et al. 2012, ApJ, 748, 142
2012
-
[14]
A., Alexander, T., Crenshaw, D
Edelson, R. A., Alexander, T., Crenshaw, D. M., et al. 1996, ApJ, 470, 364 Euclid Collaboration, Mellier, Y ., Abdurro’uf, et al. 2024, A&A, in prep., arXiv:2405.13491
1996
-
[15]
2018, MNRAS, 481, 4311
Eyheramendy, S., Elorrieta, F., & Palma, W. 2018, MNRAS, 481, 4311
2018
-
[16]
2015, A&A, 579, A115
Falocco, S., Paolillo, M., Covone, G., et al. 2015, A&A, 579, A115
2015
-
[17]
2018, MNRAS, 479, 3858
Fu, L., Liu, D., Radovich, M., et al. 2018, MNRAS, 479, 3858
2018
-
[18]
J., Djorgovski, S
Graham, M. J., Djorgovski, S. G., Drake, A. J., et al. 2017, MNRAS, 470, 4112
2017
-
[19]
J., Pulgarin-Duque, L., Anderson, S
Green, P. J., Pulgarin-Duque, L., Anderson, S. F., et al. 2022, ApJ, 933, 180
2022
-
[20]
P., Cuillandre, J.-C., Bachelet, E., et al
Guy, L. P., Cuillandre, J.-C., Bachelet, E., et al. 2022, in Zenodo id. 5836022, V ol. 58, 5836022
2022
-
[21]
A., Förster, F., et al
Huijse, P., Estévez, P. A., Förster, F., et al. 2018, ApJS, 236, 12 Ivezi´c, Ž., Kahn, S. M., Tyson, J. A., et al. 2019, ApJ, 873, 111
2018
-
[22]
2013, in Proceedings - 2013 Humaine Asso- ciation Conference on Affective Computing and Intelligent Interaction, ACII 2013, V ol
Jeni, L., Cohn, J., & De la Torre, F. 2013, in Proceedings - 2013 Humaine Asso- ciation Conference on Affective Computing and Intelligent Interaction, ACII 2013, V ol. 2013
2013
-
[23]
Kim, D.-W., Protopapas, P., Bailer-Jones, C. A. L., et al. 2014, A&A, 566, A43
2014
-
[24]
2011, ApJ, 735, 68
Kim, D.-W., Protopapas, P., Byun, Y .-I., et al. 2011, ApJ, 735, 68
2011
-
[25]
& Sarajedini, V
Klesman, A. & Sarajedini, V . 2007, ApJ, 665, 225
2007
-
[26]
M., Aussel, H., Calzetti, D., et al
Koekemoer, A. M., Aussel, H., Calzetti, D., et al. 2007, ApJS, 172, 196
2007
-
[27]
& Richstone, D
Kormendy, J. & Richstone, D. 1995, ARA&A, 33, 581
1995
-
[28]
J., Ilbert, O., et al
Laigle, C., McCracken, H. J., Ilbert, O., et al. 2016, ApJS, 224, 24
2016
-
[29]
M., Cales, S., Moran, E
LaMassa, S. M., Cales, S., Moran, E. C., et al. 2015, ApJ, 800, 144
2015
-
[30]
2020, MNRAS, 493, 3825
Liu, D., Deng, W., Fan, Z., et al. 2020, MNRAS, 493, 3825
2020
-
[31]
2018, MNRAS, 478, 2388 LSST Science Collaboration, Abell, P
Liu, D., Fu, L., Liu, X., et al. 2018, MNRAS, 478, 2388 LSST Science Collaboration, Abell, P. A., Allison, J., et al. 2009, ArXiv e-prints, arXiv:0912.0201
2018 arXiv
-
[32]
L., Ross, N
MacLeod, C. L., Ross, N. P., Lawrence, A., et al. 2016, MNRAS, 457, 389
2016
-
[33]
2016, ApJ, 817, 34
Marchesi, S., Civano, F., Elvis, M., et al. 2016, ApJ, 817, 34
2016
-
[34]
A., Mattox, J
McLaughlin, M. A., Mattox, J. R., Cordes, J. M., & Thompson, D. J. 1996, ApJ, 473, 763
1996
-
[35]
P., Andreon, S., et al
Nakos, T., Willis, J. P., Andreon, S., et al. 2009, A&A, 494, 579
2009
-
[36]
2015, arXiv e-prints, arXiv:1506.00010
Nun, I., Protopapas, P., Sim, B., et al. 2015, arXiv e-prints, arXiv:1506.00010
2015 arXiv
-
[37]
E., Paolillo, M., De Cicco, D., & Bauer, F
Petrecca, V ., Papadakis, I. E., Paolillo, M., De Cicco, D., & Bauer, F. E. 2024, A&A, 686, A286
2024
-
[38]
2020, A&A, 634, A50
Poulain, M., Paolillo, M., De Cicco, D., et al. 2020, A&A, 634, A50
2020
-
[39]
Rees, M. J. 1984, ARA&A, 22, 471
1984
-
[40]
W., Starr, D
Richards, J. W., Starr, D. L., Butler, N. R., et al. 2011, ApJ, 733, 10
2011
-
[41]
Salpeter, E. E. 1964, ApJ, 140, 796
1964
-
[42]
& Webb, G
Sammut, C. & Webb, G. I., eds. 2010, Leave-One-Out Cross-Validation (Boston, MA: Springer US), 600–601 Sánchez-Sáez, P., Arredondo, J., Bayo, A., et al. 2023, A&A, 675, A195 Sánchez-Sáez, P., Lira, P., Cartier, R., et al. 2019, ApJS, 242, 10 Sánchez-Sáez, P., Reyes, I., Valenz...
2010
-
[43]
L., Koo, D
Sarajedini, V . L., Koo, D. C., Klesman, A. J., et al. 2011, ApJ, 731, 97 Savi´c, Ð. V ., Jankov, I., Yu, W., et al. 2023, ApJ, 953, 138
2011
-
[44]
B., Marshall, P
Schmidt, K. B., Marshall, P. J., Rix, H.-W., et al. 2010, ApJ, 714, 1194
2010
-
[45]
2008, A&A, 488, 73
Trevese, D., Boutsia, K., Vagnetti, F., Cappellaro, E., & Puccetti, S. 2008, A&A, 488, 73
2008
-
[46]
G., Majewski, S
Trevese, D., Kron, R. G., Majewski, S. R., Bershady, M. A., & Koo, D. C. 1994, ApJ, 433, 494
1994
-
[47]
G., Koo, D
Trevese, D., Pittella, G., Kron, R. G., Koo, D. C., & Bershady, M. 1989, AJ, 98, 108
1989
-
[48]
& Hernandez-Rivera, E
Tschopp, M. & Hernandez-Rivera, E. 2017, Quantifying Similarity and Distance Measures for Vector-Based Datasets: Histograms, Signals, and Probability Distribution Functions
2017
-
[49]
H., Courvoisier, T
Ulrich, M. H., Courvoisier, T. J. L., & Wamsteker, W. 1993, ApJ, 411, 125
1993
-
[50]
Urry, C. M. & Padovani, P. 1995, PASP, 107, 803 Vanden Berk, D. E., Wilhite, B. C., Kron, R. G., et al. 2004, ApJ, 601, 692 Appendix A: Best hyperparameters obtained per classifier Table A.1.Set of best values obtained from a grid search-based opti- mization of the five hyperp...
1995
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.