Pith. sign in

REVIEW 3 major objections 3 minor 32 references

X-ray data were not the cause of a galaxy classifier's accuracy drop; the harder X-ray-selected sample was.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:48 UTC pith:4FNNWK37

load-bearing objection A clean controlled comparison shows X-rays don't hurt accuracy, but the paper's claim that sample selection explains the original 8.25-point drop is unsupported by its own numbers. the 3 major comments →

arxiv 2607.13233 v2 pith:4FNNWK37 submitted 2026-07-14 astro-ph.GA astro-ph.IM

When X-ray Features Fail to Identify Intrinsic Emitters: Label Noise and Luminosity Overlap in Machine Learning Classification of AGN and Star-forming Galaxies

Jaymin Ding This is my paper
classification astro-ph.GA astro-ph.IM
keywords active galactic nucleistar-forming galaxiesmachine learning classificationlabel noiseX-ray luminosityBPT diagramrandom forestpermutation importance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

A previous experiment found that adding an X-ray flux feature to a Random Forest classifier of active galaxies and star-forming galaxies lowered accuracy from 97.51% to 89.26%, contradicting the idea that X-rays are a reliable AGN diagnostic. This paper argues that the decrease is mostly an artifact of comparing two different samples: the X-ray-detected subsample is harder to classify, and the feature itself changes accuracy by only -0.01 percentage points on a fixed sample. The deeper reason X-rays fail to help is that the training labels, derived from optical emission-line ratios on the BPT diagram (the standard optical line-ratio classification), are least reliable exactly at the boundary where X-rays would be most informative, so the model learns to ignore the feature. The paper also shows that AGN and star-forming galaxies overlap heavily in X-ray luminosity in this moderate-luminosity sample, so the physical signal is weaker than the prevailing theory assumes.

Core claim

The central claim is that the previously reported accuracy decrease reflects predominantly sample selection rather than the X-ray feature itself. A controlled three-way cross-validation decomposition shows that restricting to the X-ray-detected subsample lowers accuracy by 3.52 percentage points before any X-ray feature is added, while adding the feature on the fixed subsample changes accuracy by only -0.01 points. The X-ray feature nevertheless has a small but significantly negative permutation importance: shuffling its values slightly improves held-out predictions, meaning the model's limited use of it is net-detrimental. The explanation offered is instance-dependent label noise: BPT-deriv

What carries the argument

The load-bearing analytical tool is a controlled three-way cross-validation decomposition that isolates the effect of changing the sample (full optical sample vs X-ray-detected subsample) from the effect of adding the X-ray feature, together with permutation importance to probe the feature's net contribution within a trained model. The interpretative mechanism is instance-dependent label noise: the probability of a wrong training label is highest near the BPT demarcation line, exactly where X-ray data would be most informative, so the random forest learns to avoid splitting on the X-ray feature, using it sparingly and to slight net cost.

Load-bearing premise

The 277 objects with complete photometry are assumed to represent the full 312-object X-ray-detected sample; if the 35 excluded objects cluster near the BPT boundary, the attribution of the accuracy drop to sample selection rather than the feature could be wrong.

What would settle it

Inspect the BPT positions of the 35 objects excluded for incomplete photometry: if their median distance to the Kewley demarcation curve is closer to 0.123 dex than to 0.743 dex, the sample-selection attribution would be weakened because the excluded objects would be disproportionately boundary-concentrated.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Future classifiers on optically-selected samples should use independent label sources or noise-robust training methods, since BPT-derived labels are least reliable near the boundary.
  • The canonical 10^42 erg/s X-ray luminosity threshold is not a clean separator for moderate-luminosity samples: 41.6% of AGN fall below it and 13.5% of star-forming galaxies fall above it.
  • Adding a physically informative feature does not guarantee improved accuracy when training labels are noisy; the model may down-weight the feature even if it would help on true classes.
  • The apparent 8.25% accuracy drop in the previous paper is reinterpreted as a 3.52-point sample-selection effect and a negligible -0.01-point feature effect, so X-ray data should not be abandoned as a diagnostic on this evidence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The negative permutation importance, though small, suggests a testable prediction: in a larger or deeper X-ray sample where the feature is less sparse, the X-ray signal might become positively important if the label-noise mechanism is the main obstacle.
  • The boundary-concentration result implies that retraining the same classifier with labels from an X-ray-excess selection (or another independent diagnostic) should substantially increase X-ray feature importance if the paper's explanation is correct.
  • The decomposition highlights a general methodological hazard: comparing classifier accuracy across different feature-availability regimes conflates sample and feature effects, so future multi-wavelength studies should always run a sample-holding-fixed feature ablation.
  • The luminosity overlap may be partly an artifact of the optical selection function; samples selected by X-ray brightness or deeper X-ray coverage would likely show more separation between AGN and star-forming galaxies.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. This paper investigates a previously reported counterintuitive result from Ding & Rodriguez (2024), in which adding an X-ray flux feature to a Random Forest classifier of AGN and star-forming galaxies decreased accuracy from 97.51% to 89.26%. The authors propose two explanations: an astrophysical one based on luminosity overlap between AGN and SFGs around the canonical 10^42 erg/s threshold, and a machine-learning one based on instance-dependent label noise in BPT-derived training labels. Using a three-way cross-validation decomposition, they report that restricting to the X-ray-detected subsample lowers accuracy by 3.52 percentage points before any X-ray feature is added, while adding the X-ray feature changes accuracy by only -0.01 percentage points. They also report that misclassified objects cluster near the Kewley demarcation line and that the X-ray flux feature has a small but significantly negative permutation importance. The paper concludes that the apparent accuracy decrease reflects predominantly sample selection rather than the X-ray feature itself, and recommends independent labels or noise-robust training for future classifiers.

Significance. If the central claim were quantitatively supported, the paper would be a useful cautionary study for multi-wavelength machine-learning classification in astrophysics, showing that apparent feature-induced performance drops can be dominated by sample composition rather than feature content. The controlled fixed-sample comparison (configurations b vs c) is clean and credible, and the boundary-concentration analysis (KS D=0.745, p=1.67e-9) is a specific, falsifiable diagnostic that supports the label-noise narrative. The paper also makes a fair point that X-ray luminosity overlap is real in optically selected moderate-luminosity samples. However, the central attribution of the original 8.25-point drop to sample selection is not quantitatively established by the paper's own decomposition, which leaves a large residual unexplained. This is a load-bearing issue that must be resolved before the paper's headline conclusion can be accepted.

major comments (3)
  1. [Section 4, Section 5] The paper's own numbers do not support the statement that the reported accuracy decrease 'reflects predominantly sample selection.' The original drop is 8.25 percentage points (97.51 to 89.26). The decomposition in Section 4 gives a sample effect of 3.52 pp (97.37 to 93.85 from configuration a to b) and a feature effect of -0.01 pp (b to c). This leaves roughly 4.7 pp of the original drop unaccounted for. Section 5 nevertheless concludes that the drop is 'driven predominantly by sample selection.' That conclusion is not supported by the measured sample effect, which is less than half of the total drop. Please locate the residual explicitly: possible sources include differences in sample construction (28,708 vs 25,668; 312 vs 277), the change from an 80-20 split to 5-fold CV, and any other preprocessing changes. Without reconciling this residual, the central attribution is not established
  2. [Section 4] The sample effect is measured on the 277-object subsample with complete photometry, but the original X-ray model used 312 objects. The 35 excluded objects are dropped to avoid the classifier learning missingness patterns, but the paper never tests whether those objects are disproportionately concentrated near the BPT boundary or have systematically different X-ray properties. If the excluded objects are preferentially hard boundary cases, the measured sample effect could be underestimated. This concern is directly relevant to the decomposition's validity. Even if this does not fully close the ~4.7 pp residual identified above, it should be addressed with a sensitivity test or at least an explicit discussion of the excluded objects' properties.
  3. [Section 5, 'It is worth noting...'] The Discussion draws a strong conclusion from the decomposition, stating that 'the apparent 8.25% accuracy drop reported in Ding & Rodriguez (2024) is driven predominantly by sample selection.' Given the residual gap quantified above, this is premature. The conclusion should be softened to what the controlled comparison actually shows: on a fixed sample, the X-ray feature has negligible effect on accuracy. The paper can still argue that sample selection plays a role, but the quantitative claim of predominance requires accounting for the missing ~4.7 pp.
minor comments (3)
  1. [Section 3] The statement 'k-corrections are small at these redshifts and are neglected' is unsupported. Since the X-ray band is 0.2-12 keV and the sample spans a range of redshifts, a brief justification or a quantification of the expected k-correction magnitude would strengthen the luminosity-overlap analysis.
  2. [Section 4, Figure 2] The text says the permutation importance confidence interval lies 'entirely below zero across 100 permutations,' but the error bars are described as one standard deviation. Please specify how the confidence interval is constructed (e.g., 1-sigma interval from the permutation distribution) so the reader can interpret 'significantly negative' precisely.
  3. [Abstract and Section 1] The abstract states the decrease 'reflects predominantly sample selection,' but the Introduction already notes the two accuracies were measured on different samples. The wording in the abstract should be qualified to reflect the decomposition's actual coverage (3.52 pp of the 8.25 pp drop) until the residual is explained.

Circularity Check

0 steps flagged

No significant circularity: the controlled decomposition is an empirical comparison, and the paper's stated predictions are tested rather than fitted.

full rationale

The paper's derivation chain is not circular. The central quantitative claim—that the prior accuracy drop is mainly sample selection—comes from a controlled three-way cross-validation decomposition (configurations (a), (b), (c) in Section 4) whose inputs are independently measured accuracies; no parameter is fitted to the 8.25 percentage-point gap, and the conclusion is not an identity. In fact, the reported 3.52 pp sample effect plus -0.01 pp feature effect do not reproduce the original 8.25 pp drop, which is a quantitative-support concern but not a circularity. The two stated predictions—adding the X-ray feature leaves overall accuracy essentially unchanged, and the model's limited use of it is net-negative—are tested on held-out data via 5-fold cross-validation and permutation importance, not enforced by construction. The instance-dependent label-noise mechanism is grounded in external literature (Song et al. 2020; Agostino & Salim 2019; Birchall et al. 2020, 2022), and the boundary-concentration diagnostic, while measured using the same BPT variables that also generate the labels, is presented as a consistency check rather than the sole proof. Self-citations to Ding & Rodriguez (2024) identify the prior result being explained and are not used as an unverified load-bearing theorem; the paper even flags the earlier circularity concern raised in peer review and attempts to resolve it with controlled experiments. No fitted quantity is renamed as a prediction, no uniqueness/ansatz is imported from the authors' prior work, and no step reduces by definition to its own input. Therefore no specific circular step can be exhibited; score 0.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

The central claim rests on a handful of literature-based domain assumptions (BPT labels, HMXB scaling, X-ray threshold) and standard statistical machinery, but introduces no new entities and fits no free parameters. The most fragile premise is that the 277-object subsample is representative of the original X-ray sample.

axioms (6)
  • domain assumption SDSS BPT-based classifications (Bolton et al. 2012) are the training labels and exhibit instance-dependent noise concentrated near the BPT demarcation line.
    Section 4.1 relies on this to explain why X-ray features are anti-informative; if labels were independent of BPT line ratios, the mechanism would not apply.
  • domain assumption The HMXB luminosity–SFR relation (Mineo et al. 2012; Lehmer et al. 2016) describes the SFG X-ray emission that creates the luminosity overlap.
    Section 3 uses this calibration to attribute the SFG tail above 10^42 erg/s to HMXBs; alternative models would weaken the overlap explanation.
  • domain assumption k-corrections are negligible for 0.2–12 keV luminosities at the redshifts of this sample.
    Section 3 states k-corrections are small and neglected; if they were significant, the quoted overlap fractions would shift.
  • domain assumption The 10^42 erg/s threshold is a literature-grounded benchmark for AGN dominance (Hickox & Alexander 2018), even though the paper argues it is not clean in this regime.
    Section 3 measures overlap against this threshold; the interpretation of the overlap percentages depends on the threshold's relevance.
  • standard math Random Forest with default scikit-learn hyperparameters and 5-fold CV gives stable accuracy estimates.
    Section 2 uses default hyperparameters and multiple random states; this is a standard empirical assumption.
  • standard math Kolmogorov-Smirnov and Mann-Whitney U tests on boundary distances are valid despite the small (N=17) and non-independent misclassified sample.
    Section 4.2 reports D=0.745, p=1.67e-9, but the predictions come from CV on the same data, which may inflate the significance.

pith-pipeline@v1.3.0-alltime-deepseek · 11000 in / 19795 out tokens · 182760 ms · 2026-08-02T05:48:08.427826+00:00 · methodology

0 comments
read the original abstract

In a previous paper, we found that adding an X-ray flux feature to a Random Forest classifier of active galactic nuclei (AGN) and star-forming galaxies (SFGs) coincided with a decrease in classification accuracy from 97.51% to 89.26%, a counterintuitive result given the prevailing theory that X-rays are a reliable AGN diagnostic. This paper investigates the source of that discrepancy through both astrophysical and machine learning lenses. On the astrophysical side, we show that the X-ray luminosities of AGN and SFGs in the sample substantially overlap across the canonical $10^{42}$ erg/s threshold, reflecting the moderate luminosity characteristic of an optically-selected SDSS sample and the contribution of high-mass X-ray binaries (HMXBs) to SFG emission. On the machine learning side, we argue that the BPT-derived training labels constitute instance-dependent label noise: label uncertainty is concentrated near the BPT demarcation line, where X-ray data would be most useful as a discriminator. We show that misclassified objects in 5-fold cross-validation cluster primarily near the Kewley demarcation curve, with a median distance of 0.123 dex compared to 0.743 dex for correctly classified objects (Kolmogorov-Smirnov $D = 0.745,\; p = 1.67\times 10^{-9}$). A controlled cross-validation decomposition shows that the previously reported decrease reflects predominantly sample selection rather than the X-ray feature, which exhibits a small but significantly negative permutation importance: the model makes limited, net-detrimental use of it, though its effect on overall accuracy is negligible. We conclude that future classifiers operating on optically-selected samples should employ independent label sources or noise-robust training methods.

Figures

Figures reproduced from arXiv: 2607.13233 by Jaymin Ding.

Figure 1
Figure 1. Figure 1: The distribution of the X-ray luminosities of the galaxies in the dataset. The top panel shows the luminosity histogram for both classes, with significant overlap between the two across the 1042 erg/s threshold. The bottom panel shows luminosity as a function of redshift, illustrating that both classes track the XMM flux limit, where objects at higher redshifts are only detected at higher luminosities, ind… view at source ↗
Figure 2
Figure 2. Figure 2: A permutation importance plot of the features, evaluated on a held-out test set of the 277-object X-ray subsample over 100 permutations. Error bars indicate one standard deviation across permutations. The X-ray flux feature shows a small but significantly negative importance, with its confidence interval lying entirely below zero, indicating that the model’s limited use of the feature slightly degrades its… view at source ↗
Figure 4
Figure 4. Figure 4: Normalized distribution of signed distances from the Kewley et al. (2001) demarcation line for correctly classified (𝑁 = 260) and misclassified (𝑁 = 17) objects in the 277-object X-ray subsample, evaluated via 5-fold cross-validation. Distance is the vertical offset in log ([OIII]/H𝛽) at fixed log ([NII]/H𝛼), positive on the AGN side. Misclassified objects concentrate near zero while correctly classified o… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 1 canonical work pages

  1. [1]

    , keywords =

    Classification parameters for the emission-line spectra of extragalactic objects. , keywords =. doi:10.1086/130766 , adsurl =

    doi:10.1086/130766

  2. [2]

    , keywords =

    Spectral Classification and Redshift Measurement for the SDSS-III Baryon Oscillation Spectroscopic Survey. , keywords =. doi:10.1088/0004-6256/144/5/144 , archivePrefix =. 1207.7326 , primaryClass =

    Pith/arXiv arXiv

  3. [3]

    , keywords =

    Multi-wavelength Classification of Active and Star-forming Galaxies on the BPT Diagram with Supervised Machine Learning Models. , keywords =. doi:10.1088/1538-3873/ad9b4e , adsurl =

    work page doi:10.1088/1538-3873/ad9b4e

  4. [4]

    , keywords =

    A Two-Phase Model for the X-Ray Emission from Seyfert Galaxies. , keywords =. doi:10.1086/186171 , adsurl =

    doi:10.1086/186171

  5. [5]

    , keywords =

    Obscured Active Galactic Nuclei. , keywords =. doi:10.1146/annurev-astro-081817-051803 , archivePrefix =. 1806.04680 , primaryClass =

    Pith/arXiv arXiv

  6. [6]

    , keywords =

    The Cosmic BPT Diagram: Confronting Theory with Observations. , keywords =. doi:10.1088/2041-8205/774/1/L10 , archivePrefix =. 1307.0514 , primaryClass =

    Pith/arXiv arXiv 2041

  7. [7]

    High-mass X-ray binaries

    X-ray emission from star-forming galaxies - I. High-mass X-ray binaries. , keywords =. doi:10.1111/j.1365-2966.2011.19862.x , archivePrefix =. 1105.4610 , primaryClass =

    arXiv 2011

  8. [8]

    , year = 1984, month = jan, volume =

    Black Hole Models for Active Galactic Nuclei. , year = 1984, month = jan, volume =. doi:10.1146/annurev.aa.22.090184.002351 , adsurl =

    arXiv 1984

  9. [9]

    arXiv e-prints , keywords =

    Learning from Noisy Labels with Deep Neural Networks: A Survey. arXiv e-prints , keywords =. doi:10.48550/arXiv.2007.08199 , archivePrefix =. 2007.08199 , primaryClass =

    Pith/arXiv arXiv doi:10.48550/arxiv.2007.08199 2007

  10. [10]

    , keywords =

    Theoretical Modeling of Starburst Galaxies. , keywords =. doi:10.1086/321545 , archivePrefix =. astro-ph/0106324 , primaryClass =

    Pith/arXiv arXiv

  11. [11]

    TOPCAT: Tool for OPerations on Catalogues And Tables

  12. [12]

    Journal of Machine Learning Research , keywords =

    Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research , keywords =. doi:10.48550/arXiv.1201.0490 , archivePrefix =. 1201.0490 , primaryClass =

    Pith/arXiv arXiv doi:10.48550/arxiv.1201.0490

  13. [13]

    Machine Learning , keywords =

    Random Forests. Machine Learning , keywords =. doi:10.1023/A:1010933404324 , adsurl =

    doi:10.1023/a:1010933404324

  14. [14]

    , keywords =

    The host galaxies of active galactic nuclei. , keywords =. doi:10.1111/j.1365-2966.2003.07154.x , archivePrefix =. astro-ph/0304239 , primaryClass =

    arXiv 2003

  15. [15]

    , keywords =

    Crossing the Line: Active Galactic Nuclei in the Star-forming Region of the BPT Diagram. , keywords =. doi:10.3847/1538-4357/ab1094 , archivePrefix =. 1811.11755 , primaryClass =

    Pith/arXiv arXiv

  16. [16]

    , keywords =

    The Evolution of Normal Galaxy X-Ray Emission through Cosmic History: Constraints from the 6 MS Chandra Deep Field-South. , keywords =. doi:10.3847/0004-637X/825/1/7 , archivePrefix =. 1604.06461 , primaryClass =

    Pith/arXiv arXiv

  17. [17]

    , keywords =

    Enhanced X-ray emission from Lyman break analogues and a possible L _ X -SFR-metallicity plane. , keywords =. doi:10.1093/mnras/stw284 , archivePrefix =. 1602.01091 , primaryClass =

    Pith/arXiv arXiv

  18. [18]

    , keywords =

    Elusive AGN in the XMM-Newton bright serendipitous survey. , keywords =. doi:10.1051/0004-6361:20077732 , archivePrefix =. 0705.2985 , primaryClass =

    Pith/arXiv arXiv

  19. [19]

    , keywords =

    Why X-Ray-selected Active Galactic Nuclei Appear Optically Dull. , keywords =. doi:10.1086/504067 , archivePrefix =. astro-ph/0603313 , primaryClass =

    Pith/arXiv arXiv

  20. [20]

    , keywords =

    Accretion Rate and the Physical Nature of Unobscured Active Galaxies. , keywords =. doi:10.1088/0004-637X/733/1/60 , archivePrefix =. 1103.0276 , primaryClass =

    Pith/arXiv arXiv

  21. [21]

    A new sample of X-ray selected narrow emission-line galaxies. I. The nature of optically elusive AGN. , keywords =. doi:10.1051/0004-6361/201423866 , archivePrefix =. 1407.6193 , primaryClass =

    Pith/arXiv arXiv

  22. [22]

    A new sample of X-ray selected narrow emission-line galaxies. II. Looking for True Seyfert 2. , keywords =. doi:10.1051/0004-6361/201629194 , archivePrefix =. 1608.01134 , primaryClass =

    Pith/arXiv arXiv

  23. [23]

    , keywords =

    On the X-ray, optical emission line and black hole mass properties of local Seyfert galaxies. , keywords =. doi:10.1051/0004-6361:20064894 , archivePrefix =. astro-ph/0605236 , primaryClass =

    Pith/arXiv arXiv

  24. [24]

    , keywords =

    Elusive active galactic nuclei. , keywords =. doi:10.1046/j.1365-8711.2003.07036.x , archivePrefix =. astro-ph/0307380 , primaryClass =

    arXiv 2003

  25. [25]

    , keywords =

    A New Physical Picture for Active Galactic Nuclei Lacking Optical Emission Lines. , keywords =. doi:10.3847/1538-4357/acac99 , archivePrefix =. 2211.07679 , primaryClass =

    Pith/arXiv arXiv

  26. [26]

    The OPTX Project. IV. How Reliable is [O III] as a Measure of AGN Activity?. , keywords =. doi:10.1088/0004-637X/722/1/212 , archivePrefix =. 1008.1582 , primaryClass =

    Pith/arXiv arXiv

  27. [27]

    , keywords =

    Understanding Galaxy Evolution Through Emission Lines. , keywords =. doi:10.1146/annurev-astro-081817-051832 , archivePrefix =. 1910.09730 , primaryClass =

    Pith/arXiv arXiv 1910

  28. [28]

    , keywords =

    Physical Drivers of Emission-line Diversity of SDSS Seyfert 2s and LINERs after Removal of Contributions from Star Formation. , keywords =. doi:10.3847/1538-4357/ac1e8d , archivePrefix =. 2108.07812 , primaryClass =

    Pith/arXiv arXiv

  29. [29]

    , keywords =

    From Active Stars to Black Holes: A Discovery Tool for Galactic X-Ray Sources. , keywords =. doi:10.1088/1538-3873/ad357c , archivePrefix =. 2401.09537 , primaryClass =

    Pith/arXiv arXiv

  30. [30]

    , keywords =

    X-ray detected AGN in SDSS dwarf galaxies. , keywords =. doi:10.1093/mnras/staa040 , archivePrefix =. 2001.03135 , primaryClass =

    Pith/arXiv arXiv 2001

  31. [31]

    , keywords =

    The incidence of X-ray selected AGN in nearby galaxies. , keywords =. doi:10.1093/mnras/stab3573 , archivePrefix =. 2112.03142 , primaryClass =

    Pith/arXiv arXiv

  32. [32]

    , keywords =

    WHaD diagram: Classifying the ionizing source with one single emission line. , keywords =. doi:10.1051/0004-6361/202347711 , archivePrefix =. 2311.10573 , primaryClass =

    Pith/arXiv arXiv