REVIEW 3 major objections 5 minor 21 references
Compact equations match black-box models in star-galaxy-QSO separation
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 20:03 UTC pith:QO677HJH
load-bearing objection A useful SR benchmark on SDSS DR17 with a credible MvSR result, but the headline parity claim rests on mismatched evaluation protocols and needs a matched-protocol reanalysis before it can be accepted. the 3 major comments →
Comparison of symbolic regression algorithms in Star/galaxy/quasar separation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Under a unified five-fold cross-validation protocol on 100,000 spectroscopically confirmed SDSS DR17 objects, the authors find that a ten-node analytic function of redshift—s(z)=3.134−2.320/(z+e^(−2.904z))—classifies galaxies, stars, and quasars with Cohen's kappa (chance-corrected agreement) 0.8948±0.0030 and accuracy 94.11%, statistically indistinguishable from the optimized Random Forest (κ=0.8953) and Support Vector Machine (κ=0.8954) baselines. All optimized models, symbolic and black-box, converge near κ≈0.90; the authors interpret this as an information-theoretic ceiling imposed by intrinsic physical degeneracies in the redshift distribution, particularly galaxy-quasar overlap near z≈
What carries the argument
The central object is the one-dimensional decision manifold: a scalar scoring function s(z) built from spectroscopic redshift z, together with two calibrated thresholds t1<t2, such that s(z)<t1 labels a galaxy, t1≤s(z)≤t2 labels a star, and s(z)>t2 labels a quasar. The paper evaluates four symbolic regression algorithms that search for compact expressions (complexity ≤10) for s(z) under mean-squared error against an ordinal class encoding, then optimize the thresholds by grid search over Cohen's kappa. The best expression, found by the multi-view symbolic regression variant, takes the rational-exponential form s(z)=3.134−2.320/(z+e^(−2.904z)), with thresholds t1=0.776±0.002 and t2=0.830±0.00
Load-bearing premise
The parity conclusion rests on comparing symbolic regressions evaluated by five-fold cross-validation on the full 100,000-object set with black-box baselines trained on an 8,000-sample subset and tested on a 2,000-sample holdout; if those evaluation protocols are not actually matched, 'statistically indistinguishable' is not supported.
What would settle it
Retrain the random forest and support vector machine baselines under the exact five-fold cross-validation protocol on the full 100,000 objects, and also evaluate the symbolic expressions on the 8,000/2,000 holdout; if the black-box models then outperform the symbolic expressions by more than the reported error bars, the parity claim fails. A second check is to recompute the thresholds and kappa after permuting the ordinal class encoding (e.g., galaxy=0, star=2, quasar=1); if the reported threshold stability disappears, the decision manifold is an artifact of the encoding.
If this is right
- If the parity claim holds, spectroscopic classification pipelines can replace stored black-box models with a single analytic formula and two numbers, reducing memory and compute requirements.
- The convergence of all models at κ≈0.90 implies that adding more model capacity will not improve accuracy on this one-dimensional feature; improvement must come from additional features (e.g., colors) rather than more complex decision functions.
- The threshold stability reported for the reinforcement-learning-discovered expression (σ<0.002) suggests that some symbolic search strategies yield decision boundaries robust to calibration noise, which could make them more reliable in automated survey pipelines.
- The specific failure region around z≈1.25, identified across all methods, gives a concrete target for follow-up with multi-band photometry or spectral indices.
Where Pith is reading between the lines
- A natural extension is to feed the discovered symbolic scores as a feature into a higher-dimensional classifier; because the equation saturates the one-dimensional ceiling, any gain would isolate information carried by color features.
- The claim of physical degeneracy is inferred from the plateau, but the paper does not compute a Bayes error rate from the full multi-dimensional data; estimating that rate independently would test whether the plateau truly reflects intrinsic class overlap.
- The ordinal encoding (galaxy=0, star=1, quasar=2) is one of many possible scalings; re-checking threshold stability under different monotonic encodings would show whether the fitted constants depend on the chosen scale.
- The authors caution that the expressions are empirical and carry no physical significance; a patient reader might still probe whether the e-fold scale in the denominator corresponds to any characteristic quasar spectral feature, but the paper itself disavows that interpretation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares four symbolic regression (SR) algorithms — PySR, ESR, PhySO, and MvSR — for the three-class star/galaxy/quasar classification of 100,000 SDSS DR17 objects using only the spectroscopic redshift z. For each method, a compact expression of complexity ≤10 is discovered on a 10,000-object subset, then two thresholds (t1, t2) are calibrated per fold by maximizing Cohen's κ under a 5-fold stratified cross-validation protocol on the full sample. MvSR is reported as the best SR method (κ = 0.8948 ± 0.0030, accuracy 94.11%), with Random Forest, SVM, and MLP baselines reaching κ ≈ 0.895. The paper concludes that low-complexity symbolic expressions can match black-box models and that the residual error reflects an intrinsic physical degeneracy / Bayes error in the redshift distribution.
Significance. If the parity claim survives a matched evaluation, the paper provides a useful, reproducible demonstration that interpretable low-complexity formulae can rival tuned classifiers on a standard astronomical benchmark. The explicit caveat that the equations are empirical, the unified threshold-optimization protocol for the SR methods, and the reporting of fold-to-fold standard deviations are strengths. The comparison against three ML families is a reasonable control. However, the scientific significance depends on two points that the manuscript does not yet establish: the ML baselines must be evaluated under the same 100k 5-fold CV protocol as the SR models, and the interpretation of the κ≈0.90 ceiling as a fundamental Bayes error must be justified rather than asserted.
major comments (3)
- [§3.4.4 vs §3.2, Table 4] The ML baselines are evaluated on a single stratified holdout from a 10,000-object subset (8,000 train / 2,000 test, Section 3.4.4), whereas all SR results are 5-fold CV on 100,000 with thresholds tuned per fold. Table 4 reports 'CVκ(Discovery Phase)' and 'CVκ(Global Test Set)' without explaining how a holdout yields CV quantities or what the global test set is. The Section 5.1 claim that MvSR's κ=0.8948±0.0030 is 'statistically indistinguishable' from RF/SVM is therefore not supported by a matched comparison. Please rerun baselines under the identical 100k 5-fold CV protocol, or report a matched holdout for the SR models, and adjust the parity claim accordingly.
- [Abstract vs §4.1.4, Table 3] The abstract describes 'an 80,000-sample 5-fold cross-validation threshold optimization phase and a subsequent 10,000-sample unseen hold-out test set' and reports MvSR κ=0.8956 (0.8876 on holdout). Section 4.1.4 and Table 3 report κ=0.8948±0.0030 via 5-fold CV on 100,000. These are different protocols and different numbers; the holdout value 0.8876 does not appear in Table 3 or Table 4. The headline result is thus ambiguous. Please align the abstract with the actual analysis and tabulate protocol-specific metrics explicitly.
- [§5.2, §4.2.1] The claim that all models converge at κ≈0.90 and that 'the remaining classification error is rooted in intrinsic physical degeneracies ... fundamental Bayes error rate' is an unsupported interpretation. Convergence of models trained on the same single feature z only shows that these model classes saturate the information contained in z under their training criteria; it does not establish a Bayes error of the redshift distribution. No estimate of the Bayes error, no information-theoretic bound, and no experiment with additional features are provided. Please qualify or remove this claim to avoid overstating the astrophysical conclusion.
minor comments (5)
- [§4.1.2, Fig. 16 caption] Caption reads 'The curve for recoil as a function of redshift' — should be 'recall'.
- [§3.5.1] The ordinal target mapping is never specified (e.g., STAR=0, GALAXY=1, QSO=2). Please state the exact encoding used for the regression targets; otherwise the numeric constants in Eqs. 34–40 cannot be reproduced or interpreted.
- [Table 3] Accuracy and balanced accuracy are reported in inconsistent units: text says '0.9411±0.0017' while the column header says 'Accuracy' without a % sign; the text later says '94.11±0.0017%'. Clarify whether values are fractions or percentages and use consistent notation.
- [Eq. (7)] The selection score in Eq. (7) uses Accuracy and κ, but it is not stated whether these are computed on the validation fold after threshold optimization or on the training fold. Please specify the exact evaluation set for model selection.
- [Fig. 7 caption] The caption refers to 'Φ-SO' while the method is called PhySO in the text and Table 1. Align the notation.
Circularity Check
Main SR-vs-baseline benchmark is held-out and non-circular; minor circularity in the 'information-theoretic ceiling' interpretation.
specific steps
-
self definitional
[Section 5.2 (and Section 4.2.1, Table 4)]
"The convergence of all optimized models at a performance ceiling of κ≈ 0.90 indicates that the remaining classification error is rooted in intrinsic physical degeneracies of the SDSS DR17 redshift distribution. Specifically, the overlap in spectroscopic signatures between galaxies and quasars near z≈ 1.25 represents a fundamental Bayes error rate within a one-dimensional feature space."
The 'ceiling' is not independently derived; it is the empirical CV κ of the same baselines (Table 4: 0.8953–0.8954). The paper then labels this measured value an 'information-theoretic ceiling' and a 'fundamental Bayes error rate', and uses it to conclude that the residual error is 'intrinsic'. No external or first-principles estimate of the Bayes error is given, so the explanation reduces to the observed convergence: the models' own performance defines the ceiling and is then said to reach it. This is a self-referential interpretation, though it is secondary to the main held-out benchmark.
full rationale
The central derivation is not circular. Symbolic expressions are fit on a 10,000-object discovery subset, decision thresholds are grid-searched per fold on 80,000 training samples, and κ/accuracy are computed on held-out 20,000-sample folds. The ML baselines are independent black-box models, so the reported parity is an empirical comparison rather than a fit renamed as a prediction; the mismatched evaluation protocol (8k/2k holdout vs full-data 5-fold CV) is a validity concern, not a circularity. The paper explicitly states the equations are empirical and no physical significance should be ascribed to them. Self-citations (Bethapudi & Desai 2018; Krishak & Desai 2020; Desai et al. 2012) are standard references and not load-bearing. The only circular-adjacent step is the Section 5.2 claim of an 'intrinsic' information-theoretic ceiling, which is defined by the models' own converged performance and then used to explain that same performance. Because this interpretive step is secondary and does not drive the main predictive-parity claim, the overall circularity score is 2.
Axiom & Free-Parameter Ledger
free parameters (6)
- PySR expression constants =
0.8395, 78.28, 0.1141
- ESR expression constants =
0.7, 11.88
- PhySO expression constants =
2.4027, 0.5786
- MvSR expression constants =
3.134, 2.320, 2.904
- Decision thresholds t1, t2 per method =
e.g., PySR: 0.796±0.017, 0.981±0.011; MvSR: 0.776±0.002, 0.830±0.003
- Selection weights in Eq. 7 =
0.4, 0.4, 0.2
axioms (6)
- domain assumption SDSS DR17 spectroscopic labels (GALAXY/STAR/QSO) are correct and complete.
- domain assumption An ordinal scalar encoding of the three classes is a valid regression target for the symbolic engines.
- domain assumption A single scalar function s(z) with two thresholds can represent the optimal decision boundary along the redshift axis.
- domain assumption The 10,000-sample subset used for symbolic discovery is representative of the full 100,000-object dataset.
- standard math BIC/MDL as implemented in Eq. 12 provides valid model selection for this symbolic regression task.
- ad hoc to paper Convergence of all tested models near κ≈0.90 implies a fundamental Bayes error rate for the SDSS DR17 dataset.
read the original abstract
This work investigates symbolic regression (SR) as an interpretable alternative to black-box machine learning for the classification of stars, galaxies, and quasars in the Sloan Digital Sky Survey Data Release 17 (SDSS DR17). We conduct a systematic comparative study of four state-of-the-art SR frameworks: {\tt PySR}, Exhaustive Symbolic Regression ({\tt ESR}) with MDL-based selection, Physical Symbolic Optimization ({\tt PhySO}) using deep reinforcement learning, and Multi-View Symbolic Regression ({\tt MvSR}). By deriving compact analytic functions (complexity $\leq$ 10) on a representative training subset and subsequently evaluating them via an 80,000-sample 5-fold cross-validation threshold optimization phase and a subsequent 10,000-sample unseen hold-out test set, we map spectroscopic redshift ($z$) to continuous classification scores. Our results demonstrate that these low-complexity expressions achieve high predictive reliability, with {\tt MvSR} reaching a cross-validation Cohen's Kappa of 0.8956 (0.8876 on the hold-out set) and {\tt PhySO} achieving exceptional parametric stability ($\sigma < 0.002$). We note however that the resulting equations returned by Symbolic regression are purely empirical and no physical significance should be ascribed to these equations.
Figures
Reference graph
Works this paper leans on
-
[7]
arXiv e-prints , arXiv:2305.015822305.01582
Interpretable Machine Learning for Sci- ence with PySR and SymbolicRegression.jl. arXiv e-prints , arXiv:2305.015822305.01582. Darc, P., Bom, C.R., Kilpatrick, C., Fraga, B.M.O., Teixeira, G.S.M.,
-
[8]
arXiv e-prints , arXiv:2511.087842511.08784
Symbolic Regression Is All You Need: From Simulations to Scaling Laws in Binary Neutron Star Mergers. arXiv e-prints , arXiv:2511.087842511.08784. Desai, S., Armstrong, R., Mohr, J.J., Semler, D.R., Liu, J., Bertin, E., Allam, S.S., Barkhouse, W.A., Bazin, G., Buckley- Geer, E.J., Cooper, M.C., Hansen, S.M., High, F.W., Lin, H., Lin, Y .T., Ngeow, C.C., R...
-
[10]
arXiv e- prints , arXiv:2507.130332507.13033
(Exhaustive) Symbolic Regression and model selection by minimum description length. arXiv e- prints , arXiv:2507.130332507.13033. Hassija, V ., Chamola, V ., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., Hus- sain, A.,
-
[13]
Unsupervised star, galaxy, QSO classification. Application of HDBSCAN. A&A 633, A154.1911.05107. 16 Martín, A., Yasin, T., Bartlett, D.J., Desmond, H., Ferreira, P.G.,
Pith/arXiv arXiv 1911
-
[15]
Learn- ing the relations between neutron star and nuclear matter properties with symbolic regression2601.07727. Richards, G.T., Fan, X., Newberg, H.J., Strauss, M.A., Van- den Berk, D.E., Schneider, D.P., Yanny, B., Boucher, A., Burles, S., Frieman, J.A., Gunn, J.E., Hall, P.B., Ivezi ´c, Ž., Kent, S., Loveday, J., Lupton, R.H., Rockosi, C.M., Schlegel, D...
-
[17]
arXiv e-prints , arXiv:2402.042982402.04298
Multi-View Symbolic Regression. arXiv e-prints , arXiv:2402.042982402.04298. Sevilla-Noarbe, I., Hoyle, B., Marchã, M.J., Soumagnac, M.T., Bechtol, K., Drlica-Wagner, A., Abdalla, F., Aleksi ´c, J., Avestruz, C., Balbinot, E., Banerji, M., Bertin, E., Bonnett, C., Brunner, R., Carrasco-Kind, M., Choi, A., Giannantonio, T., Kim, E., Lahav, O., Moraes, B., ...
-
[18]
MNRAS 481, 5451–5469.1805.02427
Star-galaxy classification in the Dark Energy Survey Y1 data set. MNRAS 481, 5451–5469.1805.02427. Slater, C.T., Ivezic, Z., Lupton, R.H.,
-
[19]
Morphological Star-Galaxy Separation. AJ 159, 65.1912.08210. Solorio-Ramírez, J.L., Jiménez-Cruz, R., Villuendas-Rey, Y ., Yáñez-Márquez, C.,
Pith/arXiv arXiv 1912
-
[21]
arXiv e-prints , arXiv:2502.105062502.10506
Evidence for dynamical dark energy from DESI-DR2 and SN data? A symbolic regression analysis. arXiv e-prints , arXiv:2502.105062502.10506. Tenachi, W., Ibata, R., Diakogiannis, F.I.,
- [27]
-
[35]
Bartlett, D.J., Desmond, H., Ferreira, P.G.,
2112.02026. Bartlett, D.J., Desmond, H., Ferreira, P.G.,
-
[2002]
Spectroscopic Target Selection in the Sloan Digi- tal Sky Survey: The Quasar Sample. AJ 123, 2945–2975. astro-ph/0202251. Russeil, E., Olivetti de França, F., Malanchev, K., Burlacu, B., Ishida, E.E.O., Leroux, M., Michelin, C., Moinard, G., Gangler, E.,
-
[2012]
The Blanco Cosmology Survey: Data Acquisition, Processing, Calibration, Quality Diagnostics, and Data Release. ApJ 757, 83.1204.1210. Desmond, H.,
-
[2015]
MN- RAS 450, 666–680.1306.5236
Star/galaxy separation at faint magni- tudes: application to a simulated Dark Energy Survey. MN- RAS 450, 666–680.1306.5236. Sousa-Neto, A., Bengaly, C., Gonzalez, J.E., Alcaniz, J.,
-
[2018]
Astronomy and Computing 23, 15.1704.04659
Separation of pulsar signals from noise using supervised machine learning algorithms. Astronomy and Computing 23, 15.1704.04659. Bhavanam, S.R., Channappayya, S.S., P. K, S., Desai, S.,
-
[2020]
Model comparison tests of mod- ified gravity from the Eöt-Wash experiment. J. Cosmology Astropart. Phys. 2020, 006.2003.10127. Liu, D.C., Nocedal, J.,
Pith/arXiv arXiv 2020
-
[2022]
arXiv e-prints , arXiv:2211.114612211.11461
Ex- haustive Symbolic Regression. arXiv e-prints , arXiv:2211.114612211.11461. Bartlett, D.J., Desmond, H., Ferreira, P.G., Kronberger, G.,
-
[2023]
MNRAS 518, 3123–3136.2211.08388
Photometric identification of compact galaxies, stars, and quasars using multiple neural networks. MNRAS 518, 3123–3136.2211.08388. Cohen, J.,
-
[2024]
Enhanced astronomical source classification with integration of attention mechanisms and vision transformers. Ap&SS 369, 92.2408.13634. Bishop, C.M.,
-
[2025]
arXiv e-prints , arXiv:2512.159202512.15920
Introduction to Symbolic Regression in the Physical Sciences. arXiv e-prints , arXiv:2512.159202512.15920. Bethapudi, S., Desai, S.,
-
[2026]
arXiv e-prints , arXiv:2601.052032601
Symbolically regressing dark matter halo profiles using weak lensing. arXiv e-prints , arXiv:2601.052032601. 05203. Patra, N.K., Malik, T., Zhou, K., Providência, C.,
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.