{"id":"2ee3cf77-acfd-4b43-9199-b4d0d7323cf3","arxiv_id":"2602.24022","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"On SDSS DR17 redshift data, a multi-view symbolic regression expression achieves Cohen's κ≈0.895, comparable to random forests, SVMs, and MLPs trained on the same single feature.","lead":"This paper compares four symbolic regression algorithms that derive short mathematical formulas to classify stars, galaxies, and quasars using only the redshift value from SDSS DR17. The best formula reaches a Cohen's Kappa of about 0.89, close to black-box machine learning classifiers, but the comparison has protocol inconsistencies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ML baselines are evaluated on a smaller, differently split protocol than the SR 100k 5-fold CV, so the 'statistically indistinguishable' parity claim is not yet supported.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the ML baselines and the symbolic-regression frameworks are evaluated under different protocols, so the numerical parity claim is not yet established. This mismatch is internal to the paper: Section 3.4.4 describes a single 8,000/2,000 holdout on a 10,000-object subset, while Section 4.2.1 and Table 4 report CV quantities without a clear reconciliation; the abstract introduces yet another holdout (10,000 objects, κ=0.8876) not present in the results tables. This is not a matter of astrophysical interpretation but of basic comparability. If the baselines were run on the same 100,000-object 5-fold CV folds as the SR models, the difference between MvSR (0.8948) and the top baseline (0.8954) could easily change by more than the quoted uncertainties, because the baseline error bars are computed on a 10,000-object subset and do not reflect the full-data sampling distribution. The ordinal encoding issue raised by the reader is real but secondary: threshold optimization makes the final discrete mapping somewhat invariant to monotone rescaling, and the chosen galaxy<star<qso ordering is physically motivated by the redshift distributions. The abstract/methods inconsistency is part of the same protocol-mismatch concern rather than a separate scientific flaw. Thus the correct verdict remains CONDITIONAL: the benchmark itself may be sound, but the parity claim needs a matched-protocol replication before it can be accepted. I see no reason to raise or lower the reader's confidence, only to emphasize that the required test is straightforward and should be decisive.","tokens_in":17503,"tokens_out":8156,"duration_ms":80617,"concrete_test":"Re-run the three ML baselines (RF, SVM, MLP) on the exact same five stratified folds used for the SR threshold evaluation over the full 100,000-object dataset, with X={z}, and report per-fold Cohen's κ (mean ± std) and paired differences against MvSR. Also either reproduce or explicitly remove the abstract's 10,000-object holdout (κ=0.8876) and state which number is the primary result. If the matched-CV baseline κ means exceed MvSR by more than roughly 0.005 (about one SR cross-fold σ), the parity claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central parity claim (Sec. 5.1, Table 4) says MvSR's κ=0.8948±0.0030 is statistically indistinguishable from RF (0.8953±0.008) and SVM (0.8954±0.009). But these baseline numbers come from a different evaluation protocol: Sec. 3.4.4 specifies a single stratified holdout on a 10,000-object subset (8,000 train / 2,000 test), while Sec. 4.2.1 refers to 'five-fold stratified cross-validation' and Table 4 reports 'CVκ(Discovery Phase)' and 'CVκ(Global Test Set)' without explaining how the holdout yields CV quantities or what the 'Global Test Set' is. The SR frameworks, in contrast, are evaluated by 5-fold CV on all 100,000 objects, with thresholds tuned per fold on 80,000 training samples. These are not matched protocols. The abstract compounds the problem by describing an 80,000-sample 5-fold CV phase plus a 10,000-sample unseen holdout and reporting MvSR κ=0.8876 on that holdout, a number that does not appear in Table 3 (0.8948±0.0030 on the 100k CV). A reader therefore cannot determine which evaluation actually supports the headline parity claim. If RF/SVM were evaluated on the same full-data folds, their κ could shift by more than the quoted sampling noise, and the 'statistically indistinguishable' conclusion could change. This is the load-bearing weakness: the parity claim rests on comparing apples (full-data CV SR) to oranges (subset-holdout baselines).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares four symbolic regression (SR) algorithms — PySR, ESR, PhySO, and MvSR — for the three-class star/galaxy/quasar classification of 100,000 SDSS DR17 objects using only the spectroscopic redshift z. For each method, a compact expression of complexity ≤10 is discovered on a 10,000-object subset, then two thresholds (t1, t2) are calibrated per fold by maximizing Cohen's κ under a 5-fold stratified cross-validation protocol on the full sample. MvSR is reported as the best SR method (κ = 0.8948 ± 0.0030, accuracy 94.11%), with Random Forest, SVM, and MLP baselines reaching κ ≈ 0.895. The paper concludes that low-complexity symbolic expressions can match black-box models and that the residual error reflects an intrinsic physical degeneracy / Bayes error in the redshift distribution.","tokens_in":17942,"tokens_out":6187,"duration_ms":56199,"significance":"If the parity claim survives a matched evaluation, the paper provides a useful, reproducible demonstration that interpretable low-complexity formulae can rival tuned classifiers on a standard astronomical benchmark. The explicit caveat that the equations are empirical, the unified threshold-optimization protocol for the SR methods, and the reporting of fold-to-fold standard deviations are strengths. The comparison against three ML families is a reasonable control. However, the scientific significance depends on two points that the manuscript does not yet establish: the ML baselines must be evaluated under the same 100k 5-fold CV protocol as the SR models, and the interpretation of the κ≈0.90 ceiling as a fundamental Bayes error must be justified rather than asserted.","major_comments":[{"comment":"The ML baselines are evaluated on a single stratified holdout from a 10,000-object subset (8,000 train / 2,000 test, Section 3.4.4), whereas all SR results are 5-fold CV on 100,000 with thresholds tuned per fold. Table 4 reports 'CVκ(Discovery Phase)' and 'CVκ(Global Test Set)' without explaining how a holdout yields CV quantities or what the global test set is. The Section 5.1 claim that MvSR's κ=0.8948±0.0030 is 'statistically indistinguishable' from RF/SVM is therefore not supported by a matched comparison. Please rerun baselines under the identical 100k 5-fold CV protocol, or report a matched holdout for the SR models, and adjust the parity claim accordingly.","section":"§3.4.4 vs §3.2, Table 4"},{"comment":"The abstract describes 'an 80,000-sample 5-fold cross-validation threshold optimization phase and a subsequent 10,000-sample unseen hold-out test set' and reports MvSR κ=0.8956 (0.8876 on holdout). Section 4.1.4 and Table 3 report κ=0.8948±0.0030 via 5-fold CV on 100,000. These are different protocols and different numbers; the holdout value 0.8876 does not appear in Table 3 or Table 4. The headline result is thus ambiguous. Please align the abstract with the actual analysis and tabulate protocol-specific metrics explicitly.","section":"Abstract vs §4.1.4, Table 3"},{"comment":"The claim that all models converge at κ≈0.90 and that 'the remaining classification error is rooted in intrinsic physical degeneracies ... fundamental Bayes error rate' is an unsupported interpretation. Convergence of models trained on the same single feature z only shows that these model classes saturate the information contained in z under their training criteria; it does not establish a Bayes error of the redshift distribution. No estimate of the Bayes error, no information-theoretic bound, and no experiment with additional features are provided. Please qualify or remove this claim to avoid overstating the astrophysical conclusion.","section":"§5.2, §4.2.1"}],"minor_comments":[{"comment":"Caption reads 'The curve for recoil as a function of redshift' — should be 'recall'.","section":"§4.1.2, Fig. 16 caption"},{"comment":"The ordinal target mapping is never specified (e.g., STAR=0, GALAXY=1, QSO=2). Please state the exact encoding used for the regression targets; otherwise the numeric constants in Eqs. 34–40 cannot be reproduced or interpreted.","section":"§3.5.1"},{"comment":"Accuracy and balanced accuracy are reported in inconsistent units: text says '0.9411±0.0017' while the column header says 'Accuracy' without a % sign; the text later says '94.11±0.0017%'. Clarify whether values are fractions or percentages and use consistent notation.","section":"Table 3"},{"comment":"The selection score in Eq. (7) uses Accuracy and κ, but it is not stated whether these are computed on the validation fold after threshold optimization or on the training fold. Please specify the exact evaluation set for model selection.","section":"Eq. (7)"},{"comment":"The caption refers to 'Φ-SO' while the method is called PhySO in the text and Table 1. Align the notation.","section":"Fig. 7 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's astroinformatics scope, and the SR evaluation is reasonably careful. However, the central parity claim currently rests on a mismatched baseline protocol, and the abstract contains internally inconsistent evaluation descriptions. These are fixable, but they require substantive re-analysis or re-reporting; hence major revision rather than acceptance. I saw no evidence of misconduct or duplicate publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful benchmark—four symbolic regression packages applied to the same SDSS DR17 star/galaxy/quasar task, with compact expressions and a rare honesty disclaimer that the equations are empirical. The best result (MvSR, κ≈0.895 on 5-fold CV over 100k objects) is credible and clearly beats the earlier PySR-only result. The main problem is that the headline parity claim with ML baselines rests on an apples-to-oranges comparison.\n\nWhat is actually new: the extension of Llorella & Cebrian to ESR, PhySO, and MvSR; a clean 5-fold CV protocol for the SR phase; threshold-stability analysis; and the observation that a simple 1D score can get close to black-box classifiers. The paper is careful to say the expressions are empirical and does not dress them up as physical laws. That is worth crediting.\n\nThe soft spots, in proportion:\n\n1. The parity claim (Sec 5.1, Table 4) is not supported as written. The SR numbers come from 5-fold CV on all 100k objects, with thresholds tuned per fold. The ML baselines were trained on an 8k subset and tested on a 2k holdout (Sec 3.4.4). Table 4 reports \"CVκ(Discovery Phase)\" and \"CVκ(Global Test Set)\" but never explains how those are derived from a single holdout, and the abstract reports a different MvSR holdout number (0.8876) that does not appear in Table 3. With different evaluation protocols, the error bars cannot be compared. This is fixable: run RF/SVM/MLP under the same 5-fold CV on 100k and report the numbers. Until then, \"statistically indistinguishable\" is an overstatement.\n\n2. The \"information-theoretic ceiling / intrinsic physical degeneracy\" conclusion (Sec 5.2) overreaches. All models use only redshift; the observed κ ceiling is a statement about the one-dimensional feature space, not about physical degeneracies of the SDSS sample more broadly. The authors implicitly acknowledge this limitation, but the wording in Sec 5.2 goes beyond the evidence.\n\n3. Minor: the ordinal encoding of three classes into scalar targets is a modeling choice. Since the thresholds are refit per fold and reported stable, this is not a fatal flaw, but the numerical constants in Eqs. 34–40 are scale-dependent, so no physical meaning should be read into them (the paper mostly respects this).\n\nThe citation pattern looks appropriate; the prior PySR-only work is clearly identified. No code/data package is mentioned, which is a minor omission for a benchmark paper.\n\nBottom line: the benchmark itself is solid and worth engaging with, but the parity claim and the \"physical degeneracy\" conclusion need to be reined in. This deserves a serious referee; I would send it to review, with the clear request that the baselines be re-run under the same protocol.","headline":"A useful SR benchmark on SDSS DR17 with a credible MvSR result, but the headline parity claim rests on mismatched evaluation protocols and needs a matched-protocol reanalysis before it can be accepted.","tokens_in":18431,"tokens_out":2201,"would_cite":true,"duration_ms":18132,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Compact equations match black-box models in star-galaxy-QSO separation","keywords":["symbolic regression","star-galaxy-quasar classification","SDSS DR17","interpretable machine learning","Cohen's kappa","redshift classification","multi-view symbolic regression","decision threshold optimization"],"falsifier":"Retrain the random forest and support vector machine baselines under the exact five-fold cross-validation protocol on the full 100,000 objects, and also evaluate the symbolic expressions on the 8,000/2,000 holdout; if the black-box models then outperform the symbolic expressions by more than the reported error bars, the parity claim fails. A second check is to recompute the thresholds and kappa after permuting the ordinal class encoding (e.g., galaxy=0, star=2, quasar=1); if the reported threshold stability disappears, the decision manifold is an artifact of the encoding.","tokens_in":17379,"feed_emoji":"🔭","tokens_out":8894,"duration_ms":64528,"temperature":0.7,"pith_summary":"This paper claims that simple, human-readable mathematical functions of a single quantity—spectroscopic redshift—can classify stars, galaxies, and quasars almost as well as optimized black-box machine learning models. Using four symbolic regression systems, the authors search for compact expressions (at most ten nodes) that map redshift to a score, then cut the score with two thresholds to assign the three classes. The best expression reaches a chance-corrected agreement score, Cohen's kappa, of about 0.895 and 94% accuracy, statistically indistinguishable from the random forest and support vector machine benchmarks. The paper argues this shows transparent equations can replace opaque models for this task, and that the shared ~0.90 performance ceiling reflects an actual physical overlap between galaxy and quasar spectra near redshift 1.25, not a limitation of any particular model.","feed_headline":"Compact equations match black-box models in star-galaxy-QSO separation","feed_subtitle":"Best symbolic formula reaches 94% accuracy, matching optimized random forest and SVM baselines.","key_machinery":"The central object is the one-dimensional decision manifold: a scalar scoring function s(z) built from spectroscopic redshift z, together with two calibrated thresholds t1<t2, such that s(z)<t1 labels a galaxy, t1≤s(z)≤t2 labels a star, and s(z)>t2 labels a quasar. The paper evaluates four symbolic regression algorithms that search for compact expressions (complexity ≤10) for s(z) under mean-squared error against an ordinal class encoding, then optimize the thresholds by grid search over Cohen's kappa. The best expression, found by the multi-view symbolic regression variant, takes the rational-exponential form s(z)=3.134−2.320/(z+e^(−2.904z)), with thresholds t1=0.776±0.002 and t2=0.830±0.00","core_discovery":"Under a unified five-fold cross-validation protocol on 100,000 spectroscopically confirmed SDSS DR17 objects, the authors find that a ten-node analytic function of redshift—s(z)=3.134−2.320/(z+e^(−2.904z))—classifies galaxies, stars, and quasars with Cohen's kappa (chance-corrected agreement) 0.8948±0.0030 and accuracy 94.11%, statistically indistinguishable from the optimized Random Forest (κ=0.8953) and Support Vector Machine (κ=0.8954) baselines. All optimized models, symbolic and black-box, converge near κ≈0.90; the authors interpret this as an information-theoretic ceiling imposed by intrinsic physical degeneracies in the redshift distribution, particularly galaxy-quasar overlap near z≈","pith_inferences":["A natural extension is to feed the discovered symbolic scores as a feature into a higher-dimensional classifier; because the equation saturates the one-dimensional ceiling, any gain would isolate information carried by color features.","The claim of physical degeneracy is inferred from the plateau, but the paper does not compute a Bayes error rate from the full multi-dimensional data; estimating that rate independently would test whether the plateau truly reflects intrinsic class overlap.","The ordinal encoding (galaxy=0, star=1, quasar=2) is one of many possible scalings; re-checking threshold stability under different monotonic encodings would show whether the fitted constants depend on the chosen scale.","The authors caution that the expressions are empirical and carry no physical significance; a patient reader might still probe whether the e-fold scale in the denominator corresponds to any characteristic quasar spectral feature, but the paper itself disavows that interpretation."],"forward_implications":["If the parity claim holds, spectroscopic classification pipelines can replace stored black-box models with a single analytic formula and two numbers, reducing memory and compute requirements.","The convergence of all models at κ≈0.90 implies that adding more model capacity will not improve accuracy on this one-dimensional feature; improvement must come from additional features (e.g., colors) rather than more complex decision functions.","The threshold stability reported for the reinforcement-learning-discovered expression (σ<0.002) suggests that some symbolic search strategies yield decision boundaries robust to calibration noise, which could make them more reliable in automated survey pipelines.","The specific failure region around z≈1.25, identified across all methods, gives a concrete target for follow-up with multi-band photometry or spectral indices."],"fun_headline_variants":["Tiny equation rivals black-box models on SDSS data","Redshift-only analytic function hits 94% accuracy, ties ML","A 10-node formula ties AI in celestial classification","Simple formula matches AI in star-galaxy-QSO classification"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The parity conclusion rests on comparing symbolic regressions evaluated by five-fold cross-validation on the full 100,000-object set with black-box baselines trained on an 8,000-sample subset and tested on a 2,000-sample holdout; if those evaluation protocols are not actually matched, 'statistically indistinguishable' is not supported.","fun_headline_variants_meta":{"raw":{"variants":["Tiny equation rivals black-box models on SDSS data","Redshift-only analytic function hits 94% accuracy, ties ML","A 10-node formula ties AI in celestial classification","Simple formula matches AI in star-galaxy-QSO classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00092,"raw_usage":{"total_tokens":3809,"prompt_tokens":795,"completion_tokens":3014,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":2946}},"tokens_in":539,"tokens_out":3014,"duration_ms":19851,"temperature":1.0,"reasoning_tokens":2946,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T20:03:21.648525+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the random forest and support vector machine baselines under the exact five-fold cross-validation protocol on the full 100,000 objects, and also evaluate the symbolic expressions on the 8,000/2,000 holdout; if the black-box models then outperform the symbolic expressions by more than the reported error bars, the parity claim fails. A second check is to recompute the thresholds and kappa after permuting the ordinal class encoding (e.g., galaxy=0, star=2, quasar=1); if the reported threshold stability disappears, the decision manifold is an artifact of the encoding.","supporting_citations":[],"review_version":1}