REVIEW 3 major objections 6 minor 21 references
A machine-learning pipeline that reads photometric-redshift probability distributions yields better point estimates and reliability scores than the catalog defaults across six HSC pipeline–layer combinations.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-30 20:04 UTC pith:535CEJJQ
load-bearing objection Practical, well-executed ML stack that consistently beats HSC catalog point estimates and quality flags on six PDZ sets, with public code and a VAC; main caveat is labeled-spec generalization. the 3 major comments →
Optimizing the extraction of information from redshift probability distribution functions
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Across all six HSC-SSP PDR3 pipeline–layer combinations, an optimized point estimate z_ml extracted from each galaxy’s redshift probability distribution improves σ_NMAD and η_0.15 relative to the catalog z_best, and a data-driven reliability score r_ml, derived from a calibrated uncertainty σ_ml, filters galaxies more efficiently than photoz_risk_best and photoz_conf_best as measured by the area under the quality-versus-retained-fraction curves. For the Mizuki template-fitting pipeline the catalog indicators fail dramatically (AUC up to about ten times larger), while r_ml still identifies unreliable objects across redshift regimes.
What carries the argument
turboPDZ: each PDZ is PCA-compressed and joined to magnitude and summary descriptors; one multilayer perceptron, tuned under a composite σ_NMAD–η_0.15 objective, outputs z_ml; a second network trained in log-error space and post-hoc calibrated outputs σ_ml, which is turned into the percentile-rank reliability score r_ml.
Load-bearing premise
Models trained on the spectroscopic cross-match must still rank and predict well on the full photometric catalog and on science samples whose magnitude and redshift mix differ from the labeled set.
What would settle it
On a fully held-out spectroscopic sample with no overlap to any training or calibration set used by either turboPDZ or the original HSC pipelines, check whether z_ml still beats z_best in σ_NMAD and η_0.15 and whether r_ml still yields smaller AUC under the quality-versus-retained-fraction curves than the catalog risk and confidence flags.
If this is right
- Gold photometric samples for weak lensing and clustering can keep more galaxies at fixed scatter and outlier rate by cutting on r_ml instead of catalog risk or confidence.
- Template-fitting PDZs that are sharply peaked at the wrong redshift can still be flagged as unreliable by r_ml even when analytic risk and confidence fail.
- The same PCA-plus-descriptor architecture can be retuned for other surveys’ PDZs without redesigning the pipelines that produced those PDZs.
- Value-added z_ml, σ_ml, and r_ml columns for the full HSC-SSP PDR3 catalog become available for immediate downstream use.
Where Pith is reading between the lines
- Because point estimation leans on location statistics while reliability leans on peak morphology, joint multi-task training of z and σ might trade a little redshift accuracy for still-stronger ranking of bad objects.
- The method’s dependence on a representative spectroscopic anchor implies that deep, sparse fields (or high-z tails) will need explicit domain-adaptation checks before r_ml cuts are trusted at the same thresholds as in the training domain.
- Surveys that already ship full PDZs but weak quality flags (including Stage-IV optical and near-IR catalogs) are the natural next stress tests of whether the Mizuki-style rescue generalizes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents turboPDZ, a survey-independent ML pipeline that extracts an optimized photometric-redshift point estimate z_ml and a reliability score r_ml directly from photo-z PDZs. PDZs from the three HSC-SSP PDR3 pipelines (DEmP, DNNz, Mizuki) in Wide and DUD layers are PCA-compressed, augmented with hand-crafted descriptors, and fed to Optuna-tuned MLPs. z_ml is trained under a composite σ_NMAD/η_0.15 objective; σ_ml is trained in log-space with post-hoc calibration and converted to r_ml by percentile ranking. On held-out spectroscopic test splits, z_ml improves σ_NMAD and η_0.15 over catalog z_best in all six pipeline–layer combinations (Table 3), and r_ml yields lower AUC under the σ_NMAD and η_0.15 versus retained-fraction curves than photoz_risk_best and photoz_conf_best (Table 6, Fig. 9), with a particularly large gain for Mizuki where catalog indicators fail. Feature-importance analyses show complementary roles for PCA/location versus PCA/peak statistics. Code and a value-added catalog are released.
Significance. If the reported gains hold under realistic use, the work is a useful methodological and practical contribution to photo-z post-processing for Stage-III/IV surveys. Strengths include: (i) consistent tests across six independent PDZ products rather than a single pipeline; (ii) held-out stratified evaluation with separate calibration for σ_ml; (iii) a clear demonstration that analytic risk/confidence can fail for template-fitting PDZs while a data-driven ranker still works; (iv) physically interpretable grouped permutation importances; and (v) public code plus a planned VAC of z_ml, σ_ml, and r_ml. These make the paper more than a one-off catalog tweak and support reuse on other surveys after retuning.
major comments (3)
- [§5.1, Table 3] §5.1 explicitly notes that HSC pipeline training may overlap the held-out test set, so z_best metrics in Table 3 may be optimistic and the reported z_ml gains are a lower bound. That caveat is appropriate but under-developed relative to the central comparative claim. Please quantify or bound the overlap if feasible (e.g., via public training-set documentation or a rough membership estimate), and state clearly in the abstract/results that the baseline comparison is not fully blind. Without that, readers may over-read Table 3 as a fully independent bake-off.
- [§2.4, §5.1, §6] The strongest application claim is release of z_ml/σ_ml/r_ml for all PDR3 objects (§6, abstract). Both models are supervised only on the spectroscopic cross-match after magnitude/redshift cuts (§2.4, Table 1), and §5.1 already notes possible magnitude-distribution mismatch with the full catalog. The manuscript does not show that ranking quality or point-estimate gains persist under reweighting to the photometric selection, in faint bins with sparse spec-z, or on a held-out spectroscopic survey not used in labeling. This does not invalidate the in-sample tables, but it is load-bearing for the VAC. Please add a dedicated limitations subsection with concrete guidance (e.g., recommended r_ml thresholds only where the labeled prior is representative) and, if possible, one simple stress test (magnitude reweighting or bright/faint split) for r_ml AUC and z_ml metrics.
- [§4.2, Eqs. (10)–(14), Table 5] r_ml is defined from a log-space MAE error scale on |Δz_ml| plus percentile ranking (Eqs. 10–14), not from a calibrated probabilistic posterior; the text correctly disclaims NLL-style uncertainty. Table 5 and Fig. 8 support improved ranking versus s_best, but science users may still treat σ_ml as a Gaussian width for error propagation. Please state more prominently (abstract or §4.2 opening) that σ_ml/r_ml are ranking/filter products, not drop-in replacements for PDZ moments in forward modeling, and avoid language that equates σ_ml with a standardized uncertainty without that caveat.
minor comments (6)
- [§3.3, Eq. (1)] Eq. (1): the composite objective normalizes by training-split z_best metrics; briefly note whether fold-wise renormalization was tried and whether equal 1/2 weights were ablated, even if only in the appendix.
- [Fig. 1, Table 2] Figure 1 is illustrative but has no quantitative summary of PDZ diversity (e.g., typical multimodality rates). A short sentence or small table would help motivate PCA component counts in Table 2.
- [Table 3] Table 3 highlights best results in red; ensure the monochrome/print version remains readable (bold or boxes).
- [§4.1] §4.1 odds discussion (Eqs. 7–9) is clear but slightly long relative to its use; consider tightening or moving part to an appendix.
- [Data availability] Data availability: confirm whether the VAC DOI/release path will be frozen at acceptance and whether model cards list the exact train/test object IDs for reproducibility.
- [Abstract, §1] Minor typography: “photo-zor” / “spec-zorz” spacing in the introduction; “Optunaunder” and similar missing spaces in the abstract PDF text; unify “Deep/UltraDeep” vs “DUD”.
Circularity Check
No significant circularity: empirical ML supervised on external spectroscopic redshifts, with held-out comparisons that are not true by construction.
full rationale
turboPDZ trains point-estimate and uncertainty networks on PDZ features against spectroscopic redshifts as external ground truth. z_ml is optimized with MAE (and HPO under a composite of σ_NMAD and η_0.15), then evaluated on a held-out labeled test split against catalog z_best; it is not defined to equal z_best and does not use z_best as an input feature. r_ml is built by regressing a log-error proxy for |Δz_ml|, calibrating, and percentile-ranking—coupling reliability training to the point-estimate model—but the filtering claims (AUC under σ_NMAD/η_0.15 vs retained fraction; correlations with |Δz|) remain empirical comparisons to held-out spec-z and to catalog risk/confidence indicators, not identities forced by definition. There is no load-bearing self-citation uniqueness theorem, no ansatz smuggled in as a theorem, and no renaming of a known closed-form result as a derivation. Representativeness of the spectroscopic cross-match for the full photometric catalog is a generalization/validity concern, not circularity of the in-sample derivation chain.
Axiom & Free-Parameter Ledger
free parameters (6)
- PCA explained-variance fraction / number of components =
e.g. 0.50–0.90 EV; 6–58 components depending on config (Tables 2, 4)
- MLP architecture and training hyperparameters =
Per-config best values in Tables 2 and 4
- Composite objective weights for z_ml (½ σ_NMAD + ½ η_0.15 ratios) =
1/2, 1/2
- log-space floor ε for uncertainty target =
~1e-5 to 1e-4
- Uncertainty calibration (n_bins, Gaussian bandwidth w, multiplicative m) =
Example DEmP-Wide: n_bins=156, bw=0.06 dex (Fig. 7)
- Magnitude and redshift selection cuts on labeled sample =
As in §2.4 / Table 1
axioms (5)
- domain assumption Spectroscopic redshifts in the HSC compilation are sufficiently accurate ground truth for supervised photo-z metrics.
- domain assumption Standard photo-z metrics (Δz, σ_NMAD, η_0.15, f_0.02) and retained-fraction AUC capture the scientifically relevant quality of point estimates and filters.
- ad hoc to paper A single error scale trained with log-space MAE plus percentile ranking is an adequate reliability score even though it is not a calibrated probabilistic posterior.
- ad hoc to paper PCA compression plus hand-crafted PDZ descriptors retain the information needed to beat catalog extractors.
- domain assumption Stratified hold-out evaluation on the labeled set is a valid proxy for performance on science samples drawn from the photometric catalog.
invented entities (1)
-
turboPDZ framework (z_ml, σ_ml, r_ml)
independent evidence
read the original abstract
Photometric redshifts are essential for large-scale structure analyses, yet extracting optimal point estimates and reliability measures from the probability distribution functions (PDZs) delivered by photo-$z$ pipelines remains an open challenge. We introduce turboPDZ, a machine-learning framework that optimizes both quantities directly from the PDZ. We apply the framework to PDZs from the three independent HSC-SSP PDR3 pipelines (DEmP, DNNz, Mizuki) across Wide and DUD layers. Each PDZ is compressed via PCA and combined with summary descriptors; a multilayer perceptron, optimized with Optuna under a composite objective, produces the optimized point estimate $z_{\rm ml}$. A second network, trained in log-space and calibrated, yields the uncertainty $\sigma_{\rm ml}$, from which the reliability score $r_{\rm ml}$ is derived via percentile ranking. $z_{\rm ml}$ outperforms the catalog $z_{\rm best}$ in $\sigma_{\rm NMAD}$ and $\eta_{0.15}$ across all six pipeline-layer combinations. $r_{\rm ml}$ filters galaxies more efficiently than the catalog risk and confidence indicators, as measured by the area under the $\sigma_{\rm NMAD}$ and $\eta_{0.15}$ versus retained-fraction curves. For Mizuki, the template-fitting pipeline, the catalog indicators fail dramatically, with AUC values up to ten times larger than those of $r_{\rm ml}$, whereas $r_{\rm ml}$ correctly identifies unreliable objects across all redshift regimes. Feature-importance analysis reveals complementary patterns: point estimation is dominated by PCA components and location statistics, while reliability estimation depends on PCA components and peak statistics. The pipeline is survey-independent, publicly available at https://github.com/valerio-marra/turboPDZ, and trained models plus optimized quantities are released as a value-added catalog.
Figures
Reference graph
Works this paper leans on
-
[1]
Aihara, H. et al. 2018, Publ. Astron. Soc. Jap., 70, S4, [1704.05858]
Pith/arXiv arXiv 2018
-
[2]
Aihara, H. et al. 2022, Publ. Astron. Soc. Jap., 74, 247, [2108.13045]
Pith/arXiv arXiv 2022
-
[3]
2019, arXiv e-prints, arXiv:1907.10902, [1907.10902]
Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. 2019, arXiv e-prints, arXiv:1907.10902, [1907.10902]
Pith/arXiv arXiv 2019
-
[4]
Bechtol, K. et al. 2026, Astrophys. J. Suppl., 282, 62, [2501.05739]. Benítez, N. 2000, ApJ, 536, 571, [astro-ph/9811189]
Pith/arXiv arXiv 2026
-
[5]
Bonoli, S. et al. 2021, Astron. Astrophys., 653, A31, [2007.01910]
Pith/arXiv arXiv 2021
-
[6]
Dalal, R. et al. 2023, Phys. Rev. D, 108, 123519, [2304.00701]
Pith/arXiv arXiv 2023
-
[7]
Desprez, G. et al. 2020, Astron. Astrophys., 644, A31, [2009.12112]. Hernán-Caballero, A. et al. 2021, Astron. Astrophys., 654, A101, [2108.03271]
Pith/arXiv arXiv 2020
-
[8]
Hsieh, B. C. & Yee, H. K. C. 2014, ApJ, 792, 102, [1407.5151]
Pith/arXiv arXiv 2014
-
[9]
2014, A&A, 562, A86, [1308.0063]
Jouvel, S., Host, O., Lahav, O., et al. 2014, A&A, 562, A86, [1308.0063]
Pith/arXiv arXiv 2014
-
[10]
Mellier, Y . et al. 2025, Astron. Astrophys., 697, A1, [2405.13491]
arXiv 2025
-
[11]
Newman, J. A. & Gruen, D. 2022, ARA&A, 60, 363, [2206.13633]
Pith/arXiv arXiv 2022
-
[12]
J., Hsieh, B.-C., & Tanaka, M
Nishizawa, A. J., Hsieh, B.-C., & Tanaka, M. 2021, Photometric Redshifts for the Hyper Suprime-Cam Subaru Strategic Program Data Release 3, HSC-SSP website, PDR3, available athttps: //hsc-release.mtk.nao.ac.jp/doc/wp-content/uploads/ 2022/08/pdr3_photoz.pdf
2021
-
[13]
J., Hsieh, B.-C., Tanaka, M., & Takata, T
Nishizawa, A. J., Hsieh, B.-C., Tanaka, M., & Takata, T. 2020, arXiv e-prints, arXiv:2003.01511, [2003.01511]
Pith/arXiv arXiv 2020
-
[14]
M., Seitz, S., Brimioulle, F., et al
Rau, M. M., Seitz, S., Brimioulle, F., et al. 2015, MNRAS, 452, 3710, [1503.08215]
Pith/arXiv arXiv 2015
-
[15]
2019, Nature Astronomy, 3, 212, [1805.12574]
Salvato, M., Ilbert, O., & Hoyle, B. 2019, Nature Astronomy, 3, 212, [1805.12574]
Pith/arXiv arXiv 2019
-
[16]
Schmidt, S. J. et al. 2020, Mon. Not. Roy. Astron. Soc., 499, 1587, [2001.03621]
Pith/arXiv arXiv 2020
-
[17]
Sevilla-Noarbe, I. et al. 2021, Astrophys. J. Suppl., 254, 24, [2011.03407]
arXiv 2021
- [18]
-
[19]
Tanaka, M., Coupon, J., Hsieh, B.-C., et al. 2018, Publ. Astron. Soc. Jap., 70, S9, [1704.05988]
Pith/arXiv arXiv 2018
-
[20]
Teixeira, G., Bom, C. R., Santana-Silva, L., et al. 2024, Astronomy and Computing, 49, 100886, [2408.15243]. The LSST Dark Energy Science Collaboration, Mandelbaum, R., Eifler, T., et al. 2018, arXiv e-prints, arXiv:1809.01669, [1809.01669]
Pith/arXiv arXiv 2024
-
[21]
2016, MNRAS, 457, 4005, [1601.07857]
Wittman, D., Bhaskar, R., & Tobin, R. 2016, MNRAS, 457, 4005, [1601.07857]
Pith/arXiv arXiv 2016
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.