Pith. sign in

REVIEW 3 major objections 6 minor 21 references

A machine-learning pipeline that reads photometric-redshift probability distributions yields better point estimates and reliability scores than the catalog defaults across six HSC pipeline–layer combinations.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · grok-4.5

2026-07-30 20:04 UTC pith:535CEJJQ

load-bearing objection Practical, well-executed ML stack that consistently beats HSC catalog point estimates and quality flags on six PDZ sets, with public code and a VAC; main caveat is labeled-spec generalization. the 3 major comments →

arxiv 2607.26822 v1 pith:535CEJJQ submitted 2026-07-29 astro-ph.IM astro-ph.CO

Optimizing the extraction of information from redshift probability distribution functions

classification astro-ph.IM astro-ph.CO
keywords photometric redshiftsredshift PDFspoint estimatesreliability scoresHSC-SSPmachine learninglarge-scale structurecatalogs
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Photometric redshifts are delivered as full probability curves, yet large-scale structure work usually collapses each curve to a single number and a quality flag. This paper argues that those two numbers can be optimized directly from the curve with a survey-independent neural pipeline called turboPDZ. On Hyper Suprime-Cam data, the optimized redshift beats the catalog “best” estimate in scatter and outlier rate for every one of the six pipeline-and-depth combinations tested. The companion reliability score filters galaxies more efficiently than the catalog risk and confidence indicators, and it still works when the catalog flags fail—most dramatically for the template-fitting pipeline. The practical payoff is cleaner gold samples for weak lensing and clustering without throwing away as many galaxies.

Core claim

Across all six HSC-SSP PDR3 pipeline–layer combinations, an optimized point estimate z_ml extracted from each galaxy’s redshift probability distribution improves σ_NMAD and η_0.15 relative to the catalog z_best, and a data-driven reliability score r_ml, derived from a calibrated uncertainty σ_ml, filters galaxies more efficiently than photoz_risk_best and photoz_conf_best as measured by the area under the quality-versus-retained-fraction curves. For the Mizuki template-fitting pipeline the catalog indicators fail dramatically (AUC up to about ten times larger), while r_ml still identifies unreliable objects across redshift regimes.

What carries the argument

turboPDZ: each PDZ is PCA-compressed and joined to magnitude and summary descriptors; one multilayer perceptron, tuned under a composite σ_NMAD–η_0.15 objective, outputs z_ml; a second network trained in log-error space and post-hoc calibrated outputs σ_ml, which is turned into the percentile-rank reliability score r_ml.

Load-bearing premise

Models trained on the spectroscopic cross-match must still rank and predict well on the full photometric catalog and on science samples whose magnitude and redshift mix differ from the labeled set.

What would settle it

On a fully held-out spectroscopic sample with no overlap to any training or calibration set used by either turboPDZ or the original HSC pipelines, check whether z_ml still beats z_best in σ_NMAD and η_0.15 and whether r_ml still yields smaller AUC under the quality-versus-retained-fraction curves than the catalog risk and confidence flags.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Gold photometric samples for weak lensing and clustering can keep more galaxies at fixed scatter and outlier rate by cutting on r_ml instead of catalog risk or confidence.
  • Template-fitting PDZs that are sharply peaked at the wrong redshift can still be flagged as unreliable by r_ml even when analytic risk and confidence fail.
  • The same PCA-plus-descriptor architecture can be retuned for other surveys’ PDZs without redesigning the pipelines that produced those PDZs.
  • Value-added z_ml, σ_ml, and r_ml columns for the full HSC-SSP PDR3 catalog become available for immediate downstream use.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because point estimation leans on location statistics while reliability leans on peak morphology, joint multi-task training of z and σ might trade a little redshift accuracy for still-stronger ranking of bad objects.
  • The method’s dependence on a representative spectroscopic anchor implies that deep, sparse fields (or high-z tails) will need explicit domain-adaptation checks before r_ml cuts are trusted at the same thresholds as in the training domain.
  • Surveys that already ship full PDZs but weak quality flags (including Stage-IV optical and near-IR catalogs) are the natural next stress tests of whether the Mizuki-style rescue generalizes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents turboPDZ, a survey-independent ML pipeline that extracts an optimized photometric-redshift point estimate z_ml and a reliability score r_ml directly from photo-z PDZs. PDZs from the three HSC-SSP PDR3 pipelines (DEmP, DNNz, Mizuki) in Wide and DUD layers are PCA-compressed, augmented with hand-crafted descriptors, and fed to Optuna-tuned MLPs. z_ml is trained under a composite σ_NMAD/η_0.15 objective; σ_ml is trained in log-space with post-hoc calibration and converted to r_ml by percentile ranking. On held-out spectroscopic test splits, z_ml improves σ_NMAD and η_0.15 over catalog z_best in all six pipeline–layer combinations (Table 3), and r_ml yields lower AUC under the σ_NMAD and η_0.15 versus retained-fraction curves than photoz_risk_best and photoz_conf_best (Table 6, Fig. 9), with a particularly large gain for Mizuki where catalog indicators fail. Feature-importance analyses show complementary roles for PCA/location versus PCA/peak statistics. Code and a value-added catalog are released.

Significance. If the reported gains hold under realistic use, the work is a useful methodological and practical contribution to photo-z post-processing for Stage-III/IV surveys. Strengths include: (i) consistent tests across six independent PDZ products rather than a single pipeline; (ii) held-out stratified evaluation with separate calibration for σ_ml; (iii) a clear demonstration that analytic risk/confidence can fail for template-fitting PDZs while a data-driven ranker still works; (iv) physically interpretable grouped permutation importances; and (v) public code plus a planned VAC of z_ml, σ_ml, and r_ml. These make the paper more than a one-off catalog tweak and support reuse on other surveys after retuning.

major comments (3)
  1. [§5.1, Table 3] §5.1 explicitly notes that HSC pipeline training may overlap the held-out test set, so z_best metrics in Table 3 may be optimistic and the reported z_ml gains are a lower bound. That caveat is appropriate but under-developed relative to the central comparative claim. Please quantify or bound the overlap if feasible (e.g., via public training-set documentation or a rough membership estimate), and state clearly in the abstract/results that the baseline comparison is not fully blind. Without that, readers may over-read Table 3 as a fully independent bake-off.
  2. [§2.4, §5.1, §6] The strongest application claim is release of z_ml/σ_ml/r_ml for all PDR3 objects (§6, abstract). Both models are supervised only on the spectroscopic cross-match after magnitude/redshift cuts (§2.4, Table 1), and §5.1 already notes possible magnitude-distribution mismatch with the full catalog. The manuscript does not show that ranking quality or point-estimate gains persist under reweighting to the photometric selection, in faint bins with sparse spec-z, or on a held-out spectroscopic survey not used in labeling. This does not invalidate the in-sample tables, but it is load-bearing for the VAC. Please add a dedicated limitations subsection with concrete guidance (e.g., recommended r_ml thresholds only where the labeled prior is representative) and, if possible, one simple stress test (magnitude reweighting or bright/faint split) for r_ml AUC and z_ml metrics.
  3. [§4.2, Eqs. (10)–(14), Table 5] r_ml is defined from a log-space MAE error scale on |Δz_ml| plus percentile ranking (Eqs. 10–14), not from a calibrated probabilistic posterior; the text correctly disclaims NLL-style uncertainty. Table 5 and Fig. 8 support improved ranking versus s_best, but science users may still treat σ_ml as a Gaussian width for error propagation. Please state more prominently (abstract or §4.2 opening) that σ_ml/r_ml are ranking/filter products, not drop-in replacements for PDZ moments in forward modeling, and avoid language that equates σ_ml with a standardized uncertainty without that caveat.
minor comments (6)
  1. [§3.3, Eq. (1)] Eq. (1): the composite objective normalizes by training-split z_best metrics; briefly note whether fold-wise renormalization was tried and whether equal 1/2 weights were ablated, even if only in the appendix.
  2. [Fig. 1, Table 2] Figure 1 is illustrative but has no quantitative summary of PDZ diversity (e.g., typical multimodality rates). A short sentence or small table would help motivate PCA component counts in Table 2.
  3. [Table 3] Table 3 highlights best results in red; ensure the monochrome/print version remains readable (bold or boxes).
  4. [§4.1] §4.1 odds discussion (Eqs. 7–9) is clear but slightly long relative to its use; consider tightening or moving part to an appendix.
  5. [Data availability] Data availability: confirm whether the VAC DOI/release path will be frozen at acceptance and whether model cards list the exact train/test object IDs for reproducibility.
  6. [Abstract, §1] Minor typography: “photo-zor” / “spec-zorz” spacing in the introduction; “Optunaunder” and similar missing spaces in the abstract PDF text; unify “Deep/UltraDeep” vs “DUD”.

Circularity Check

0 steps flagged

No significant circularity: empirical ML supervised on external spectroscopic redshifts, with held-out comparisons that are not true by construction.

full rationale

turboPDZ trains point-estimate and uncertainty networks on PDZ features against spectroscopic redshifts as external ground truth. z_ml is optimized with MAE (and HPO under a composite of σ_NMAD and η_0.15), then evaluated on a held-out labeled test split against catalog z_best; it is not defined to equal z_best and does not use z_best as an input feature. r_ml is built by regressing a log-error proxy for |Δz_ml|, calibrating, and percentile-ranking—coupling reliability training to the point-estimate model—but the filtering claims (AUC under σ_NMAD/η_0.15 vs retained fraction; correlations with |Δz|) remain empirical comparisons to held-out spec-z and to catalog risk/confidence indicators, not identities forced by definition. There is no load-bearing self-citation uniqueness theorem, no ansatz smuggled in as a theorem, and no renaming of a known closed-form result as a derivation. Representativeness of the spectroscopic cross-match for the full photometric catalog is a generalization/validity concern, not circularity of the in-sample derivation chain.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 1 invented entities

The central claims are empirical ML performance claims. They rest on standard photo-z evaluation definitions, the representativeness of the HSC spectroscopic compilation after magnitude/redshift cuts, architectural and objective choices fixed by Optuna on training folds, and a post-hoc uncertainty calibration mapped to percentile reliability. No new physical entities are postulated; free parameters are hyperparameters and calibration knobs, not cosmological constants.

free parameters (6)
  • PCA explained-variance fraction / number of components = e.g. 0.50–0.90 EV; 6–58 components depending on config (Tables 2, 4)
    Treated as an Optuna hyperparameter per pipeline-layer; directly changes the PDZ representation fed to both networks.
  • MLP architecture and training hyperparameters = Per-config best values in Tables 2 and 4
    Layers, units, dropout, LR, weight decay, batch size, activation chosen by Optuna; capacity and regularization affect reported metrics.
  • Composite objective weights for z_ml (½ σ_NMAD + ½ η_0.15 ratios) = 1/2, 1/2
    Equal weights are a hand choice (Eq. 1); different weights would change which point estimates are preferred.
  • log-space floor ε for uncertainty target = ~1e-5 to 1e-4
    ε fixed to ~0.1th percentile of |Δz_ml|; shifts the transformed regression target t=log(|Δz|+ε).
  • Uncertainty calibration (n_bins, Gaussian bandwidth w, multiplicative m) = Example DEmP-Wide: n_bins=156, bw=0.06 dex (Fig. 7)
    Post-hoc m(σ̃) fit on calibration split via grid/CV; rescales raw network outputs into σ_ml used for r_ml.
  • Magnitude and redshift selection cuts on labeled sample = As in §2.4 / Table 1
    Wide 18.5<i<25, DUD 19<i<26, 0.01<z_spec<5 define the training domain; metrics and models depend on these cuts.
axioms (5)
  • domain assumption Spectroscopic redshifts in the HSC compilation are sufficiently accurate ground truth for supervised photo-z metrics.
    All losses and σ_NMAD/η evaluations use z_spec as truth (§2.3, §3.4).
  • domain assumption Standard photo-z metrics (Δz, σ_NMAD, η_0.15, f_0.02) and retained-fraction AUC capture the scientifically relevant quality of point estimates and filters.
    Optimization objective and headline comparisons are defined entirely in these metrics (Eqs. 1–6, 17–18).
  • ad hoc to paper A single error scale trained with log-space MAE plus percentile ranking is an adequate reliability score even though it is not a calibrated probabilistic posterior.
    Authors explicitly reject NLL-style probabilistic uncertainty because it hurt ranking performance (§4.2).
  • ad hoc to paper PCA compression plus hand-crafted PDZ descriptors retain the information needed to beat catalog extractors.
    Pipeline design choice in §3.2; supported empirically but not proven complete.
  • domain assumption Stratified hold-out evaluation on the labeled set is a valid proxy for performance on science samples drawn from the photometric catalog.
    Load-bearing for releasing VAC quantities on all PDR3 objects (§5–6).
invented entities (1)
  • turboPDZ framework (z_ml, σ_ml, r_ml) independent evidence
    purpose: Package PCA+MLP point estimation and calibrated log-error reliability ranking as reusable survey post-processing.
    Methodological construct rather than a physical entity; independent evidence is empirical outperformance on HSC and public code/catalog, not a new predicted particle or force.

pith-pipeline@v1.2.0-daily-grok45 · 23634 in / 3643 out tokens · 73436 ms · 2026-07-30T20:04:19.626336+00:00 · methodology

0 comments
read the original abstract

Photometric redshifts are essential for large-scale structure analyses, yet extracting optimal point estimates and reliability measures from the probability distribution functions (PDZs) delivered by photo-$z$ pipelines remains an open challenge. We introduce turboPDZ, a machine-learning framework that optimizes both quantities directly from the PDZ. We apply the framework to PDZs from the three independent HSC-SSP PDR3 pipelines (DEmP, DNNz, Mizuki) across Wide and DUD layers. Each PDZ is compressed via PCA and combined with summary descriptors; a multilayer perceptron, optimized with Optuna under a composite objective, produces the optimized point estimate $z_{\rm ml}$. A second network, trained in log-space and calibrated, yields the uncertainty $\sigma_{\rm ml}$, from which the reliability score $r_{\rm ml}$ is derived via percentile ranking. $z_{\rm ml}$ outperforms the catalog $z_{\rm best}$ in $\sigma_{\rm NMAD}$ and $\eta_{0.15}$ across all six pipeline-layer combinations. $r_{\rm ml}$ filters galaxies more efficiently than the catalog risk and confidence indicators, as measured by the area under the $\sigma_{\rm NMAD}$ and $\eta_{0.15}$ versus retained-fraction curves. For Mizuki, the template-fitting pipeline, the catalog indicators fail dramatically, with AUC values up to ten times larger than those of $r_{\rm ml}$, whereas $r_{\rm ml}$ correctly identifies unreliable objects across all redshift regimes. Feature-importance analysis reveals complementary patterns: point estimation is dominated by PCA components and location statistics, while reliability estimation depends on PCA components and peak statistics. The pipeline is survey-independent, publicly available at https://github.com/valerio-marra/turboPDZ, and trained models plus optimized quantities are released as a value-added catalog.

Figures

Figures reproduced from arXiv: 2607.26822 by Rodrigo Duarte, Valerio Marra.

Figure 1
Figure 1. Figure 1: Example of the six PDZs available for a single galaxy, obtained from the three pipelines (DEmP, Mizuki, DNNz) in both the Wide and Deep/UltraDeep (DUD) layers. The comparison highlights systematic dif￾ferences in shape, width, and modality across the pipelines. rameter) and fits a quadratic model to estimate the redshift. PDZs are generated by combining Monte Carlo perturbations of the photometry (500 real… view at source ↗
Figure 2
Figure 2. Figure 2: Spectroscopic redshift versus i-band magnitude for the labeled sets used in this work. The panels show the distributions for the DEmP pipeline in the DUD and Wide layers. The dashed lines indicate the applied cuts. The corresponding plots for Mizuki and DNNz are basically the same [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Cumulative photo-z metrics as a function of i-band magnitude cut, i.e. the x-axis denotes the faintest magnitude of the subsample. est reliability scores (and analogously for the catalog indicators). Graphically, these areas correspond to the regions under the curves shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Photo-z metrics versus spec-z in non-cumulative bins of width 0.20. PCA block Location stats Shape stats Peak stats icmodel DEmP Wide DEmP DUD DNNz Wide DNNz DUD Mizuki Wide Mizuki DUD 9.48 (1102%) 0.28 (32%) 0.13 (15%) 0.02 (2%) 0.04 (4%) 1.50 (160%) 7.97 (851%) 0.45 (49%) 0.29 (31%) 0.11 (12%) 8.35 (949%) 1.38 (157%) 0.43 (49%) 0.31 (35%) 0.17 (19%) 7.60 (825%) 1.59 (173%) 0.53 (57%) 0.15 (16%) 0.40 (43%… view at source ↗
Figure 5
Figure 5. Figure 5: Grouped permutation feature importance across the six pipeline￾layer datasets. Cell colors encode the within-pipeline rank class of each feature group according to ∆Jcomp (rank 1 = highest importance). Cell annotations report both the raw ∆Jcomp and its relative change (%). baseline in MAE and in Jcomp (Eq. 1), with ∆Jcomp as the primary ranking metric [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Representative DEmP-DUD slice analysis for PC2, PC4, and PC6. For each component (one column), low score denotes the bottom 20% and high score the top 20% of PC scores. Top panels show mean PDZs for low-score (blue) and high-score (orange) objects; bottom panels show the corresponding zspec distributions and the metrics for each slice. The opposite tails of each component isolate distinct regimes: for PC4 … view at source ↗
Figure 7
Figure 7. Figure 7: shows the calibration curve m(˜σml) for the DEmP￾Wide configuration, with the raw network output σ˜ml on the x-axis and the multiplicative correction m on the y-axis. The curve is close to unity across most of the range, indicating that the raw net￾work output is well-calibrated. Deviations from unity at low and high σ˜ml reflect residual miscalibration in those regimes, which 10 2 10 1 median ml per bin 0… view at source ↗
Figure 8
Figure 8. Figure 8: Comparison of the ML estimate σml with the catalog estimator sbest of Eq. (16) on the test set. 0.005 0.010 0.015 0.020 0.025 0.030 N M A D DEmP DNNz Mizuki 0.2 0.4 0.6 0.8 1.0 retained fraction 0.00 0.02 0.04 0.06 0.08 0.10 0.15 0.2 0.4 0.6 0.8 1.0 retained fraction 0.2 0.4 0.6 0.8 1.0 retained fraction wide dud rml photoz_conf_best photoz_risk_best [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: σNMAD and η0.15 versus retained fraction for selections based on rml , photoz_risk_best, and photoz_conf_best. Cuts on rml provide the best trade-off between sample size and photo-z quality [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: summarizes the cross-pipeline results across all six pipeline-layer configurations. The feature-importance pattern for the rml model differs markedly from that of the point-estimate model zml. For zml, the PDF-PCA block and location statistics dominate, with mean ranks of 1.2 and 1.8 respectively across the six configurations, while the remaining groups are clearly secondary. For rml, this ordering is lar… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 18 linked inside Pith

  1. [1]

    Aihara, H. et al. 2018, Publ. Astron. Soc. Jap., 70, S4, [1704.05858]

  2. [2]

    Aihara, H. et al. 2022, Publ. Astron. Soc. Jap., 74, 247, [2108.13045]

  3. [3]

    2019, arXiv e-prints, arXiv:1907.10902, [1907.10902]

    Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. 2019, arXiv e-prints, arXiv:1907.10902, [1907.10902]

  4. [4]

    Bechtol, K. et al. 2026, Astrophys. J. Suppl., 282, 62, [2501.05739]. Benítez, N. 2000, ApJ, 536, 571, [astro-ph/9811189]

  5. [5]

    Bonoli, S. et al. 2021, Astron. Astrophys., 653, A31, [2007.01910]

  6. [6]

    Dalal, R. et al. 2023, Phys. Rev. D, 108, 123519, [2304.00701]

  7. [7]

    Desprez, G. et al. 2020, Astron. Astrophys., 644, A31, [2009.12112]. Hernán-Caballero, A. et al. 2021, Astron. Astrophys., 654, A101, [2108.03271]

  8. [8]

    Hsieh, B. C. & Yee, H. K. C. 2014, ApJ, 792, 102, [1407.5151]

  9. [9]

    2014, A&A, 562, A86, [1308.0063]

    Jouvel, S., Host, O., Lahav, O., et al. 2014, A&A, 562, A86, [1308.0063]

  10. [10]

    Mellier, Y . et al. 2025, Astron. Astrophys., 697, A1, [2405.13491]

  11. [11]

    Newman, J. A. & Gruen, D. 2022, ARA&A, 60, 363, [2206.13633]

  12. [12]

    J., Hsieh, B.-C., & Tanaka, M

    Nishizawa, A. J., Hsieh, B.-C., & Tanaka, M. 2021, Photometric Redshifts for the Hyper Suprime-Cam Subaru Strategic Program Data Release 3, HSC-SSP website, PDR3, available athttps: //hsc-release.mtk.nao.ac.jp/doc/wp-content/uploads/ 2022/08/pdr3_photoz.pdf

  13. [13]

    J., Hsieh, B.-C., Tanaka, M., & Takata, T

    Nishizawa, A. J., Hsieh, B.-C., Tanaka, M., & Takata, T. 2020, arXiv e-prints, arXiv:2003.01511, [2003.01511]

  14. [14]

    M., Seitz, S., Brimioulle, F., et al

    Rau, M. M., Seitz, S., Brimioulle, F., et al. 2015, MNRAS, 452, 3710, [1503.08215]

  15. [15]

    2019, Nature Astronomy, 3, 212, [1805.12574]

    Salvato, M., Ilbert, O., & Hoyle, B. 2019, Nature Astronomy, 3, 212, [1805.12574]

  16. [16]

    Schmidt, S. J. et al. 2020, Mon. Not. Roy. Astron. Soc., 499, 1587, [2001.03621]

  17. [17]

    Sevilla-Noarbe, I. et al. 2021, Astrophys. J. Suppl., 254, 24, [2011.03407]

  18. [18]

    2015, ApJ, 801, 20, [1501.02047]

    Tanaka, M. 2015, ApJ, 801, 20, [1501.02047]

  19. [19]

    2018, Publ

    Tanaka, M., Coupon, J., Hsieh, B.-C., et al. 2018, Publ. Astron. Soc. Jap., 70, S9, [1704.05988]

  20. [20]

    R., Santana-Silva, L., et al

    Teixeira, G., Bom, C. R., Santana-Silva, L., et al. 2024, Astronomy and Computing, 49, 100886, [2408.15243]. The LSST Dark Energy Science Collaboration, Mandelbaum, R., Eifler, T., et al. 2018, arXiv e-prints, arXiv:1809.01669, [1809.01669]

  21. [21]

    2016, MNRAS, 457, 4005, [1601.07857]

    Wittman, D., Bhaskar, R., & Tobin, R. 2016, MNRAS, 457, 4005, [1601.07857]