REVIEW 3 major objections 4 minor 1 cited by
Regression for Astronomical Data with Realistic Distributions, Errors and Non-linearity
T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper claims that a normalizing-flow-based maximum-likelihood regression, and a 2D KS-test variant, recover unbiased slopes, intercepts, and intrinsic scatter from astronomical data with unknown intrinsic distributions and error…
desk verdict A genuine advance in empirical-Bayes regression for astronomical data, with an unusually thorough mock-test campaign; the latent-distribution validation and code release need attention before I'd trust the headline low-S/N claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the normalizing flow: a bijective map $f_\phi$ that turns the variable of interest into a latent variable $z$ with a tractable density, so the modeled density is $P_\phi(x)=P_z(f_\phi(x))\,|\det J_{f_\phi}(x)|$. The paper uses two such flows, one for the pair $(x_{\rm err},x)$ and one for $(y_{\rm err},y)$, each factorized as $P(x_{\rm err}|x;\phi_1)P(x;\phi_2)$, and fits them by maximizing the marginal likelihood of the observed $(x_{\rm obs},x_{\rm err})$ pairs, with the latent $x$ integrated out by Gauss-Hermite quadrature — a variational-inference/empirical-Bayes step. The fitted densities enter the regression likelihood as if they were known. In the KS-test variant, the same generative machinery produces synthetic $(x_{\rm obs},y_{\rm obs})$ data under trial parameters, and the optimization objective is the p-value of the 2D Kolmogorov-Smirnov test comparing synthetic and observed data; because that p-value is not differentiable, the search is done by iterative grid refinement. This is the mechanism that lets the method avoid assuming a Gaussian mixture for $P(x)$ and avoid requiring the user to supply the intrinsic distribution.
What would settle it
Take a mock sample with $N=2000$, a known bimodal $P(\log x)$, $k=0.5$, $b=0$, $\sigma=0.3$, typical S/N of 1 for both variables, and known Gaussian errors; run the public raddest implementation with the NF given too few training epochs so that its 2D-KS goodness-of-fit p-value is below $10^{-3}$. If the recovered $(k,b,\sigma)$ are then as biased as LINMIX's, the claim that the method itself, rather than a well-fitted NF, delivers the advantage would be called into question.
Extended reading notes
Core claim
The central claim is that the bias that plagues regression with noisy independent variables can be removed by learning the nuisance distributions from the data itself rather than assuming them. Specifically, the paper argues that a normalizing flow trained on the observed pairs $(x_{\rm obs}, x_{\rm err})$ can recover the latent intrinsic distribution $P(x)$ and the conditional error-level distribution $P(x_{\rm err}|x)$ through the empirical-Bayes marginal likelihood in Eq. (9); plugging these estimates into the full likelihood of Eq. (3) then yields maximum-likelihood estimates of the slope $k$, intercept $b$, and intrinsic scatter $\sigma$ that are unbiased for log-linear relations, provided the sample is large enough (roughly $N>1000$). The KS-test variant replaces the likelihood objective with the p-value of a 2D KS test between data generated by the model and the observed data, and is claimed to be markedly more robust when the model assumptions are violated. The paper further claims that for $300\lesssim N\lesssim 1000$ the ML variant remains the best performer, while for $N<300$ it is competitive with, though no better than, LINMIX and Leopy.
Load-bearing premise
The load-bearing premise is that the normalizing flow trained on observed $(x_{\rm obs},x_{\rm err})$ pairs gives an accurate estimate of the latent intrinsic distribution $P(x)$ and the conditional error distribution $P(x_{\rm err}|x)$, and that these estimates can be treated as known when the regression likelihood is evaluated; if the flow is undertrained or too rigid, the likelihood is misspecified and the regression parameters can be biased, which is consistent with the paper's finding that the advantage shrinks below about 300 samples.
Editorial extensions
If this is right
- At sample sizes above about 1000, the method is claimed to outperform all eight comparison methods on mock log-linear data, with the KS variant recording an extreme-error fraction of zero for slope, intercept, and scatter across 200 mock datasets.
- The KS variant is claimed to be robust to violations of the log-linear assumption, complex intrinsic distributions, complex uncertainty-intrinsic correlations, misestimated uncertainties, outliers, and saturation, so it can be applied where standard methods degrade.
- For samples of roughly 300–1000, the ML variant is claimed to retain the best median absolute errors, making it the recommended choice in that regime.
- The posterior uncertainties from the new methods are closer to calibrated than LINMIX's (coverage-area 0.09 and 0.10 vs 0.15), though all three remain imperfect.
- On PHANGS-ALMA and PHANGS-JWST maps of CO(2-1) versus mid-infrared flux, the KS method gives steeper slopes and smaller intercepts than mODR and LINMIX, differences the paper traces to saturation at high flux, noise in the independent variable, and the log-linear versus linear error model.
Reading between the lines
- Editorial inference: the two-step empirical-Bayes design means the quality of the regression is gated by the quality of the normalizing-flow fit; the paper's own sample-size results are consistent with this, so a practical prescription would be to require a goodness-of-fit check on the flow before trusting the regression output.
- Editorial inference: because the likelihood in Eq. (3) is written for a multidimensional independent variable and normalizing flows scale to high dimensions, the same machinery should extend to bivariate or multivariate scaling relations, where Gaussian-mixture-based methods struggle.
- Editorial inference: the KS objective, being a distribution-distance criterion, could be reused outside regression — for example, to fit parameters of physical models to noisy observations in other domains where likelihoods are misspecified.
- Editorial inference: when $N<300$, the method's advantage disappears; a user with a small sample might do just as well with a simpler method, and a hybrid that injects prior knowledge of $P(x)$ into the flow could extend the regime.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes two regression methods for astronomical data with measurement errors in both variables, an unknown intrinsic distribution of the independent variable, and correlations between uncertainty levels and intrinsic values. The ML-based method uses a normalizing flow to estimate these nuisance distributions in an empirical-Bayes step, then maximizes the resulting likelihood. The KS-test based variant replaces the likelihood with a 2D Kolmogorov-Smirnov p-value as the optimization objective. Both are tested on extensive mock data covering nonlinearity, complex intrinsic distributions, outliers, saturation, and inaccurate uncertainties, and are applied to PHANGS-ALMA/JWST CO-MIR flux correlations. The central claim is that for sample sizes above 1000, especially at low signal-to-noise ratios, the new methods outperform several established methods (OLS, WLS, ODR, wODR, mODR, LINMIX, LtsFit, Leopy).
Significance. If validated, the methods address a genuine gap: existing Bayesian regression tools such as LINMIX, ROXY, and t-cup rely on Gaussian mixtures and ignore uncertainty-level correlations, while Leopy requires the intrinsic distribution as user input. The mock evaluation is extensive, uses known ground truth (so the performance evidence is external, not circular), and covers a wide range of realistic failure modes. The paper also includes coverage tests and a clear sample-size-dependent recommendation. The code will be made public upon acceptance. However, the central claim rests on the empirical-Bayes step recovering the latent P(x) and P(x_err|x), and this latent recovery is not directly validated; this is the main risk to the soundness of the proposed methods.
major comments (3)
- [Section 2.1, Eqs. (7)-(9)] The likelihood in Eq. (3) treats the NF-estimated P(x; phi_2) and P(x_err|x; phi_1) as known. The only validation reported is a 2D KS test comparing the joint observable distribution (x_obs, x_err) to that generated by the fitted NF. This does not certify that the latent decomposition into P(x) and P(x_err|x) is correct; different latent pairs can lead to the same observable joint distribution, especially when the error distribution is flexible. Since the likelihood and the KS generative model use these latent densities directly, a misspecified NF can bias theta in a way that the observable-only KS test may not detect. I ask the authors to add a direct validation of the recovered latent densities on mocks with known truth, or a sensitivity test in which the NF is intentionally misspecified, to demonstrate that the low-S/N advantage is not an artifact of the empirical-Bayes plug-in.
- [Section 3.1] The paper states that in more than 95% of datasets the default NF configuration yields a 2D KS p-value above 3e-3, and that 'for the cases where the 2D KS test fails, we manually tune the NF hyperparameters to pass the test.' This manual tuning introduces researcher degrees of freedom into the reported results; for up to 5% of the datasets the final model is not produced by the default algorithm. Since the headline claims include extreme-error fractions of 0 (Figure 2), the fraction of manually tuned datasets should be reported per sample, and the analysis should be repeated without those datasets to show that the conclusions do not depend on manual intervention.
- [Section 2.1] The KS-test based method optimizes the p-value of a 2D KS test, which the authors themselves note 'lacks a rigorous mathematical foundation.' The grid-search adaptation and the use of p-values as posterior weights are heuristic. Given that the KS method is recommended as first choice for N>1000, the paper should provide evidence that this objective is well-behaved: for example, a small simulation showing the p-value surface is unimodal, or that the grid search converges to the same solution from different initial ranges. Without this, it is unclear whether the reported robustness of the KS method is a property of the method or of the particular grid-search implementation.
minor comments (4)
- [Section 3.2.1] The definition of the extreme error fraction eta (residuals exceeding 1.0 for k and b, and 0.5 for sigma) first appears only in the Figure 1 caption; it would be helpful to define it in the main text.
- [Throughout] There are several typographical issues, e.g., 'T able 1' in Section 2.3, 'with espect to' in Section 4.2, and subject-verb agreement errors ('exhibits' vs. 'exhibit' in Sections 3.2.7 and 3.3).
- [Section 4.2] The statement that 'different randomly selected subsamples yield highly similar results' is not quantified; please provide the scatter of the fitted parameters across subsamples.
- [Section 3.2.9] The coverage test is only performed on three methods; since the paper emphasizes the KS method's robustness, a coverage test on the other compared methods might be informative, but this is optional.
Circularity Check
No significant circularity: central claims are validated on mock data with known ground truth, and the only self-reference is a non-load-bearing announcement of future work.
full rationale
The paper's derivation chain is self-contained and externally tested. The likelihood (Eq. 3) is a standard marginalization over latent intrinsic values; P(x) and P(x_err|x) are nuisance components estimated from the observed data by empirical Bayes (Eqs. 7-9), while the target parameters theta remain free and are optimized against either the likelihood or the 2D KS distance between generated and observed (x_obs, y_obs). No equation defines theta in terms of the NF fit or vice versa, so there is no self-definitional reduction. The principal performance claims are evaluated on mock samples with known k, b, sigma, so the evidence is external to the fitted values; the real-data section is a case study, not a claim of first-principles prediction. The paper explicitly credits prior derivations of similar likelihood formulas (Kelly 2007; Feldmann 2019; Bartlett & Desmond 2023; Martin & Mortlock 2024), so it is not renaming a known result. The only self-references are announcements of a parallel paper (Jing & Li, in prep.) and a fork of the tarp package, neither of which supports the central argument. The reader's concern that NF-estimated latent distributions may be misspecified is a statistical identifiability and robustness risk, not a circularity: the paper's own Figure 12 and the reported >95% successful-fit rate quantify this risk rather than assuming it away.
Assumptions & free parameters
free parameters (1)
- NF architecture hyperparameters =
2 layers, 16 bins, learning rate 3e-3
assumptions (4)
- domain assumption Data points are independent
- domain assumption Measurement errors are Gaussian
- ad hoc to paper The normalizing flow can represent the true joint distribution of intrinsic value and error size
- domain assumption The intrinsic scatter in the y-x relation is Gaussian and independent of x
Cite this review
Pith. "Pith review of Regression for Astronomical Data with Realistic Distributions, Errors and Non-linearity." pith.science (2026). https://pith.science/paper/OFBIUB6G
@misc{pith2026241108747,
author = {Pith},
title = {Pith review of: Regression for Astronomical Data with Realistic Distributions, Errors and Non-linearity},
year = {2026},
howpublished = {\url{https://pith.science/paper/OFBIUB6G}},
note = {Machine review of arXiv:2411.08747}
}
read the original abstract
We have developed a new regression technique, the maximum likelihood (ML)-based method and its variant, the KS-test based method, designed to obtain unbiased regression results from typical astronomical data. A normalizing flow model is employed to automatically estimate the unobservable intrinsic distribution of the independent variable as well as the unobservable correlation between uncertainty level and intrinsic value of both independent and dependent variables from the observed data points in a variational inference based empirical Bayes approach. By incorporating these estimated distributions, our method comprehensively accounts for the uncertainties associated with both independent and dependent variables. Our test on both mock data and real astronomical data from PHANGS-ALMA and PHANGS-JWST demonstrates that, given a sufficiently large sample size (> 1000), both the ML-based method and the KS-test based method significantly outperform the existing widely-used methods, particularly in cases of low signal-to-noise ratios. The KS-test based method exhibits remarkable robustness against deviations from underlying assumptions, complex intrinsic distributions, varying correlations between uncertainty levels and intrinsic values, inaccuracies in uncertainty estimations, outliers, and saturation effects. For sample sizes between 300 and 1000, the ML-based method yields the best performance. In the low data regime (< 300), the ML-based method maintains comparable performance to other state-of-the-art methods. A GPU-compatible Python implementation of our methods, nicknamed ``raddest'', will be made publicly available upon acceptance of this paper.
Figures
Figures from the paper (12 more)
Forward citations
Cited by 1 Pith paper
-
The first exploration of the correlations between \textit{WISE} 12 \micron\ and CO emission in early-type galaxies
CO luminosity and WISE 12 micron luminosity are tightly correlated in early-type galaxies, with steeper slopes and a constant CO(2-1)-to-CO(1-0) ratio.
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter doi edition editor eprint howpublished institution journal key month number organization pages publisher school series title misctitle type volume year version url label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts ...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION format.url url empty "" new.block "" url * "" * if FUNCTION format.eprint eprint empty "" archivePrefix empty "" archivePrefix "arXiv" = new.block " " eprint * " " * new.block " " eprint * " " * if if if FUNCTION format.doi doi empty "" " " doi * " " * if FUNCTION format.pid doi empty eprint empty ur...
-
[3]
thebibliography [1] 20pt to REFERENCES 6pt =0pt -12pt 10pt plus 3pt =0pt =0pt =1pt plus 1pt =0pt =0pt -12pt =13pt plus 1pt =20pt =13pt plus 1pt \@M =10000 =-1.0em =0pt =0pt 0pt =0pt =1.0em @enumiv\@empty 10000 10000 `\.\@m \@noitemerr \@latex@warning Empty `thebibliography' environment \@ifnextchar \@reference \@latexerr Missing key on reference command E...
-
[4]
Akritas , M. G., & Bershady , M. A. 1996, , 470, 706, 10.1086/177901
doi:10.1086/177901 1996
-
[5]
Bartlett , D. J., & Desmond , H. 2023, The Open Journal of Astrophysics, 6, 42, 10.21105/astro.2309.00948
arXiv 2023
-
[6]
2013, MNRAS, 432, 1709, 10.1093/mnras/stt562
Cappellari , M., Scott , N., Alatalo , K., et al. 2013, MNRAS, 432, 1709, 10.1093/mnras/stt562
-
[7]
Chown , R., Leroy , A. K., Sandstrom , K., et al. 2024, arXiv e-prints, arXiv:2410.05397, 10.48550/arXiv.2410.05397
-
[8]
Crenshaw, J. F., Kalmbach, J. B., Gagliano, A., et al. 2024 a , Probabilistic Forward Modeling of Galaxy Catalogs with Normalizing Flows. 2405.04740
arXiv 2024
Show all 26 references
-
[9]
F., Yan, Z., & Doster, V
Crenshaw, J. F., Yan, Z., & Doster, V. 2024 b , jfcrenshaw/pzflow: v3.1.3, v3.1.3, Zenodo, 10.5281/zenodo.10710271
2024 doi
- [10]
-
[11]
L., Lin , L., Thorp , M
Ellison , S. L., Lin , L., Thorp , M. D., et al. 2021, , 501, 4777, 10.1093/mnras/staa3822
2021 doi
-
[12]
2019, Astronomy and Computing, 29, 100331, 10.1016/j.ascom.2019.100331
Feldmann , R. 2019, Astronomy and Computing, 29, 100331, 10.1016/j.ascom.2019.100331
2019
-
[13]
Fuller, W. A. 2009, Measurement error models (John Wiley & Sons)
2009
- [14]
-
[15]
D., Akritas , M
Isobe , T., Feigelson , E. D., Akritas , M. G., & Babu , G. J. 1990, , 364, 104, 10.1086/169390
1990 doi
- [16]
-
[17]
Kelly , B. C. 2007, , 665, 1489, 10.1086/519947
2007 doi
-
[18]
2021, Research Notes of the American Astronomical Society, 5, 39, 10.3847/2515-5172/abe8df
Klein , R. 2021, Research Notes of the American Astronomical Society, 5, 39, 10.3847/2515-5172/abe8df
2021 doi
-
[19]
C., Sandstrom , K
Lee , J. C., Sandstrom , K. M., Leroy , A. K., et al. 2023, , 944, L17, 10.3847/2041-8213/acaaae
2023 doi
-
[20]
K., Hughes , A., Liu , D., et al
Leroy , A. K., Hughes , A., Liu , D., et al. 2021 a , , 255, 19, 10.3847/1538-4365/abec80
2021 doi
-
[21]
K., Schinnerer , E., Hughes , A., et al
Leroy , A. K., Schinnerer , E., Hughes , A., et al. 2021 b , , 257, 43, 10.3847/1538-4365/ac17f3
2021 doi
-
[22]
K., Sandstrom , K., Rosolowsky , E., et al
Leroy , A. K., Sandstrom , K., Rosolowsky , E., et al. 2023, , 944, L9, 10.3847/2041-8213/acaf85
2023 doi
-
[23]
Peacock , J. A. 1983, , 202, 615, 10.1093/mnras/202.3.615
1983 doi
-
[24]
H., Teukolsky , S
Press , W. H., Teukolsky , S. A., Vetterling , W. T., & Flannery , B. P. 1992, Numerical recipes in FORTRAN. The art of scientific computing
1992
-
[25]
Robotham , A. S. G., & Obreschkow , D. 2015, , 32, e033, 10.1017/pasa.2015.33
2015 doi
-
[26]
G., Lee , J
Williams , T. G., Lee , J. C., Larson , K. L., et al. 2024, , 273, 13, 10.3847/1538-4365/ad4be5
2024 doi
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.