REVIEW 3 major objections 5 minor 12 references
Projection Pursuit Density Ratio Estimation
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Projection pursuit turns high-dimensional density ratio estimation into a chain of one-dimensional curve fits.
desk verdict Genuinely new multiplicative projection pursuit idea for DRE, but the theory covers the linear-sieve estimator while the implemented algorithm also fits Gaussian locations, so the rates do not actually cover the method being reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the multiplicative projection pursuit approximation $r_K(x)=\prod_{k=1}^K f_k(\boldsymbol a_k^\top x)$, where each $f_k$ is a univariate ridge function (a one-dimensional function of the projection $\boldsymbol a_k^\top x$). The fitting step minimizes $H(f,\boldsymbol a)=\mathbb E_q[r_{k-1}^2 f^2(\boldsymbol a^\top x)]-2\mathbb E_p[r_{k-1} f(\boldsymbol a^\top x)]$, which is the nonconstant part of the $L^2$ distance to the target ratio; in practice this is replaced by its regularized empirical version with a univariate linear sieve basis and an $\ell^2$ penalty on the coefficients. The sieve makes the coefficient update a closed-form ridge regression for any fixed direction, and the direction is updated by stochastic gradient descent. The product form is what converts the dimensionality problem into a sequence of low-dimensional projections, and the theorem tracks how the per-iteration sieve bias, the propagation of previous errors through the products, and the variance from finite samples combine.
What would settle it
Fix one dataset, run Algorithm 1 many times from different random initializations of the direction, and record the returned $\hat{\boldsymbol a}_k$ and the resulting uniform error. If the estimated directions vary across runs while the data, sieve size, and penalty are held fixed, then the algorithm is not producing the global minimizer assumed by Theorem 4.2, and the stated rates cannot be attributed to the implemented procedure; alternatively, estimates of the ratio on a known dense-grid target that violate the theorem's rate while the assumptions on smoothness and eigenvalues hold would also settle the claim.
Extended reading notes
Core claim
The central claim is that the multiplicative projection pursuit estimator $\hat r_K(x)=\prod_{k=1}^K \hat f_k(\hat{\boldsymbol a}_k^\top x)$, built by iteratively minimizing a ridge-penalized squared loss, is consistent and attains the rates in Theorem 4.2. At iteration $k$, given the previous estimate $\hat r_{k-1}$, the paper optimizes over a direction $\boldsymbol a$ and the coefficients of a univariate linear sieve for $f$, with the coefficients available in closed form once the direction is fixed. The theorem states that the uniform error of each fitted ridge function and the error of the estimated direction both converge at rates driven by the sieve approximation order, the accumulated error from previous iterations, and the sample size. Consequently, the paper claims that a hard full-dimensional density ratio estimation problem can be converted into a sequence of one-dimensional curve-fitting problems without losing consistency.
Load-bearing premise
The convergence rates in Theorem 4.2 assume that, at each step, the estimator finds a global minimizer of a nonconvex empirical loss, but Algorithm 1 searches for the direction with stochastic gradient descent from random starting points and includes no proof that the returned local optimum is global.
Editorial extensions
If this is right
- Density ratio estimation becomes a sequence of one-dimensional fits, so the practical cost scales with the number of projections $K$ and the sieve size $J_k$ rather than with the raw dimension $d$.
- Because the estimator makes no parametric assumption about the ratio, it keeps the flexibility of nonparametric methods while inheriting a quantified bias-variance tradeoff from the sieve rate $J^{-(s-1)}$ and the variance $\sqrt{J/(n_p \wedge n_q)}$.
- The same estimated ratio can be plugged into any downstream procedure that uses importance weights, and the experiments in the paper demonstrate gains in continuous treatment effect estimation, mutual information estimation, and covariate shift adaptation.
- The theory supplies guidance for tuning: $K$ and $J_k$ can be selected by validation loss with the understanding that too few projections or too few basis functions show up directly in the bias term.
Reading between the lines
- If the SGD-computed directions actually behave like the global minimizers assumed in Theorem 4.2, the same construction could serve as a low-cost importance-weighting module in high-dimensional causal inference and domain adaptation pipelines.
- The paper lists independence testing as future work; the multiplicative form makes this concrete, since mutual information estimated through $\hat r_K$ gives a test statistic whose null distribution would have to be derived.
- A practical diagnostic suggested by the theorem is to run the algorithm from many random direction initializations: if the returned directions are stable across seeds, the global-minimization assumption is plausible for that dataset, and if not, the stated rates should not be expected to hold.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes projection pursuit density ratio estimation (ppDRE), in which the density ratio r*(x) is approximated by a product of univariate ridge functions, r_K(x) = prod_{k=1}^K f_k(a_k^T x). At each iteration, f_k is estimated from a regularized squared loss using a univariate linear sieve basis, with a closed-form ridge update for the sieve coefficients when the direction a is fixed (Proposition 4.1). The main theoretical result, Theorem 4.2, states sup-norm convergence rates for the estimated sieve function and direction at each iteration, and a final rate for the product estimator r_hat_K relative to r_K, under regularity assumptions F.1--F.4. The method is implemented in Algorithm 1 with Gaussian basis functions whose locations gamma_j are also optimized by SGD, and hyperparameters are selected by cross-validation. Experiments cover stabilized weights estimation for continuous treatments, dose-response function estimation on IHDP-continuous, mutual information estimation, and covariate shift adaptation, reporting favorable performance against several baselines.
Significance. If the theoretical claims can be matched to the implemented method, the paper would make a useful contribution: it provides a projection-pursuit alternative to high-dimensional density ratio estimation with explicit rates, and the numerical study is broad and mostly careful. The paper ships a substantial proof appendix, states regularity conditions explicitly, and does not appear to derive rates from fitted values, so the core theoretical program is not circular. The main concern is that the estimator analyzed in Theorem 4.2 (fixed linear sieve, global argmin) differs structurally from the estimator implemented and evaluated in Algorithm 1 (adaptive Gaussian centers, SGD), and the final rate is stated only for the projection pursuit approximation r_K rather than for the target r*, so the headline consistency claim is not yet established for the reported method.
major comments (3)
- [Section 4.2, Remark 4.3, Algorithm 1, and Theorem 4.2] The theory covers the linear-sieve estimator (3) in which Phi_k(z) is a pre-specified univariate basis and only (a, beta) are optimized, but the implemented Algorithm 1 (lines 6 and 9) and Remark 4.3 jointly fit the Gaussian basis locations gamma_j together with a and beta by stochastic gradient descent. With trainable centers, the hypothesis class is the nonlinear set {sum_j beta_j exp(-(z-gamma_j)^2/2)}, not the linear sieve F_{J_k}. Consequently, the closed-form beta update in Proposition 4.1 is not the minimizer of the joint loss, the first-order condition (34) used in the proof of (9) does not hold with respect to gamma, and the sieve approximation bound (Assumption F.2(a)), the eigenvalue condition (Assumption F.3), and the metric-entropy bound (Lemma F.5) all refer to a fixed basis. Since all reported experiments use Algorithm 1, the central consistency and rate claims do not apply to the method whose performance is presented. The authors should either implement and analyze a fixed-basis version with centers chosen by a pre-specified grid or cross-validation, or extend the theory to a genuinely nonlinear basis with adaptive centers.
- [Theorem 4.2 and Algorithm 1 (SGD)] Theorem 4.2 assumes that (a_hat_k, beta_hat_k) is the global argmin of the empirical loss in (3), and the proof of the direction rate uses the global first-order condition (33)--(34). Algorithm 1, however, replaces this argmin by a stochastic gradient descent solution with random initialization and no global optimality guarantee. Because the loss is nonconvex in the direction (and, in the implementation, in the Gaussian centers), a locally optimal or merely stationary point returned by SGD is not covered by the stated rates. The authors need to add explicit conditions under which SGD reaches the global argmin, or prove rates for stationary points, or clearly reposition the theoretical result as holding for a different (e.g., profile-gridded) optimizer than the one used in the experiments.
- [Theorem 4.2, final display] The final statement of Theorem 4.2 bounds sup_{x in X} |r_hat_K(x) - r_K(x)|, where r_K is the best multiplicative projection pursuit approximation, but the abstract and introduction claim consistency and a convergence rate for estimating the density ratio r*. No bound is given for the approximation error sup_x |r_K(x) - r*(x)|, nor are conditions stated under which this error vanishes as K grows. Without such a bound, the results only show that the estimator tracks its finite-K population target, not that it estimates r*. The authors should either add an approximation-error theorem for the multiplicative projection pursuit class, or restrict the consistency claim to the projection pursuit approximation r_K.
minor comments (5)
- [Algorithm 1, line 6] The definition of W_k uses Phi_k(a_k^{(t-1)T} x_i^p; gamma^{(t-1)}) but the loop variable is a^{(t-1)}; this appears to be a typographical inconsistency that should be corrected.
- [Proposition 4.1 and Section 4.2] In the definition of W_k(a) in Proposition 4.1, the basis is evaluated at a_k^T x_i^p, whereas the loss (4) and the surrounding text use the free direction a; the notation should be made uniform.
- [Theorem 4.2, notation] The display for the final rate uses sqrt(tilde zeta_1(J_i)) or zeta_0^2(J_i) with a max operator, while the explanation below the display defines max{tilde zeta_1(J_k), zeta_0(J_k)} without the square; the notation should be aligned.
- [Table 2] The mutual information results are averaged over only three replicates and the standard errors are large for some methods; a statement about the stability of these comparisons would strengthen the empirical claims.
- [Appendix C.1] The RMSLE definition uses log(r(x)), but the estimator can be negative before truncation; the paper does mention truncation in Section 4.2, but it would be clearer to state explicitly that the reported RMSLE is computed on the truncated nonnegative estimate.
Circularity Check
No circular derivation: Theorem 4.2 rates follow from explicit sieve and smoothness assumptions; the Algorithm 1 / theory mismatch is a scope gap, not a circular reduction.
full rationale
The paper's derivation chain is not circular. The population objective H in (2) is obtained by expanding E_q[(r^* - r_{k-1} f)^2] and using r^* = p/q, which is a direct algebraic identity rather than an assumed conclusion. Proposition 4.1 is an exact closed-form solution of a convex quadratic in beta for fixed a. Theorem 4.2 and Appendix F prove rates conditional on standard assumptions: F.2(a) sieve approximation, F.3 eigenvalue and basis bounds, and F.4 positive-definite Hessian. These assumptions do not contain the conclusion; the proof then bounds sampling error via metric entropy and empirical process arguments (Lemma F.5 and maximal inequalities). No fitted parameter is renamed as a prediction, and the hyperparameters selected by cross-validation in Remark 4.3 are treated as deterministic sequences in the asymptotic statements. The self-citations (Ai et al. 2021, Stojanov et al. 2019, Gao et al. 2023) are used for application frameworks or data-generation protocols, not as load-bearing uniqueness theorems. The most serious concern is non-circular: Algorithm 1 jointly optimizes Gaussian RBF centers gamma while Theorem 4.2 analyzes the fixed linear sieve estimator in (3), so the implemented estimator is not covered by the stated rates even under global optimization; likewise Theorem 4.2 bounds |hat r_K - r_K| rather than |hat r_K - r^*|, leaving the projective pursuit approximation error to r^* unquantified. These are correctness and completeness risks, not equivalences of outputs to inputs, so they do not raise the circularity score.
Assumptions & free parameters
free parameters (5)
- K =
5 to 15 via CV
- J_k =
20 to 150 via CV
- lambda =
0.5 to 10 via CV
- delta =
0.001 to 0.1 via CV
- gamma_j =
data-dependent (optimized)
assumptions (6)
- domain assumption q dominates p: p(x) > 0 implies q(x) > 0
- domain assumption Support X and parameter space A are compact (Assumption F.1)
- domain assumption f_{a,k} is s-times continuously differentiable, bounded and bounded away from zero (Assumption F.2)
- standard math Sieve basis eigenvalues bounded away from zero and infinity (Assumption F.3)
- domain assumption Hessian of the sieve approximation is positive definite at the minimum (Assumption F.4)
- standard math Diaconis-Shahshahani ridge approximation theorem (1984)
Cite this review
Pith. "Pith review of Projection Pursuit Density Ratio Estimation." pith.science (2026). https://pith.science/paper/DYH2AAS4
@misc{pith2026250600866,
author = {Pith},
title = {Pith review of: Projection Pursuit Density Ratio Estimation},
year = {2026},
howpublished = {\url{https://pith.science/paper/DYH2AAS4}},
note = {Machine review of arXiv:2506.00866}
}
read the original abstract
Density ratio estimation (DRE) is a paramount task in machine learning, for its broad applications across multiple domains, such as covariate shift adaptation, causal inference, independence tests and beyond. Parametric methods for estimating the density ratio possibly lead to biased results if models are misspecified, while conventional non-parametric methods suffer from the curse of dimensionality when the dimension of data is large. To address these challenges, in this paper, we propose a novel approach for DRE based on the projection pursuit (PP) approximation. The proposed method leverages PP to mitigate the impact of high dimensionality while retaining the model flexibility needed for the accuracy of DRE. We establish the consistency and the convergence rate for the proposed estimator. Experimental results demonstrate that our proposed method outperforms existing alternatives in various applications.
Figures
Reference graph
Works this paper leans on
-
[1]
1 𝑛 𝑛∑︁ 𝑖=1 | ˆ𝑟𝑘−1(𝒙𝑖)−𝑟𝑘−1(𝒙𝑖)| 2=𝑂𝑝(𝜉𝑛,𝑘−1). (16) We shall show that, for any𝑘∈{ 1,2,...,𝐾 }, given (16) holds, we have sup 𝒙∈X | ˆ𝑓 ˆ𝒂𝑘,𝑘( ˆ𝒂⊤ 𝑘 𝒙)− 𝑓𝒂𝑘,𝑘(𝒂⊤ 𝑘 𝒙)| =𝑂𝑃 {𝐽−(𝑠−1) 𝑘 + √︁ 𝜉𝑛,𝑘−1+ √︃ 𝐽𝑘/(𝑛𝑝∧𝑛𝑞)}·{ √︃ ˜𝜁1(𝐽𝑘)∨ 𝜁 2 0(𝐽𝑘)} , (17) where ˜𝜁1(𝐽𝑘) is the rate of the maximum eigenvalue ofΩ(1) 𝐽𝑘 (𝒂) and √︁ ˜𝜁1(𝐽𝑘)∨𝜁0(𝐽𝑘)= max{ √︁ ˜𝜁1(𝐽𝑘),𝜁 0(𝐽𝑘)}....
work page 1996
-
[5]
Search Grid for Probabilistic Classification approach and nnDRE Method Parameter Search Space Classification n_estimators {100, 300} learning_rate {0.0001, 0.001, 0.01} num_leaves {20, 30} nnDRE depth {2, 3} width {8, 32, 64} learning_rate {0.0001, 0.001, 0.01} For fDRE, we have implemented a version that employs KLIEP as the second-stage DRE method. Adhe...
work page 2021
-
[6]
Hyperparameter setting for the normalizing flow model in fDRE Dataset n_blocks n_hidden hidden_size n_epochs IHDP 5 1 100 100 Regression Benchmarks 5 1 100 100 MI Gaussians 5 1 100 200 For RRND, in the absence of open-source code, we implemented the algorithm by adhering to the implementation details outlined in the Numerical Illustrations section in Nguy...
work page 2024
-
[8]
Computation Time Comparison Method 𝑑= 2 𝑑= 10 𝑑= 30 𝑑= 50 𝑑= 100 uLSIF 0.38 0.35 0.38 0.43 0.46 KLIEP 0.36 0.39 0.49 0.57 0.72 Classification 0.21 0.16 0.19 0.23 0.40 fDRE 3.78 3.88 3.92 4.28 7.25 RRND 3.78 3.87 3.77 3.82 3.81 nnDRE 5.91 3.84 8.76 6.02 8.63 𝐷3-LHSS 5.95 11.25 10.60 10.79 10.97 ppDRE 0.32 0.91 5.64 10.78 15.22 method consistently outperfor...
work page 2020
-
[9]
Abalone age is determined by cutting the shell, staining it, and counting the number of growth rings
Observation and Feature Numbers of Regression Benchmark Datasets Dataset Abalone Billboard Spotify Cancer Mortality Computer Activity Diamond Prices Observation 4177 8930 3047 8192 6000 Feature 8 18 31 21 7 Abalone The Abalone dataset, accessible online4, is a popular dataset used in machine learning and statistics to predict the age of abalone from physi...
work page 1961
-
[10]
(a) The eigenvalues ofΩ(0) 𝐽𝑘 (𝒂) are bounded and bounded away from zero uniformly in𝐽𝑘 and 𝒂∈A
Assumption F.3. (a) The eigenvalues ofΩ(0) 𝐽𝑘 (𝒂) are bounded and bounded away from zero uniformly in𝐽𝑘 and 𝒂∈A . (b) There exist sequences of constants ˜𝜁𝑗(𝐽𝑘), 𝑗= 1,2, such that the eigenvalues ofΩ(𝑗) 𝐽𝑘 (𝒂) are bounded by ˜𝜁𝑗(𝐽𝑘) uniformly in𝐽𝑘 and 𝒂∈A (c) There exist sequences of constants 𝜁𝑗(𝐽𝑘), 𝑗= 0,1,2, such that sup𝑗≤𝑚 sup𝑧∈Z∥𝚽(𝑗) 𝑘 (𝑧)∥≤ 𝜁𝑚(𝐽𝑘) ...
work page 1986
-
[12]
Finally, under Assumption F.3, we have (48)=𝑂𝑝( √︁ 𝐽𝑘/𝑛𝑞)
Consequently, we have ∥(47)∥=𝑂𝑝( √𝐽𝑘 √𝑛𝑝∧𝑛𝑞 ). Finally, under Assumption F.3, we have (48)=𝑂𝑝( √︁ 𝐽𝑘/𝑛𝑞). Thus, we obtain the result. □ Lemma F.9. For any𝑘∈{ 1,2,...,𝐾 }, suppose Assumptions F .1 – F .4 and(16) hold. For any 𝒂∈A and 𝑗= 0,1, sup 𝒂∈A sup 𝒙∈X 𝑓(𝑗) 𝒂,𝑘(𝒂⊤𝒙)− 𝑓∗(𝑗) 𝒂,𝑘 (𝒂⊤𝒙) =𝑂 𝐽−(𝑠−𝑗) 𝑘 𝜁𝑗(𝐽𝑘) , E h |𝑓(𝑗) 𝒂,𝑘(𝒂⊤𝒙)− 𝑓∗(𝑗) 𝒂,𝑘 (𝒂⊤𝒙)| 2 i =𝑂 𝐽−2...
work page 1986
-
[2009]
doi: 10.1109/ISIT.2009.5205712. 11 Projection Pursuit Density Ratio Estimation Tsuboi, Y ., Kashima, H., Hido, S., Bickel, S., and Sugiyama, M. Direct density ratio estimation for large-scale covari- ate shift adaptation. Journal of Information Processing, 17:138–155,
Show all 12 references
-
[2012]
Mutual informa- tion approximation via maximum likelihood estimation of density ratio
Suzuki, T., Sugiyama, M., and Tanaka, T. Mutual informa- tion approximation via maximum likelihood estimation of density ratio. In 2009 IEEE International Sympo- sium on Information Theory , pp. 463–467,
2009
-
[2016]
Computer Activity The Computer Activity databases consist of a collection of metrics related to the activity of computer systems
It incorporates 2013 census data, including county-level features such as population, income, households, and other demographic attributes. Computer Activity The Computer Activity databases consist of a collection of metrics related to the activity of computer systems. The dat...
2013
-
[2022]
For each song, the corresponding audio features were sourced from the Spotify API. The regression task for the Billboard Spotify dataset is to predict the danceability of a song, which describes how suitable a track is for dancing based on a combination of musical elements inc...
2010
-
[2025]
Baseline Methods This section presents the baseline methods used in our numerical studies
12 Projection Pursuit Density Ratio Estimation A. Baseline Methods This section presents the baseline methods used in our numerical studies. uLSIF The unconstrained Least-Squares Importance Fitting (uLSIF) method, proposed by Kanamori et al. (2009), models the density ratio𝑟∗(...
2009
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.