Pith. sign in

REVIEW 3 major objections 5 minor 12 references

Projection Pursuit Density Ratio Estimation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Projection pursuit turns high-dimensional density ratio estimation into a chain of one-dimensional curve fits.

desk verdict Genuinely new multiplicative projection pursuit idea for DRE, but the theory covers the linear-sieve estimator while the implemented algorithm also fits Gaussian locations, so the rates do not actually cover the method being reported. read the letter →

arxiv 2506.00866 v1 pith:DYH2AAS4 submitted 2025-06-01 stat.ML cs.LGstat.ME

classification stat.MLcs.LGstat.ME MSC 62G0762G20
keywords densityratioestimationprojectionpursuitridgefunctionslinearsievecurseofdimensionalityconvergenceratecovariateshiftmutualinformation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes estimating the density ratio $r^*(x)=p(x)/q(x)$ as a product of $K$ univariate functions of one-dimensional projections of $x$, and fits these functions one by one with a regularized squared loss. It claims that the estimator is consistent and supplies an explicit convergence rate (Theorem 4.2), so the error in estimating the ratio can be driven to zero as samples and basis functions grow. This matters because classical linear sieve density-ratio estimators are flexible but degrade sharply as dimension grows, while neural-network alternatives lack comparable guarantees. The proposal takes a middle path: no parametric model assumption, a solution that stays computationally simple, and a theory that quantifies the tradeoff between approximation bias and variance.

What carries the argument

The load-bearing object is the multiplicative projection pursuit approximation $r_K(x)=\prod_{k=1}^K f_k(\boldsymbol a_k^\top x)$, where each $f_k$ is a univariate ridge function (a one-dimensional function of the projection $\boldsymbol a_k^\top x$). The fitting step minimizes $H(f,\boldsymbol a)=\mathbb E_q[r_{k-1}^2 f^2(\boldsymbol a^\top x)]-2\mathbb E_p[r_{k-1} f(\boldsymbol a^\top x)]$, which is the nonconstant part of the $L^2$ distance to the target ratio; in practice this is replaced by its regularized empirical version with a univariate linear sieve basis and an $\ell^2$ penalty on the coefficients. The sieve makes the coefficient update a closed-form ridge regression for any fixed direction, and the direction is updated by stochastic gradient descent. The product form is what converts the dimensionality problem into a sequence of low-dimensional projections, and the theorem tracks how the per-iteration sieve bias, the propagation of previous errors through the products, and the variance from finite samples combine.

What would settle it

Fix one dataset, run Algorithm 1 many times from different random initializations of the direction, and record the returned $\hat{\boldsymbol a}_k$ and the resulting uniform error. If the estimated directions vary across runs while the data, sieve size, and penalty are held fixed, then the algorithm is not producing the global minimizer assumed by Theorem 4.2, and the stated rates cannot be attributed to the implemented procedure; alternatively, estimates of the ratio on a known dense-grid target that violate the theorem's rate while the assumptions on smoothness and eigenvalues hold would also settle the claim.

Watch

Extended reading notes

Core claim

The central claim is that the multiplicative projection pursuit estimator $\hat r_K(x)=\prod_{k=1}^K \hat f_k(\hat{\boldsymbol a}_k^\top x)$, built by iteratively minimizing a ridge-penalized squared loss, is consistent and attains the rates in Theorem 4.2. At iteration $k$, given the previous estimate $\hat r_{k-1}$, the paper optimizes over a direction $\boldsymbol a$ and the coefficients of a univariate linear sieve for $f$, with the coefficients available in closed form once the direction is fixed. The theorem states that the uniform error of each fitted ridge function and the error of the estimated direction both converge at rates driven by the sieve approximation order, the accumulated error from previous iterations, and the sample size. Consequently, the paper claims that a hard full-dimensional density ratio estimation problem can be converted into a sequence of one-dimensional curve-fitting problems without losing consistency.

Load-bearing premise

The convergence rates in Theorem 4.2 assume that, at each step, the estimator finds a global minimizer of a nonconvex empirical loss, but Algorithm 1 searches for the direction with stochastic gradient descent from random starting points and includes no proof that the returned local optimum is global.

Editorial extensions

If this is right

  • Density ratio estimation becomes a sequence of one-dimensional fits, so the practical cost scales with the number of projections $K$ and the sieve size $J_k$ rather than with the raw dimension $d$.
  • Because the estimator makes no parametric assumption about the ratio, it keeps the flexibility of nonparametric methods while inheriting a quantified bias-variance tradeoff from the sieve rate $J^{-(s-1)}$ and the variance $\sqrt{J/(n_p \wedge n_q)}$.
  • The same estimated ratio can be plugged into any downstream procedure that uses importance weights, and the experiments in the paper demonstrate gains in continuous treatment effect estimation, mutual information estimation, and covariate shift adaptation.
  • The theory supplies guidance for tuning: $K$ and $J_k$ can be selected by validation loss with the understanding that too few projections or too few basis functions show up directly in the bias term.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the SGD-computed directions actually behave like the global minimizers assumed in Theorem 4.2, the same construction could serve as a low-cost importance-weighting module in high-dimensional causal inference and domain adaptation pipelines.
  • The paper lists independence testing as future work; the multiplicative form makes this concrete, since mutual information estimated through $\hat r_K$ gives a test statistic whose null distribution would have to be derived.
  • A practical diagnostic suggested by the theorem is to run the algorithm from many random direction initializations: if the returned directions are stable across seeds, the global-minimization assumption is plausible for that dataset, and if not, the stated rates should not be expected to hold.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes projection pursuit density ratio estimation (ppDRE), in which the density ratio r*(x) is approximated by a product of univariate ridge functions, r_K(x) = prod_{k=1}^K f_k(a_k^T x). At each iteration, f_k is estimated from a regularized squared loss using a univariate linear sieve basis, with a closed-form ridge update for the sieve coefficients when the direction a is fixed (Proposition 4.1). The main theoretical result, Theorem 4.2, states sup-norm convergence rates for the estimated sieve function and direction at each iteration, and a final rate for the product estimator r_hat_K relative to r_K, under regularity assumptions F.1--F.4. The method is implemented in Algorithm 1 with Gaussian basis functions whose locations gamma_j are also optimized by SGD, and hyperparameters are selected by cross-validation. Experiments cover stabilized weights estimation for continuous treatments, dose-response function estimation on IHDP-continuous, mutual information estimation, and covariate shift adaptation, reporting favorable performance against several baselines.

Significance. If the theoretical claims can be matched to the implemented method, the paper would make a useful contribution: it provides a projection-pursuit alternative to high-dimensional density ratio estimation with explicit rates, and the numerical study is broad and mostly careful. The paper ships a substantial proof appendix, states regularity conditions explicitly, and does not appear to derive rates from fitted values, so the core theoretical program is not circular. The main concern is that the estimator analyzed in Theorem 4.2 (fixed linear sieve, global argmin) differs structurally from the estimator implemented and evaluated in Algorithm 1 (adaptive Gaussian centers, SGD), and the final rate is stated only for the projection pursuit approximation r_K rather than for the target r*, so the headline consistency claim is not yet established for the reported method.

major comments (3)
  1. [Section 4.2, Remark 4.3, Algorithm 1, and Theorem 4.2] The theory covers the linear-sieve estimator (3) in which Phi_k(z) is a pre-specified univariate basis and only (a, beta) are optimized, but the implemented Algorithm 1 (lines 6 and 9) and Remark 4.3 jointly fit the Gaussian basis locations gamma_j together with a and beta by stochastic gradient descent. With trainable centers, the hypothesis class is the nonlinear set {sum_j beta_j exp(-(z-gamma_j)^2/2)}, not the linear sieve F_{J_k}. Consequently, the closed-form beta update in Proposition 4.1 is not the minimizer of the joint loss, the first-order condition (34) used in the proof of (9) does not hold with respect to gamma, and the sieve approximation bound (Assumption F.2(a)), the eigenvalue condition (Assumption F.3), and the metric-entropy bound (Lemma F.5) all refer to a fixed basis. Since all reported experiments use Algorithm 1, the central consistency and rate claims do not apply to the method whose performance is presented. The authors should either implement and analyze a fixed-basis version with centers chosen by a pre-specified grid or cross-validation, or extend the theory to a genuinely nonlinear basis with adaptive centers.
  2. [Theorem 4.2 and Algorithm 1 (SGD)] Theorem 4.2 assumes that (a_hat_k, beta_hat_k) is the global argmin of the empirical loss in (3), and the proof of the direction rate uses the global first-order condition (33)--(34). Algorithm 1, however, replaces this argmin by a stochastic gradient descent solution with random initialization and no global optimality guarantee. Because the loss is nonconvex in the direction (and, in the implementation, in the Gaussian centers), a locally optimal or merely stationary point returned by SGD is not covered by the stated rates. The authors need to add explicit conditions under which SGD reaches the global argmin, or prove rates for stationary points, or clearly reposition the theoretical result as holding for a different (e.g., profile-gridded) optimizer than the one used in the experiments.
  3. [Theorem 4.2, final display] The final statement of Theorem 4.2 bounds sup_{x in X} |r_hat_K(x) - r_K(x)|, where r_K is the best multiplicative projection pursuit approximation, but the abstract and introduction claim consistency and a convergence rate for estimating the density ratio r*. No bound is given for the approximation error sup_x |r_K(x) - r*(x)|, nor are conditions stated under which this error vanishes as K grows. Without such a bound, the results only show that the estimator tracks its finite-K population target, not that it estimates r*. The authors should either add an approximation-error theorem for the multiplicative projection pursuit class, or restrict the consistency claim to the projection pursuit approximation r_K.
minor comments (5)
  1. [Algorithm 1, line 6] The definition of W_k uses Phi_k(a_k^{(t-1)T} x_i^p; gamma^{(t-1)}) but the loop variable is a^{(t-1)}; this appears to be a typographical inconsistency that should be corrected.
  2. [Proposition 4.1 and Section 4.2] In the definition of W_k(a) in Proposition 4.1, the basis is evaluated at a_k^T x_i^p, whereas the loss (4) and the surrounding text use the free direction a; the notation should be made uniform.
  3. [Theorem 4.2, notation] The display for the final rate uses sqrt(tilde zeta_1(J_i)) or zeta_0^2(J_i) with a max operator, while the explanation below the display defines max{tilde zeta_1(J_k), zeta_0(J_k)} without the square; the notation should be aligned.
  4. [Table 2] The mutual information results are averaged over only three replicates and the standard errors are large for some methods; a statement about the stability of these comparisons would strengthen the empirical claims.
  5. [Appendix C.1] The RMSLE definition uses log(r(x)), but the estimator can be negative before truncation; the paper does mention truncation in Section 4.2, but it would be clearer to state explicitly that the reported RMSLE is computed on the truncated nonnegative estimate.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: Theorem 4.2 rates follow from explicit sieve and smoothness assumptions; the Algorithm 1 / theory mismatch is a scope gap, not a circular reduction.

full rationale

The paper's derivation chain is not circular. The population objective H in (2) is obtained by expanding E_q[(r^* - r_{k-1} f)^2] and using r^* = p/q, which is a direct algebraic identity rather than an assumed conclusion. Proposition 4.1 is an exact closed-form solution of a convex quadratic in beta for fixed a. Theorem 4.2 and Appendix F prove rates conditional on standard assumptions: F.2(a) sieve approximation, F.3 eigenvalue and basis bounds, and F.4 positive-definite Hessian. These assumptions do not contain the conclusion; the proof then bounds sampling error via metric entropy and empirical process arguments (Lemma F.5 and maximal inequalities). No fitted parameter is renamed as a prediction, and the hyperparameters selected by cross-validation in Remark 4.3 are treated as deterministic sequences in the asymptotic statements. The self-citations (Ai et al. 2021, Stojanov et al. 2019, Gao et al. 2023) are used for application frameworks or data-generation protocols, not as load-bearing uniqueness theorems. The most serious concern is non-circular: Algorithm 1 jointly optimizes Gaussian RBF centers gamma while Theorem 4.2 analyzes the fixed linear sieve estimator in (3), so the implemented estimator is not covered by the stated rates even under global optimization; likewise Theorem 4.2 bounds |hat r_K - r_K| rather than |hat r_K - r^*|, leaving the projective pursuit approximation error to r^* unquantified. These are correctness and completeness risks, not equivalences of outputs to inputs, so they do not raise the circularity score.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The theoretical result rests on standard smoothness, compactness, and sieve conditions, plus the background ridge-approximation theorem. The practical algorithm adds data-dependent Gaussian basis locations that the theory does not cover, which is a gap rather than a circularity.

free parameters (5)
  • K = 5 to 15 via CV
    Number of projection iterations; selected by cross-validation. The theory treats K as fixed and does not require a specific value.
  • J_k = 20 to 150 via CV
    Number of Gaussian sieve basis functions; selected by CV. In theory J_k grows with sample size.
  • lambda = 0.5 to 10 via CV
    Ridge penalty strength; selected by CV.
  • delta = 0.001 to 0.1 via CV
    SGD learning rate; selected by CV.
  • gamma_j = data-dependent (optimized)
    Gaussian basis centers; optimized jointly with beta and a in Algorithm 1, but the convergence theory assumes a fixed basis.
assumptions (6)
  • domain assumption q dominates p: p(x) > 0 implies q(x) > 0
    Stated in Section 3 to make the density ratio well-defined.
  • domain assumption Support X and parameter space A are compact (Assumption F.1)
    Used in the proof for uniform convergence.
  • domain assumption f_{a,k} is s-times continuously differentiable, bounded and bounded away from zero (Assumption F.2)
    Needed for sieve approximation rate.
  • standard math Sieve basis eigenvalues bounded away from zero and infinity (Assumption F.3)
    Standard sieve regression condition to avoid multicollinearity.
  • domain assumption Hessian of the sieve approximation is positive definite at the minimum (Assumption F.4)
    Ensures the projection direction is identifiable.
  • standard math Diaconis-Shahshahani ridge approximation theorem (1984)
    Invoked in Section 1 to justify that the product of ridge functions can approximate the density ratio as K grows.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Projection Pursuit Density Ratio Estimation." pith.science (2026). https://pith.science/paper/DYH2AAS4

@misc{pith2026250600866,
  author       = {Pith},
  title        = {Pith review of: Projection Pursuit Density Ratio Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DYH2AAS4}},
  note         = {Machine review of arXiv:2506.00866}
}
read the original abstract

Density ratio estimation (DRE) is a paramount task in machine learning, for its broad applications across multiple domains, such as covariate shift adaptation, causal inference, independence tests and beyond. Parametric methods for estimating the density ratio possibly lead to biased results if models are misspecified, while conventional non-parametric methods suffer from the curse of dimensionality when the dimension of data is large. To address these challenges, in this paper, we propose a novel approach for DRE based on the projection pursuit (PP) approximation. The proposed method leverages PP to mitigate the impact of high dimensionality while retaining the model flexibility needed for the accuracy of DRE. We establish the consistency and the convergence rate for the proposed estimator. Experimental results demonstrate that our proposed method outperforms existing alternatives in various applications.

Figures

Figures reproduced from arXiv: 2506.00866 by the authors.

Figure 1
Figure 1. 2-D DRE Experiment. The warmer color represents a higher estimated density ratio value. The top-left plot in each panel shows the true density ratio, while the remaining plots illustrate the estimates using various methods. 5. Experiments and Applications In this section, we compare our proposed projection pur￾suit density ratio estimation (ppDRE) method with existing alternatives using experimental and real-world d… view at source ↗
Figure 3
Figure 3. Average RMSLE over 50 replicates for stabilized weights estimation with varying dimensions of covariates. the 𝜏th quantile dose-response function (QDRF) 𝑔(𝑡; 𝜽 ∗ ) = 𝐹 −1 𝑌∗ (𝑡) (𝜏) = inf{𝑞 : P{𝑌 ∗ (𝑡) ≥ 𝑞} ≤ 𝜏}. Note that, by definition, the stabilized weights 𝜋0 (𝑡, 𝒙) can be viewed as a density ratio. We compare our method with different estimators of 𝜋0 (𝑡, 𝒙) based on simulated data sets in Section 5.2.1. In Se… view at source ↗
Figure 4
Figure 4. Boxplot of RMSLE in stabilized weights estimation for different covariate dimension settings 𝑑𝑋 ∈ {2,10,30,50,100} estimation across 50 data replications [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

  1. [1]

    1 𝑛 𝑛∑︁ 𝑖=1 | ˆ𝑟𝑘−1(𝒙𝑖)−𝑟𝑘−1(𝒙𝑖)| 2=𝑂𝑝(𝜉𝑛,𝑘−1). (16) We shall show that, for any𝑘∈{ 1,2,...,𝐾 }, given (16) holds, we have sup 𝒙∈X | ˆ𝑓 ˆ𝒂𝑘,𝑘( ˆ𝒂⊤ 𝑘 𝒙)− 𝑓𝒂𝑘,𝑘(𝒂⊤ 𝑘 𝒙)| =𝑂𝑃 {𝐽−(𝑠−1) 𝑘 + √︁ 𝜉𝑛,𝑘−1+ √︃ 𝐽𝑘/(𝑛𝑝∧𝑛𝑞)}·{ √︃ ˜𝜁1(𝐽𝑘)∨ 𝜁 2 0(𝐽𝑘)} , (17) where ˜𝜁1(𝐽𝑘) is the rate of the maximum eigenvalue ofΩ(1) 𝐽𝑘 (𝒂) and √︁ ˜𝜁1(𝐽𝑘)∨𝜁0(𝐽𝑘)= max{ √︁ ˜𝜁1(𝐽𝑘),𝜁 0(𝐽𝑘)}....

  2. [5]

    Search Grid for Probabilistic Classification approach and nnDRE Method Parameter Search Space Classification n_estimators {100, 300} learning_rate {0.0001, 0.001, 0.01} num_leaves {20, 30} nnDRE depth {2, 3} width {8, 32, 64} learning_rate {0.0001, 0.001, 0.01} For fDRE, we have implemented a version that employs KLIEP as the second-stage DRE method. Adhe...

  3. [6]

    Hyperparameter setting for the normalizing flow model in fDRE Dataset n_blocks n_hidden hidden_size n_epochs IHDP 5 1 100 100 Regression Benchmarks 5 1 100 100 MI Gaussians 5 1 100 200 For RRND, in the absence of open-source code, we implemented the algorithm by adhering to the implementation details outlined in the Numerical Illustrations section in Nguy...

  4. [8]

    The observed increase in computation time is expected, as higher dimensions naturally require more iterations to reach convergence

    Computation Time Comparison Method 𝑑= 2 𝑑= 10 𝑑= 30 𝑑= 50 𝑑= 100 uLSIF 0.38 0.35 0.38 0.43 0.46 KLIEP 0.36 0.39 0.49 0.57 0.72 Classification 0.21 0.16 0.19 0.23 0.40 fDRE 3.78 3.88 3.92 4.28 7.25 RRND 3.78 3.87 3.77 3.82 3.81 nnDRE 5.91 3.84 8.76 6.02 8.63 𝐷3-LHSS 5.95 11.25 10.60 10.79 10.97 ppDRE 0.32 0.91 5.64 10.78 15.22 method consistently outperfor...

  5. [9]

    Abalone age is determined by cutting the shell, staining it, and counting the number of growth rings

    Observation and Feature Numbers of Regression Benchmark Datasets Dataset Abalone Billboard Spotify Cancer Mortality Computer Activity Diamond Prices Observation 4177 8930 3047 8192 6000 Feature 8 18 31 21 7 Abalone The Abalone dataset, accessible online4, is a popular dataset used in machine learning and statistics to predict the age of abalone from physi...

  6. [10]

    (a) The eigenvalues ofΩ(0) 𝐽𝑘 (𝒂) are bounded and bounded away from zero uniformly in𝐽𝑘 and 𝒂∈A

    Assumption F.3. (a) The eigenvalues ofΩ(0) 𝐽𝑘 (𝒂) are bounded and bounded away from zero uniformly in𝐽𝑘 and 𝒂∈A . (b) There exist sequences of constants ˜𝜁𝑗(𝐽𝑘), 𝑗= 1,2, such that the eigenvalues ofΩ(𝑗) 𝐽𝑘 (𝒂) are bounded by ˜𝜁𝑗(𝐽𝑘) uniformly in𝐽𝑘 and 𝒂∈A (c) There exist sequences of constants 𝜁𝑗(𝐽𝑘), 𝑗= 0,1,2, such that sup𝑗≤𝑚 sup𝑧∈Z∥𝚽(𝑗) 𝑘 (𝑧)∥≤ 𝜁𝑚(𝐽𝑘) ...

  7. [12]

    Finally, under Assumption F.3, we have (48)=𝑂𝑝( √︁ 𝐽𝑘/𝑛𝑞)

    Consequently, we have ∥(47)∥=𝑂𝑝( √𝐽𝑘 √𝑛𝑝∧𝑛𝑞 ). Finally, under Assumption F.3, we have (48)=𝑂𝑝( √︁ 𝐽𝑘/𝑛𝑞). Thus, we obtain the result. □ Lemma F.9. For any𝑘∈{ 1,2,...,𝐾 }, suppose Assumptions F .1 – F .4 and(16) hold. For any 𝒂∈A and 𝑗= 0,1, sup 𝒂∈A sup 𝒙∈X 𝑓(𝑗) 𝒂,𝑘(𝒂⊤𝒙)− 𝑓∗(𝑗) 𝒂,𝑘 (𝒂⊤𝒙) =𝑂 𝐽−(𝑠−𝑗) 𝑘 𝜁𝑗(𝐽𝑘) , E h |𝑓(𝑗) 𝒂,𝑘(𝒂⊤𝒙)− 𝑓∗(𝑗) 𝒂,𝑘 (𝒂⊤𝒙)| 2 i =𝑂 𝐽−2...

  8. [2009]

    11 Projection Pursuit Density Ratio Estimation Tsuboi, Y ., Kashima, H., Hido, S., Bickel, S., and Sugiyama, M

    doi: 10.1109/ISIT.2009.5205712. 11 Projection Pursuit Density Ratio Estimation Tsuboi, Y ., Kashima, H., Hido, S., Bickel, S., and Sugiyama, M. Direct density ratio estimation for large-scale covari- ate shift adaptation. Journal of Information Processing, 17:138–155,

Show all 12 references
  1. [2012]

    Mutual informa- tion approximation via maximum likelihood estimation of density ratio

    Suzuki, T., Sugiyama, M., and Tanaka, T. Mutual informa- tion approximation via maximum likelihood estimation of density ratio. In 2009 IEEE International Sympo- sium on Information Theory , pp. 463–467,

  2. [2016]

    Computer Activity The Computer Activity databases consist of a collection of metrics related to the activity of computer systems

    It incorporates 2013 census data, including county-level features such as population, income, households, and other demographic attributes. Computer Activity The Computer Activity databases consist of a collection of metrics related to the activity of computer systems. The dat...

  3. [2022]

    For each song, the corresponding audio features were sourced from the Spotify API. The regression task for the Billboard Spotify dataset is to predict the danceability of a song, which describes how suitable a track is for dancing based on a combination of musical elements inc...

  4. [2025]

    Baseline Methods This section presents the baseline methods used in our numerical studies

    12 Projection Pursuit Density Ratio Estimation A. Baseline Methods This section presents the baseline methods used in our numerical studies. uLSIF The unconstrained Least-Squares Importance Fitting (uLSIF) method, proposed by Kanamori et al. (2009), models the density ratio𝑟∗(...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.