Pith. sign in

REVIEW 4 major objections 6 minor 9 references

Semi-knockoffs let any pre-trained ML model test conditional independence in finite samples, without a train-test split.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:14 UTC pith:M5QFQ7LY

load-bearing objection Genuinely new two-sided resampling trick with a clean oracle FDR proof, but the practical algorithm's finite-sample guarantees rest on an openly stated conjecture and the abstract overclaims. the 4 major comments →

arxiv 2601.23124 v2 pith:M5QFQ7LY submitted 2026-01-30 math.ST stat.TH

Semi-knockoffs: a model-agnostic conditional independence testing method with finite-sample guarantees

Angel Reyero-Lobo , Bertrand Thirion , Pierre Neuvial This is my paper
classification math.ST stat.TH MSC 62G1062F0362J07
keywords conditional independence testingknockoffsmodel-agnosticfalse discovery ratevariable selectiondouble robustnessfinite-sample guaranteesmachine learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Conditional independence testing asks whether a feature matters beyond the others, but model-agnostic tests currently pay a power penalty by splitting the data to evaluate a trained model. Semi-knockoffs removes that split: it samples the feature of interest twice, once conditioned on the other features and once conditioned on the other features plus the response, then compares the pre-trained model's loss on the two synthetic inputs. Under the null hypothesis the two sampling distributions coincide, so the loss difference is symmetric about zero — a property that yields exact finite-sample p-values with a Wilcoxon or sign test and false discovery rate control with the standard knockoff threshold. The method needs only the conditional expectations E[X_j|X_{-j}] and E[X_j|X_{-j}, y], which are estimated from data, not the full model-X knowledge of the input distribution. The paper proves the oracle guarantees and provides stability, distributional convergence, and double-robustness results for the estimated version.

Core claim

On its own terms, the paper's claim is that conditional independence can be tested with an arbitrary black-box predictive model while retaining finite-sample type-I error and FDR control and no train-test split, by the asymmetric construction of two synthetic populations: one mixes the feature with residuals from a regression on the remaining variables, the other from a regression on the remaining variables together with the response. Under the null, ρ_j(X_{-j}, y) = ν_j(X_{-j}), so the two populations are exchangeable and the loss difference l(m(X̃1), y) − l(m(X̃2), y) is symmetric; this symmetry is exactly what makes valid p-values and knockoff-style FDR control possible. For the practical

What carries the argument

Two conditional-expectation imputers — ν_j(X_{-j}) = E[X_j|X_{-j}] and ρ_j(X_{-j}, y) = E[X_j|X_{-j}, y] — generate the synthetic features X̃1 and X̃2 whose losses are compared. The identity ρ_j = ν_j under the null turns the loss difference into an exchangeable statistic whose sign, in the oracle version, is provably a Rademacher random variable, entitling the knockoff threshold T_q to control the FDR. Two supporting results carry the estimated version: a stability theorem (Theorem 4.1) showing an l2-regularized empirical risk minimizer changes by O_P(sqrt(log(1/δ)/n)) when an uninformative coordinate is suppressed, and a double-robustness theorem (Theorem 4.3) showing the loss discrepancy

Load-bearing premise

The practical version's finite-sample guarantees rest on the unproved conjecture that each estimated statistic's sign is a fair coin flip independent of the other features; the paper proves the oracle case and only bounds the error magnitude, not the sign, in the estimated case.

What would settle it

Simulate data where a null feature is strongly correlated with a signal feature, use a deliberately overfit or small-sample model so its derivative with respect to the null feature is not small, and estimate ν and ρ with regularized linear imputers; then check whether the signs of the Semi-knockoff statistics across many null features are i.i.d. Rademacher. If the sign distribution is biased or dependent in this regime, or the empirical FDR exceeds the target q substantially, the conjecture and the practical guarantee fail.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Variable selection with complex models — random forests, gradient boosting, neural networks — becomes feasible in high dimensions with finite-sample FDR control, without splitting the sample.
  • Scientists no longer need the model-X assumption of a known input distribution; estimating two conditional expectations suffices, which widens the class of accessible problems.
  • The method sidesteps the difficult construction of exact knockoff variables and antisymmetric statistics, removing a major practical bottleneck of knockoffs.
  • Because the model is used only as an importance scorer, any pre-trained model can be reused directly, saving the cost of refitting per feature.
  • Empirically, a few extra permutations (derandomization) substantially raise power while preserving error control.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the sign-exchangeability conjecture fails in regimes where the model keeps nontrivial sensitivity to null features or the imputers are too noisy relative to the signal, the practical FDR guarantee would degrade gracefully to asymptotic rather than finite-sample control; the paper's own limits motivate a diagnostic.
  • The two-imputer trick is a general symmetry-engine: any statistic symmetric in X̃1 and X̃2 under the null, not just loss differences, could carry the same argument, suggesting immediate extension to rank-based or kernel-based scores.
  • The double-robustness product structure suggests a practical safeguard: monitor a_n·b_n per feature and flag cases where the product is not small, treating them as outside the verified regime.
  • Because the imputers condition on the response, the method connects naturally to missing-data imputation practice, where conditioning on y is known to matter; transferring those insights could raise power further.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Semi-knockoffs, a method for conditional independence testing with arbitrary pre-trained ML models that avoids a train-test split. The central idea is to compare the loss of a model evaluated on two perturbed versions of each feature: one sampled using the conditional expectation of X_j given X_{-j} (ν_j), and one using the conditional expectation given X_{-j} and y (ρ_j). Under the null hypothesis X_j ⊥ y | X_{-j}, both samplers coincide, producing exchangeable loss distributions; under the alternative, the second sampler is less perturbing. The authors prove oracle type-I error control (Theorem 3.3) and oracle FDR control (Theorem 3.4) via knockoff exchangeability. For the practical version with estimated imputers (Algorithm 4), they provide a stability theorem for regularized ERMs (Theorem 4.1), a Wasserstein convergence result (Theorem 4.2), and a double-robustness rate bound (Theorem 4.3). The finite-sample guarantees for the estimated version, however, rest on an explicit conjecture in Section 4.5 that the sign of the empirical statistic matches the oracle sign with probability tending to one.

Significance. The oracle version of Semi-knockoffs is an elegant and potentially useful construction. The idea of comparing two conditional samplers (with and without y) to eliminate the train-test split is novel, and the oracle FDR proof (Appendix E.2) is a correct application of Barber–Candès exchangeability, giving finite-sample FDR control when the conditional expectations are known. The stability and double-robustness results are of independent interest, and the experiments are extensive and well documented. However, the advertised contribution—finite-sample guarantees for a practical method that works with arbitrary pre-trained models and estimated imputers—is not established. The load-bearing sign-preservation conjecture is explicitly left unproved, and Theorem 3.3's proof appears incomplete for the actual paired-test implementation. If the oracle results are separated from the estimated-version claims and the practical claims are weakened to asymptotic or conditional guarantees, the paper could be a useful contribution.

major comments (4)
  1. [§3.1, Theorem 3.3; Appendix E.1] The proof of Theorem 3.3 only shows that under the null the two populations have the same marginal law. Algorithm 2 returns the standard Wilcoxon or sign-test p-value, whose validity requires the paired differences to be independent (or the test statistic to have the usual randomization null). Here the residual permutations π1 and π2 are global permutations without replacement, so the pairs (l(bm(eX1,i), y_i), l(bm(eX2,i), y_i)) are dependent across i. Marginal symmetry of each difference does not imply that the signed-rank or sign statistic has the standard null distribution. Consequently, finite-sample type-I error control for the oracle version is not established as stated. Please either prove exactness under the dependent permutation structure or modify the algorithm (e.g., independent residual resampling with replacement) and re-derive the test.
  2. [§4.5 and Algorithm 4; abstract] The abstract and Section 1 claim finite-sample FDR control for the practical Semi-knockoffs (Algorithm 4), but Theorem 3.4 is proved only for oracle νj and ρj. For estimated imputers, the route to Lemma 2.1 is the conjecture in Section 4.5 that the sign of the empirical statistic equals the oracle sign with probability tending to 1. Even if this sign-matching held asymptotically, it would give at best asymptotic exchangeability, not the exact conditional Rademacher signs required for the finite-sample bound FDR(Ŝ^SKO) ≤ q. Theorems 4.2 and 4.3 are magnitude/rate bounds (W1 convergence and O_P(an bn) for loss differences) and do not control the sign of the average difference, nor do they address joint sign-flip invariance across features. Please either prove a finite-sample sign-preservation result under explicit assumptions, or weaken the practical claims to asymptotic FDR control.
  3. [§4.2, Theorem 4.1; Appendix E.3.2] The proof of Theorem 4.1 applies Prop. 5.2 of Reyero-Lobo et al. (2025b) to assert that the population coefficient of the null input (y for ρj) is zero. Under H0: X_j ⊥⊥ y | X_{-j}, the coefficient of y in the best linear predictor of X_j on (X_{-j}, y) is not zero in general when E[X_j | X_{-j}] is nonlinear; conditional independence implies zero partial correlation only if the model is correctly specified. Thus Theorem 4.1 appears to require correct specification of the imputer (e.g., linear/Gaussian), a condition not stated in the theorem and not guaranteed for arbitrary practical imputers. In addition, the proof's final bound includes a √p factor from Eq. (15), so the statement O_P(√(log(1/δ)/n)) in Eq. (3) omits the dimension dependence.
  4. [§4.4, Theorem 4.3] The statement says l(bm(eX'), y) − l(bm(eX), y) = O_P(an bn), with an = ∂_{x_j} bm(X_{-j}, t) for a t between eX'^j and eX^j. Since t is an unspecified intermediate point from the mean value theorem, an is not a well-defined deterministic or simple random sequence, so the O_P statement is ambiguous. Please define an as a supremum over the interval or state the bound uniformly. This matters because Theorem 4.3 is the formal basis for the sign-preservation conjecture in §4.5.
minor comments (6)
  1. [§4.1] "the FDR procedure is provided in Appendix 4" should read "provided in Algorithm 4".
  2. [Notation] Use consistent notation for eX'^{(j)} vs eX'^j and W_j^SKO vs W_j SKO; the current mixing is confusing.
  3. [Abstract] The abstract states "provides valid p-values and FDR control" without qualification; this overclaims because the practical estimated version relies on the unproved conjecture in §4.5. Please qualify the guarantees as oracle or asymptotic.
  4. [Appendix F.1, Table 2] The table lists "Semi KO Ours FDR" and "Semi KO ST/ Wilcox Ours Type-I error"; the main text uses "Semi-knockoffs". Please harmonize the naming.
  5. [Figure 3 and §4.5] The caption says the blue distribution is the semi-knockoff statistic and the orange is the difference between theoretical and estimated imputers; Figure 2 and the surrounding text refer to similar quantities. Clarify the exact pair of random variables in each histogram to avoid ambiguity.
  6. [References] The paper relies on a companion preprint (Reyero-Lobo et al., 2025b) for a key proposition used in Theorem 4.1. If this proposition is not published, include its statement and assumptions in the appendix or state the dependence explicitly in the main text.

Circularity Check

1 steps flagged

Practical finite-sample FDR control rests on the Section 4.5 sign-preservation conjecture, an ansatz inherited from the authors' own double-robustness work; the oracle theorems are self-contained.

specific steps
  1. ansatz smuggled in via citation [Section 4.5 ('Applications of double robustness to Semi-knockoffs'), with Section 4.1 and Theorem 4.3]
    "We conjecture that this is the case, based on a double robustness argument. Indeed, we expect that the sign of the empirical difference matches the sign of the theoretical one, yielding the desired independent Rademacher variable."

    The abstract and contribution bullets advertise finite-sample FDR control for the practical Semi-knockoffs (Algorithm 4). Such control requires the null statistics Ŵ_j^SKO to have exactly i.i.d. Rademacher signs conditional on their magnitudes (Lemma 2.1). For the estimated imputers bν and bρ, the only stated route to this property is the Section 4.5 conjecture. Theorem 4.3 bounds the loss difference only in magnitude (O_P(a_n b_n)), and Theorem 4.2 bounds W1 distance; neither establishes exact sign exchangeability. Thus the advertised practical guarantee is not derived from the proved theorems—it is assumed as an ansatz motivated by the authors' own double-robustness preprint (Reyero-Lobo et al., 2025a). The claim of finite-sample FDR therefore reduces, by construction, to an unproven co

full rationale

The oracle results are not circular: Theorem 3.3 and Theorem 3.4 are proved directly from conditional-exchangeability of the two samplers under the null, exactly as in the standard knockoff argument, and they do not depend on the authors' prior work. The stability and double-robustness theorems (4.1 and 4.3) are also genuine derivations given their assumptions, although Theorem 4.1 uses a zero-coefficient fact cited from the authors' companion preprint (Prop. 5.2 of Reyero-Lobo et al. 2025b); that citation is not the source of the oracle guarantees and is a standard regression fact, so it is not circular by itself. The single significant circularity-adjacent step is the practical FDR claim: the paper explicitly labels the required sign-preservation property for estimated imputers as a conjecture, then still presents the method as providing finite-sample FDR control. Since the conjecture is the missing condition needed to apply Lemma 2.1 and Theorem 3.4 to Algorithm 4, the practical guarantee is not an independent derivation but an assumed premise. I therefore set the circularity score at 3: the oracle core is self-contained, but the headline practical claim leans on an ansatz from the authors' own self-cited double-robustness framework.

Axiom & Free-Parameter Ledger

3 free parameters · 7 axioms · 0 invented entities

The oracle theorems are built on standard CIT/knockoff assumptions plus existence of conditional means. The estimated-version results add regularity assumptions on loss, design, and model, and depend on a proposition from the authors' own companion preprint. The sign-preservation conjecture is the main unproven input for the practical guarantees.

free parameters (3)
  • ridge regularization λ in imputer regressions
    Theoretical results (Theorem 4.1) require λ > 0 for strong convexity; in practice chosen as default ridge. It affects the stability bound, not the null validity.
  • number of derandomization permutations K = 5
    Section 5: 'we apply 5 permutations (SKO Wcx p5)'; chosen empirically, gives a modest power gain.
  • loss function ℓ for the importance statistic
    User-specified; theory requires C2 or Lipschitz conditions. The choice affects power and the meaning of 'importance'.
axioms (7)
  • domain assumption The conditional expectations ν_j(X_{-j}) = E[X_j|X_{-j}] and ρ_j(X_{-j},y) = E[X_j|X_{-j},y] exist and are estimable for each continuous feature j.
    Assumed in Section 3; Remark 3.2 extends to categorical features via logistic regression. Additivity of the residual construction is not required for validity under the null, only for power.
  • standard math No covariate is exactly a function of the others, so the Markov blanket is unique.
    Inherited from Candès et al. 2018 and Verdinelli & Wasserman 2024; standard identification condition for CIT (Section 1).
  • domain assumption Assumptions E.1–E.4: design χ and score s* are sub-Gaussian, loss is L-Lipschitz in the prediction argument, regularizer is λ-strongly convex, and the population minimizer satisfies E[∂_u l(μ(χ),z)|χ] = 0.
    Verified for quadratic and cross-entropy losses (Lemma E.5); restricts the imputers to l2-regularized ERM, not the predictive model bm.
  • domain assumption For Theorem 4.2: model bm and loss are Lipschitz in the prediction argument.
    Required to convert parameter-level stability into W1 distributional control (Appendix E.4).
  • domain assumption For Theorem 4.3: loss t→l(t,y) is C2 with bounded derivatives, bm is differentiable in the j-th coordinate, and bν−ν = O_P(b_n).
    Double-robustness bound; paper notes strict differentiability is not necessary in practice (e.g., random forests in Section 4.4).
  • standard math Prop. 5.2 of Reyero-Lobo et al. (2025b): under X_j ⊥ z | X_{-j}, the population coefficient β_j of the regression of z on X is 0.
    Used in Appendix E.3.2 to justify the null-feature stability theorem; cited from the authors' companion preprint rather than proven here.
  • ad hoc to paper Sign-preservation conjecture: the sign of the empirical Semi-knockoff statistic matches the oracle sign with probability tending to 1.
    Not proven; explicitly labeled a conjecture in Section 4.5. It is load-bearing for finite-sample type-I and FDR guarantees with estimated imputers.

pith-pipeline@v1.3.0-alltime-deepseek · 33688 in / 27345 out tokens · 300974 ms · 2026-08-03T06:14:32.561471+00:00 · methodology

0 comments
read the original abstract

Conditional independence testing (CIT) is essential for reliable scientific discovery. It prevents spurious findings and enables controlled feature selection. Recent CIT methods have used machine learning (ML) models as surrogates of the underlying distribution. However, model-agnostic approaches require a train-test split, which reduces statistical power. We introduce Semi-knockoffs, a CIT method that can accommodate any pre-trained model, avoids this split, and provides valid p-values and false discovery rate (FDR) control for high-dimensional settings. Unlike methods that rely on the model-$X$ assumption (known input distribution), Semi-knockoffs only require conditional expectations for continuous variables. This makes the procedure less restrictive and more practical for machine learning integration. To ensure validity when estimating these expectations, we present two new theoretical results of independent interest: (i) stability for regularized models trained with a null feature and (ii) the double-robustness property.

Figures

Figures reproduced from arXiv: 2601.23124 by Angel Reyero-Lobo, Bertrand Thirion, Pierre Neuvial.

Figure 1
Figure 1. Figure 1: Optimization stability. Data are generated from z = χβ + ϵ, where β is 0.25-sparse with important features grouped in blocks of 5 sampled uniformly. We set n = 300, p = 50, noise level at ∥χβ∥/2 and χ ∼ N (0, Σ) with Σi,j = 0.6 |i−j| . The y-axis shows the difference between the model coefficients with and without the x-axis coordinate. 4.3. Distribution control The validity of the method relies on the fac… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of the conjecture that P  l(mb (Xe′ 1), y) > l(mb (Xe′ 2), y) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Type-I error with adjacent support. Let X ∼ N (0, Σ) with Σ ij = 0.6 |i−j| , and y = β ⊤X + ϵ, where the first 0.25p coordinates of β lie in [1, 2] and the remaining are zero, with ϵ ∼ N (0, 1). The black-box pretrained model mb is a gradient boosting, achieving R 2 = 0.886 with a train-test split and R 2 = 0.91 without split. 0.0 0.2 0.4 0.6 0.8 Power SCPI(1)_sqrt SCPI(1)_Wcx SCPI(100)_sqrt LOCO-W LOCO_sq… view at source ↗
Figure 5
Figure 5. Figure 5: Type-I error with masked correlation. Let X ∼ N (0, Σ) with Σij = 0.6 |i−j| . A unique relevant coordinate l is sampled and y = Xl + 0.5 ϵ1, where ϵ1 ∼ N (0, 1). A correlated null variable is created as Xl−1 = Xl+0.5 ϵ2, with ϵ2 ∼ N (0, 1). The black-box pretrained model mb is a neural network, achieving R 2 = 0.567 with a train-test split and R 2 = 0.581 without split. modate arbitrary models. In contrast… view at source ↗
Figure 7
Figure 7. Figure 7: Double Robustness: Distribution of the semi-knockoff statistic, i.e., at two independently sampled estimated residuals (blue: l(mb (Xe′ 1), y) − l(mb (Xe′ 2), y)), and distribution of the difference between the theoretical and estimated residuals (orange: l(mb (Xe′ 1), y) − l(mb (Xe1), y)) for a null coordinate (j = 0). The data-generating model is y = 0.8X 1 + 0.6X 2 + 0.4X 3 + 0.2X 4 + sin(X 1 ) + ϵ, wit… view at source ↗
Figure 8
Figure 8. Figure 8: Double Robustness: Distribution of the semi-knockoff statistic, i.e., the difference in loss evaluated at two independently sampled estimated residuals (blue: l(mb (Xe′ 1), y) − l(mb (Xe′ 2), y)), and distribution of the difference between the theoretical and estimated residuals (orange: l(mb (Xe′ 1), y) − l(mb (Xe1), y)) for a null coordinate (j = 0). The data-generating model is y = 0.8X 1 + 0.6X 2 + 0.4… view at source ↗
Figure 9
Figure 9. Figure 9: Double Robustness: Distribution of the semi-knockoff statistic, i.e., the difference in loss evaluated at two independently sampled estimated residuals (blue: l(mb (Xe′ 1), y) − l(mb (Xe′ 2), y)), and distribution of the difference between the theoretical and estimated residuals (orange: l(mb (Xe′ 1), y) − l(mb (Xe1), y)) for a null coordinate (j = 0). The data-generating model is y = 0.8X 1 + 0.6X 2 + 0.4… view at source ↗
Figure 10
Figure 10. Figure 10: Double Robustness: Distribution of the semi-knockoff statistic, i.e., the difference in loss evaluated at two independently sampled estimated residuals (blue: l(mb (Xe′ 1), y) − l(mb (Xe′ 2), y)), and distribution of the difference between the theoretical and estimated residuals (orange: l(mb (Xe′ 1), y) − l(mb (Xe1), y)) for a null coordinate (j = 0). The data-generating model is y = X ⊤β + ϵ, with ϵ ∼ N… view at source ↗
Figure 11
Figure 11. Figure 11: Double Robustness: Distribution of the semi-knockoff statistic, i.e., the difference in loss evaluated at two independently sampled estimated residuals (blue: l(mb (Xe′ 1), y) − l(mb (Xe′ 2), y)), and distribution of the difference between the theoretical and estimated residuals (orange: l(mb (Xe′ 1), y) − l(mb (Xe1), y)) for a null coordinate (j = 0). The data-generating model is y = X ⊤β + ϵ, with ϵ ∼ N… view at source ↗
Figure 12
Figure 12. Figure 12: Semi-knockoff statistics: y = Xβ + ϵ, where β is 0.25-sparse with important features grouped in blocks of 5 sampled uniformly. The noise level is ∥Xβ∥/2. The design matrix satisfies X ∼ N (0, Σ) with Σi,j = 0.6 |i−j| . We set n = 300 and p = 50. The figure displays the distribution of the Semi-knockoff statistics for several standard ML models. In each plot, the null features are shown on the left and the… view at source ↗
Figure 13
Figure 13. Figure 13: Type-I error adjacent support with Neural Network. Let X ∼ N (0, Σ) with Σ ij = 0.6 |i−j| , and y = β ⊤X + ϵ, where the first 0.25p coordinates of β are between 1 and 2 and the remaining entries are zero, and ϵ ∼ N (0, 1). The black-box pretrained model mb is a neural network. power. F.4.2. MASKED CORRELATION This setting consists of a single important coordinate that is highly correlated with an unimport… view at source ↗
Figure 14
Figure 14. Figure 14: Type-I error adjacent support with Gradient Boosting. Let X ∼ N (0, Σ) with Σ ij = 0.6 |i−j| , and y = β ⊤X + ϵ, where the first 0.25p coordinates of β are between 1 and 2 and the remaining entries are zero, and ϵ ∼ N (0, 1). The black-box pretrained model mb is a gradient boosting. 10 1 10 2 10 3 Computation Time (s) Sobol-CPI(1)_sqrt Sobol-CPI(1)_Wilcox Sobol-CPI(100)_sqrt LOCO-W LOCO_sqrt LOCO_Wilcox d… view at source ↗
Figure 15
Figure 15. Figure 15: Type-I error adjacent support with Random Forest. Let X ∼ N (0, Σ) with Σ ij = 0.6 |i−j| , and y = β ⊤X + ϵ, where the first 0.25p coordinates of β are between 1 and 2 and the remaining entries are zero, and ϵ ∼ N (0, 1). The black-box pretrained model mb is a random forest. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Type-I error masked correlation with Neural Network. Let X ∼ N (0, Σ) with Σij = 0.6 |i−j| . A unique relevant coordinate l is sampled and y = Xl + 0.5 ϵ1, where ϵ1 ∼ N (0, 1). A correlated null variable is created as Xl−1 = Xl + 0.5 ϵ2, with ϵ2 ∼ N (0, 1). The black-box pretrained model mb is a neural network. 10 1 10 2 Computation Time (s) Sobol-CPI(1)_sqrt Sobol-CPI(1)_Wilcox Sobol-CPI(100)_sqrt LOCO-W… view at source ↗
Figure 17
Figure 17. Figure 17: Type-I error masked correlation with Gradient Boosting. Let X ∼ N (0, Σ) with Σij = 0.6 |i−j| . A unique relevant coordinate l is sampled and y = Xl + 0.5 ϵ1, where ϵ1 ∼ N (0, 1). A correlated null variable is created as Xl−1 = Xl + 0.5 ϵ2, with ϵ2 ∼ N (0, 1). The black-box pretrained model mb is a gradient boosting. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Type-I error masked correlation with Random Forest. Let X ∼ N (0, Σ) with Σij = 0.6 |i−j| . A unique relevant coordinate l is sampled and y = Xl + 0.5 ϵ1, where ϵ1 ∼ N (0, 1). A correlated null variable is created as Xl−1 = Xl + 0.5 ϵ2, with ϵ2 ∼ N (0, 1). The black-box pretrained model mb is a random forest. ( [PITH_FULL_IMAGE:figures/full_fig_p028_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: FDR adjacent support with Neural Network. Let X ∼ N (0, Σ) with Σ ij = 0.6 |i−j| , and y = β ⊤X + ϵ, where the first 0.25p coordinates of β are between 1 and 2 and the remaining entries are zero, and ϵ ∼ N (0, 1). The black-box pretrained model mb is a neural network, achieving R 2 = 0.964 with a train-test split and R 2 = 0.965 without split. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: FDR adjacent support with Gradient Boosting. Let X ∼ N (0, Σ) with Σ ij = 0.6 |i−j| , and y = β ⊤X + ϵ, where the first 0.25p coordinates of β are between 1 and 2 and the remaining entries are zero, and ϵ ∼ N (0, 1). The black-box pretrained model mb is a gradient boosting, achieving R 2 = 0.884 with a train-test split and R 2 = 0.910 without split. 10 1 10 0 10 1 10 2 10 3 Time (s) Knockoff dCRT HRT Semi… view at source ↗
Figure 21
Figure 21. Figure 21: FDR adjacent support with Random Forest. Let X ∼ N (0, Σ) with Σ ij = 0.6 |i−j| , and y = β ⊤X + ϵ, where the first 0.25p coordinates of β are between 1 and 2 and the remaining entries are zero, and ϵ ∼ N (0, 1). The black-box pretrained model mb is a random forest, achieving R 2 = 0.796 with a train-test split and R 2 = 0.824 without split. 10 0 10 1 10 2 Time (s) Knockoff dCRT HRT Semi_KO Semi_KO_perm5 … view at source ↗
Figure 22
Figure 22. Figure 22: FDR heavy-tailed with Neural Networks. Let X = Σ1/2Z, where Z has i.i.d. t-Student entries Zij ∼ t3 and Σij = 0.6 |i−j| . We generate y = β ⊤X + ϵ, where the first 0.25p coordinates of β lie in [1, 2] and the remaining entries are zero, with ϵ ∼ N (0, 1). The black-box pretrained model mb is a neural network, achieving R 2 = 0.798 with a train-test split and R 2 = 0.828 without split. 29 [PITH_FULL_IMAGE… view at source ↗
Figure 23
Figure 23. Figure 23: FDR heavy-tailed with Gradient Boosting. Let X = Σ1/2Z, where Z has i.i.d. t-Student entries Zij ∼ t3 and Σij = 0.6 |i−j| . We generate y = β ⊤X + ϵ, where the first 0.25p coordinates of β lie in [1, 2] and the remaining entries are zero, with ϵ ∼ N (0, 1). The black-box pretrained model mb is a gradient boosting, achieving R 2 = 0.688 with a train-test split and R 2 = 0.737 without split. 10 1 10 0 10 1 … view at source ↗
Figure 24
Figure 24. Figure 24: FDR heavy-tailed with Random Forest. Let X = Σ1/2Z, where Z has i.i.d. t-Student entries Zij ∼ t3 and Σij = 0.6 |i−j| . We generate y = β ⊤X + ϵ, where the first 0.25p coordinates of β lie in [1, 2] and the remaining entries are zero, with ϵ ∼ N (0, 1). The black-box pretrained model mb is a random forest, achieving R 2 = 0.601 with a train-test split and R 2 = 0.641 without split. 30 [PITH_FULL_IMAGE:fi… view at source ↗
Figure 25
Figure 25. Figure 25: Discoveries on real data using a Random Forest. Left: computation time; middle: number of discoveries; right: type-I error, estimated using an artificial null feature correlated at 0.6 with the original features. Semi-knockoffs makes discoveries while controlling the error. F.6. Real data experiments The Wisconsin Diagnostic Breast Cancer (WDBC) dataset consists of n = 569 patient samples with p = 30 nume… view at source ↗
Figure 26
Figure 26. Figure 26: Discoveries on real data using a Neural Network. Left: computation time; middle: number of discoveries; right: type-I error, estimated using an artificial null feature correlated at 0.6 with the original features. Semi-knockoffs makes discoveries while controlling the error. 10 1 10 2 Computation Time (s) Sobol-CPI(1)_sqrt Sobol-CPI(1)_Wilcox Sobol-CPI(100)_sqrt LOCO-W LOCO_sqrt LOCO_Wilcox dCRT HRT Semi_… view at source ↗
Figure 27
Figure 27. Figure 27: Discoveries on real data using a Gradient Boosting. Left: computation time; middle: number of discoveries; right: type-I error, estimated using an artificial null feature correlated at 0.6 with the original features. Semi-knockoffs makes discoveries while controlling the error. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_27.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

9 extracted references · 1 linked inside Pith

  1. [1]

    , p}, the original and the knockoff variables are exchangeable, i.e

    For any subset s⊂ {1, . . . , p}, the original and the knockoff variables are exchangeable, i.e. (X, eX) swap(s) d = (X, eX)

  2. [2]

    We observe that the second property can be obtained by constructing the knockoffs without using the output

    eX⊥ ⊥y|X. We observe that the second property can be obtained by constructing the knockoffs without using the output. Moreover, these imitation variables must satisfy the exchangeability condition that is stronger than merely sampling from the conditional distribution, as it must hold over the joint distribution rather than feature-wise. Consequently, com...

    2019

  3. [8]

    It is a function of the inputX, the knockoff eXand the outputy:W j =w j h X, eX i , y

  4. [9]

    This statistic provides a hint of how important the feature is

    It satisfies the flip-sign property: fors⊂[p], wj h X, eX i swap(s) , y =    wj h X, eX i , y ifj ∈s, −wj h X, eX i , y ifj∈s. This statistic provides a hint of how important the feature is. The most intuitive example is given by the Lasso Coefficient Difference (LCD), which, to assign importance, compares the coefficients of a Lasso regression of y on...

    2018

  5. [1995]

    doi: 10.1111/j.2517-6161.1995

    ISSN 0035-9246. doi: 10.1111/j.2517-6161.1995. tb02031.x. URL https://doi.org/10.1111/j. 2517-6161.1995.tb02031.x. Blain, A., Lobo, A. R., Linhart, J., Thirion, B., and Neu- vial, P. When knockoffs fail: diagnosing and fixing non-exchangeability of knockoffs, 2025. URL https: //arxiv.org/abs/2407.06892. Bottou, L. and Bousquet, O. The tradeoffs of large s...

    arXiv 1995

  6. [2007]

    why” behind including “y

    URL https://proceedings.neurips. cc/paper_files/paper/2007/file/ 0d3180d672e08b4c5312dcdafdf6ef36-Paper. pdf. Bousquet, O. and Elisseeff, A. Stability and generalization. Journal of machine learning research, 2(Mar):499–526, 2002. Breiman, L. Random forests.Machine Learning, 45 (1):5–32, 2001. ISSN 1573-0565. doi: 10.1023/ A:1010933404324. URL https://doi...

    Pith/arXiv arXiv 2007

  7. [2020]

    URL https://proceedings.mlr.press/ v119/nguyen20a.html. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V ., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V ., Vanderplas, J., Passos, A., Cour- napeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in Python.Journal of Machine Learning Resea...

    arXiv 2011

  8. [2023]

    URL https://proceedings.mlr.press/ v206/shaer23a.html. Shah, R. and Peters, J. The hardness of conditional in- dependence testing and the generalised covariance mea- sure.Annals of Statistics, 48, 04 2018. doi: 10.1214/ 19-AOS1857. Strobl, C., Boulesteix, A.-L., Kneib, T., Augustin, T., and Zeileis, A. Conditional variable importance for random forests.BM...

    2018

  9. [2105]

    URL https: //doi.org/10.1186/1471-2105-9-307

    doi: 10.1186/1471-2105-9-307. URL https: //doi.org/10.1186/1471-2105-9-307. Tansey, W., Veitch, V ., Zhang, H., Rabadan, R., and and, D. M. B. The holdout randomization test for feature selection in black box models.Journal of Computational and Graphical Statistics, 31(1):151–162, 2022. doi: 10. 1080/10618600.2021.1923520. URL https://doi. org/10.1080/106...

    arXiv 2022