Pith. sign in

REVIEW 2 major objections 2 minor 29 references

Counterfactual edits that optimize a deep two-sample test's discrepancy measure increase the resulting p-values.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A method that produces minimal edits to source samples via diffusion autoencoder and MMD optimization, raising p-values to show statistical closeness to a target group in deep two-sample testing.

T0 review reviewed 2026-06-28 challenge →

load-bearing objection The paper adds counterfactual edits to deep two-sample tests via MMD optimization in a frozen test model's space, but p-value gains follow directly from the objective. the 2 major comments →

arxiv 2606.04009 v2 pith:EPT37QVY submitted 2026-05-29 stat.ML cs.AIcs.LG

Counterfactual Explanations for Deep Two-Sample Testing

Wei-Cheng Lai , Marco Simnacher , Christoph Lippert This is my paper
classification stat.ML cs.AIcs.LG
keywords counterfactual explanationsdeep two-sample testingmaximum mean discrepancydiffusion autoencoderMRI cohortsimage datadistributional differencesp-value changes
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper develops a method to explain deep two-sample tests by creating edited versions of source samples that move them toward a target group while lowering the test's measured discrepancy. It uses a diffusion autoencoder to generate plausible changes and optimizes the maximum mean discrepancy objective inside the space learned by a pretrained test model. On both synthetic shape data and real MRI scans, these edits reliably raise the p-values, showing the source set is now closer to the target under the test. This matters because it turns an opaque rejection of the null hypothesis into specific, minimal changes in the data that reveal which features the test is using to detect differences.

Core claim

By optimizing an MMD objective in the representation space of a pretrained deep two-sample test model, the diffusion-autoencoder-based edits transform source observations so that the two-sample test statistic decreases and the associated p-values increase, indicating reduced distributional discrepancy while keeping changes minimal as measured by LPIPS.

What carries the argument

Diffusion autoencoder guided by MMD optimization in the test model's representation space, which generates sample-level edits that reduce the test discrepancy.

Load-bearing premise

Minimizing the MMD objective in the representation space produces edits that reflect genuine distributional features instead of model artifacts or optimization side effects.

What would settle it

Observing that the p-values do not increase or that they decrease after applying the counterfactual edits on the same datasets would falsify the claim that the transformations make the edited source closer to the target under the test.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • The counterfactual transformations increase p-values relative to original samples.
  • The edited source set becomes statistically closer to the target distribution under the test.
  • Localized changes on MRI data are consistent with known anatomical differences between cohorts.
  • LPIPS distance ensures the edits remain close to the originals.
  • The method provides interpretable evidence of features driving the group differences detected by the test.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the test model is accurate, these edits could validate that the detected differences correspond to real features rather than spurious correlations.
  • Applying the same approach to other data types like text or time series might reveal domain-specific distinguishing features.
  • Future work could test whether human experts agree that the generated changes match their understanding of the group differences.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper proposes a counterfactual explanation method for deep two-sample tests: a diffusion autoencoder is combined with a pretrained test model to optimize an MMD objective directly in the model's representation space, producing sample-level edits from source toward target that raise p-values (and thus reduce the test statistic). The approach is evaluated on synthetic 2D shape datasets and two MRI cohorts, with LPIPS used to enforce minimality of edits; the resulting changes are claimed to yield interpretable evidence of the features driving distributional differences.

Significance. If the edits can be shown to capture genuine distributional features rather than test-model artifacts, the framework would address a clear gap in interpretability for deep two-sample tests on high-dimensional structured data. The evaluation on both synthetic and real MRI data, together with explicit minimality checks, is a positive step toward practical utility.

major comments (2)
  1. [Abstract] Abstract (method paragraph): the optimization directly minimizes the MMD statistic of the pretrained test model in its own representation space; any successful optimization will therefore reduce that statistic and raise p-values by construction. The interpretive claim that the edits reflect 'genuine distributional features' therefore rests on the unverified assumption that the test model's representations are faithful to the true data-generating process, yet no independent validation (different test architecture, held-out training of the test model, or non-learned baseline) is described.
  2. [Abstract] Abstract (evaluation paragraph): the reported consistency of p-value increases is presented without quantitative tables, error bars, ablation studies, or comparison against alternative explanation methods; this makes it difficult to assess whether the observed effect size is robust or merely an artifact of the joint optimization pipeline.
minor comments (2)
  1. [Abstract] Abstract: the description of the MRI cohorts lacks sample sizes, preprocessing details, or specific quantitative results on anatomical changes, limiting reproducibility assessment.
  2. [Abstract] Abstract: the LPIPS minimality check is mentioned but no numerical values or thresholds are provided, making it hard to judge how 'close' the counterfactuals remain to the originals.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the thoughtful and constructive report. We address the two major comments point-by-point below, indicating where revisions will be made to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract (method paragraph): the optimization directly minimizes the MMD statistic of the pretrained test model in its own representation space; any successful optimization will therefore reduce that statistic and raise p-values by construction. The interpretive claim that the edits reflect 'genuine distributional features' therefore rests on the unverified assumption that the test model's representations are faithful to the true data-generating process, yet no independent validation (different test architecture, held-out training of the test model, or non-learned baseline) is described.

    Authors: We agree that successful optimization of the MMD objective in the test model's representation space will reduce the test statistic (and raise p-values) by construction for that specific model. The framework is explicitly designed to generate counterfactuals that explain the pretrained deep test; the resulting edits therefore highlight features that drive rejection under the learned representation. To address the concern about model-specific artifacts, we will add new experiments in the revision that repeat the procedure with (i) an alternative test architecture and (ii) a non-learned baseline (pixel-space MMD optimization) to check whether the identified features remain consistent. revision: yes

  2. Referee: [Abstract] Abstract (evaluation paragraph): the reported consistency of p-value increases is presented without quantitative tables, error bars, ablation studies, or comparison against alternative explanation methods; this makes it difficult to assess whether the observed effect size is robust or merely an artifact of the joint optimization pipeline.

    Authors: The abstract is a concise summary; the full manuscript (Sections 4–5 and supplementary material) already contains quantitative tables of p-value changes, error bars across multiple random seeds, ablation studies on the diffusion autoencoder and MMD components, and LPIPS-based minimality checks. No direct comparison to other explanation methods is currently present. We will revise the abstract to report key numerical effect sizes and will add a comparison subsection against at least one alternative counterfactual method in the revision. revision: partial

Circularity Check

1 steps flagged

P-value increases forced by construction via direct MMD optimization in test representation space

specific steps
  1. fitted input called prediction [abstract]
    "Our method combines a diffusion autoencoder with a pretrained deep two-sample test model and optimizes a maximum mean discrepancy (MMD) objective in the test model's representation space to produce plausible counterfactuals. We quantify distribution-level effects through changes in the test statistic and the resulting two-sample p-values. ... Across both settings, the counterfactual transformations consistently increase p-values relative to the original samples, indicating that the edited source set becomes statistically closer to the target distribution under the test."

    The procedure explicitly minimizes the MMD (test statistic) to produce edits, then presents the resulting p-value increases and reduced discrepancy as evidence that the edited set is closer under the test. Because any successful optimization of that exact objective must lower the statistic and raise p-values, the reported distributional effect is forced by construction of the optimization step rather than an independent finding.

full rationale

The paper optimizes the MMD objective (core of the pretrained test statistic) inside the test model's representation space to generate counterfactual edits, then reports that these edits raise p-values and reduce distributional discrepancy under the same test. This reported effect reduces directly to the optimization success rather than providing independent evidence. The interpretive claim that edits reflect genuine features (vs. model artifacts) rests on an unverified assumption about representation faithfulness, with no described independent validation (e.g., alternate test models or non-learned baselines) to break the dependence. This matches fitted-input-called-prediction circularity on the central evaluation metric while the overall method remains a novel pipeline.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim depends on the pretrained test model correctly capturing distributional differences and on the diffusion autoencoder generating edits that remain within the data manifold without introducing spurious signals that the test model would misinterpret.

axioms (1)
  • domain assumption The pretrained deep two-sample test model accurately measures distributional discrepancy in its representation space.
    The optimization target and evaluation both rely on this model being a faithful proxy for the true difference.

reviewed 2026-06-28 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Counterfactual Explanations for Deep Two-Sample Testing." pith.science (2026). https://pith.science/paper/EPT37QVY

@misc{pith2026260604009,
  author       = {Pith},
  title        = {Pith review of: Counterfactual Explanations for Deep Two-Sample Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EPT37QVY}},
  note         = {Machine review of arXiv:2606.04009}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Two-sample testing is a fundamental tool for detecting distributional differences across scientific domains, but classical tests (including kernel-based tests) can be ineffective on high-dimensional structured data such as images. Recent deep two-sample tests improve sensitivity in these settings by learning informative representations, yet they provide limited insight into which data features drive rejection of the null hypothesis $H_0$. To address this issue, we propose a counterfactual explanation framework for deep two-sample testing that generates sample-level edits moving observations from a source group toward a target group while explicitly reducing the discrepancy measured by the test. Our method combines a diffusion autoencoder with a pretrained deep two-sample test model and optimizes a maximum mean discrepancy (MMD) objective in the test model's representation space to produce plausible counterfactuals. We quantify distribution-level effects through changes in the test statistic and the resulting two-sample p-values. We evaluate the method on synthetic 2D shape datasets and two MRI cohorts. Across both settings, the counterfactual transformations consistently increase p-values relative to the original samples, indicating that the edited source set becomes statistically closer to the target distribution under the test. We measure minimality using LPIPS to ensure the counterfactuals remain close to the original samples. The resulting edits provide interpretable evidence of the features associated with the detected group differences. On MRI, the localized changes are consistent with known anatomical differences between cohorts.

Figures

Figures reproduced from arXiv: 2606.04009 by Christoph Lippert, Marco Simnacher, Wei-Cheng Lai.

Figure 1
Figure 1. Figure 1: Overview of the proposed deep-test-guided counterfactual generation framework. A pretrained deep two-sample test f defines a discrepancy objective between source and target groups. A source sample Xi is encoded in the latent space of a diffusion autoencoder z sem i and edited along a direction d given by the gradient of this objective, moving it toward the target distribution Ym. The resulting coun￾terfact… view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative counterfactual examples generated from the proposed method for the MRI experiments. Each panel shows source images, generated counterfactuals, and absolute difference heatmaps. The heatmaps highlight the spatially localized edits induced by the counterfactual optimization. On the left column (a), the counterfactual generation is optimized by the CDR criteria separation, compared to the counterf… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative counterfactual examples generated from the proposed method for the dSprites shape experiments. Each panel shows source images, generated counterfactuals, and absolute difference heatmaps. The heatmaps highlight the spatially localized edits induced by the counterfactual optimization. Generated counterfactuals move toward the target group (ellipse). improvement with increasing λ. Overall, the ab… view at source ↗
Figure 4
Figure 4. Figure 4: Ablation study over the regularization weight λ and the number of edited source samples n in the MRI experiments. Top row: CDR-based source/target sepa￾ration. Bottom row: Age-based source/target separation. Left column: gain in downstream two-sample p-value (∆p). Right column: gain in discrepancy reduc￾tion (∆t). Larger values are better in all panels. Across both criteria, increasing λ consistently impro… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 4 canonical work pages

  1. [1]

    UK Biobank: An Open Access Resource for Identifying the Causes of a Wide Range of Complex Diseases of Middle and Old Age , volume =

    Cathie Sudlow and John Gallacher and Naomi Allen and Valerie Beral and Paul Burton and John Danesh and Paul Downey and Paul Elliott and Jane Green and Martin Landray and Bette Liu and Paul Matthews and Giok Ong and Jill Pell and Alan Silman and Alan Young and Tim Sprosen and Tim Peakman and Rory Collins , doi =. UK Biobank: An Open Access Resource for Ide...

  2. [2]

    Alzheimer's Disease Neuroimaging Initiative ( ADNI) : clinical characterization

    Petersen, R C and Aisen, P S and Beckett, L A and Donohue, M C and Gamst, A C and Harvey, D J and Jack, Jr, C R and Jagust, W J and Shaw, L M and Toga, A W and Trojanowski, J Q and Weiner, M W. Alzheimer's Disease Neuroimaging Initiative ( ADNI) : clinical characterization. Neurology

  3. [3]

    Statistical Science , pages=

    Permutation methods: a basis for exact inference , author=. Statistical Science , pages=. 2004 , publisher=

    2004

  4. [4]

    Borgwardt and Malte J

    Arthur Gretton and Karsten M. Borgwardt and Malte J. Rasch and Bernhard Sch. A Kernel Two-Sample Test , journal =. 2012 , volume =

    2012

  5. [5]

    Two-sample Testing Using Deep Learning , ISSN=

    Kirchler, Matthias and Khorasani, Shahryar and Kloft, Marius and Lippert, Christoph , year=. Two-sample Testing Using Deep Learning , ISSN=. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , publisher=

  6. [6]

    Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LXIII , pages =

    Sobieski, Bartlomiej and Biecek, Przemyslaw , title =. Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LXIII , pages =. 2024 , isbn =. doi:10.1007/978-3-031-73036-8_5 , abstract =

    work page doi:10.1007/978-3-031-73036-8_5 2024

  7. [7]

    IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    Diffusion Autoencoders: Toward a Meaningful and Decodable Representation , author=. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , year=

  8. [8]

    International conference on machine learning , pages=

    Learning deep kernels for non-parametric two-sample tests , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  9. [9]

    International Conference on Learning Representations , year=

    Revisiting Classifier Two-Sample Tests , author=. International Conference on Learning Representations , year=

  10. [10]

    and Cogswell, Michael and Das, Abhishek and Vedantam, Ramakrishna and Parikh, Devi and Batra, Dhruv , booktitle=

    Selvaraju, Ramprasaath R. and Cogswell, Michael and Das, Abhishek and Vedantam, Ramakrishna and Parikh, Devi and Batra, Dhruv , booktitle=. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , year=

  11. [11]

    Proceedings of the 34th International Conference on Machine Learning - Volume 70 , pages =

    Shrikumar, Avanti and Greenside, Peyton and Kundaje, Anshul , title =. Proceedings of the 34th International Conference on Machine Learning - Volume 70 , pages =. 2017 , publisher =

    2017

  12. [12]

    Springenberg and A

    J.T. Springenberg and A. Dosovitskiy and T. Brox and M. Riedmiller. Striving for Simplicity: The All Convolutional Net. ICLR (workshop track). 2015

    2015

  13. [13]

    Proceedings of the 34th International Conference on Machine Learning - Volume 70 , pages =

    Sundararajan, Mukund and Taly, Ankur and Yan, Qiqi , title =. Proceedings of the 34th International Conference on Machine Learning - Volume 70 , pages =. 2017 , publisher =

    2017

  14. [14]

    Proceedings of the Fourth Conference on Medical Imaging with Deep Learning , pages =

    Gifsplanation via Latent Shift: A Simple Autoencoder Approach to Counterfactual Generation for Chest X-rays , author =. Proceedings of the Fourth Conference on Medical Imaging with Deep Learning , pages =. 2021 , editor =

    2021

  15. [15]

    Explaining the black-box smoothly—A counterfactual approach , journal =

    Sumedha Singla and Motahhare Eslami and Brian Pollack and Stephen Wallace and Kayhan Batmanghelich , keywords =. Explaining the black-box smoothly—A counterfactual approach , journal =. 2023 , issn =. doi:https://doi.org/10.1016/j.media.2022.102721 , url =

    work page doi:10.1016/j.media.2022.102721 2023

  16. [16]

    International Conference on Learning Representations , year=

    Explanation by Progressive Exaggeration , author=. International Conference on Learning Representations , year=

  17. [17]

    arXiv preprint arXiv:2412.00613 , year=

    A unified data representation learning for non-parametric two-sample testing , author=. arXiv preprint arXiv:2412.00613 , year=

    work page arXiv

  18. [18]

    CVPR , year=

    The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , author=. CVPR , year=

  19. [19]

    Proceedings of the 34th International Conference on Neural Information Processing Systems , articleno =

    Ho, Jonathan and Jain, Ajay and Abbeel, Pieter , title =. Proceedings of the 34th International Conference on Neural Information Processing Systems , articleno =. 2020 , isbn =

    2020

  20. [20]

    Proceedings of the 38th International Conference on Machine Learning , pages =

    Improved Denoising Diffusion Probabilistic Models , author =. Proceedings of the 38th International Conference on Machine Learning , pages =. 2021 , editor =

    2021

  21. [21]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj\"orn , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2022 , pages =

    2022

  22. [22]

    Kingma and Max Welling , editor =

    Diederik P. Kingma and Max Welling , editor =. Auto-Encoding Variational Bayes , booktitle =. 2014 , url =

    2014

  23. [23]

    and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and Warde-Farley, David and Ozair, Sherjil and Courville, Aaron and Bengio, Yoshua , booktitle =

    Goodfellow, Ian J. and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and Warde-Farley, David and Ozair, Sherjil and Courville, Aaron and Bengio, Yoshua , booktitle =. Generative Adversarial Nets , volume =

  24. [24]

    2007 , doi =

    Lim, Wee Shiong and Chong, Mei Sian and Sahadevan, Suresh , title =. 2007 , doi =. http://www.clinmedres.org/content/5/1/61.full.pdf , journal =

    2007

  25. [25]

    , title =

    Melistas, Thomas and Spyrou, Nikos and Gkouti, Nefeli and Sanchez, Pedro and Vlontzos, Athanasios and Panagakis, Yannis and Papanastasiou, Giorgos and Tsaftaris, Sotirios A. , title =. Proceedings of the 38th International Conference on Neural Information Processing Systems , articleno =. 2024 , isbn =

    2024

  26. [26]

    Friedrich and Felix Nensa , keywords =

    Katarzyna Borys and Yasmin Alyssa Schmitt and Meike Nauta and Christin Seifert and Nicole Krämer and Christoph M. Friedrich and Felix Nensa , keywords =. Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches , journal =. 2023 , issn =. doi:https://doi.org/10.1016/j.ejrad.2023.110786 , url =

    work page doi:10.1016/j.ejrad.2023.110786 2023

  27. [27]

    Proceedings of the 38th International Conference on Machine Learning , pages =

    Benchmarks, Algorithms, and Metrics for Hierarchical Disentanglement , author =. Proceedings of the 38th International Conference on Machine Learning , pages =. 2021 , editor =

    2021

  28. [28]

    Loic Matthey and Irina Higgins and Demis Hassabis and Alexander Lerchner , title =. 2017

    2017

  29. [29]

    C. K. Chow and C. N. Liu , title =. IEEE Transactions on Information Theory , year =

This paper was first reviewed by grok-4.3 on June 28, 2026.